Credit Card Attrition Classification Through Neuronets

The PS-WASD Based Neuronet

Let $X$ = $\left[\begin{array}{l}{X}_{\mathrm{1}},X_{\mathrm{2}},\dots ,X_m\end{array}\right]\in {\mathbb{R}}^{s\times m}$ be the normalized input matrix in the interval $[-\mathrm{0.5},-\mathrm{0.25}]$ with $s$ denoting the number of the samples and $Y\in {\mathbb{R}}^s$ be a binary target vector. Notice that in order to avoid over-fitting, the inputs $X$ must be normalized in the interval $[-\mathrm{0.5},-\mathrm{0.25}]$, which can be done by applying the following transformation (Zhang et al., 2019): $X_{nor}=\frac{X-\mathrm{m}\mathrm{i}\mathrm{n}\left(x\right)}{\mathrm{4}\left(\mathrm{m}\mathrm{a}\mathrm{x}\right(x)-\mathrm{m}\mathrm{i}\mathrm{n}(x\left)\right)}-\frac{\mathrm{1}}{\mathrm{2}}$. It should also be noted that all operations on $X$ are element-wise in this section.

Assume that there exists an underlying relationship $f$ such that $Y=f(X_{\mathrm{1}},X_{\mathrm{2}},\dots ,X_m)$. The neuronet aims at approximating $f$ through a linear combination of $n$ activation functions $F$, where $F_d\left(x\right)=\mathrm{l}\mathrm{n}(\mathrm{1}+e^{X^d})$ is the power softplus activation as in (Simos et al., 2022), with $d=\mathrm{0,1},\dots ,n-\mathrm{1}$. For each activation $F_d$, let $q_d$ denote the image of $X$ under $F_d$. Thus, $q_d\in {\mathbb{R}}^{s\times m}$. Assuming a weights vector $W=\left[\begin{array}{l}{W}_{\mathrm{0}},W_{\mathrm{1}},\dots ,W_{n-\mathrm{1}}\end{array}\right]\in {\mathbb{R}}^{mn\times n}$, a linear combination of all $n$ images may be written as ${\sum }_{i=\mathrm{0}}^{n-\mathrm{1}}\mathrm{‍}{W}_i{q}_i=\hat{Y}$, where $Q=\left[\begin{array}{l}{q}_{\mathrm{0}},q_{\mathrm{1}},\dots ,q_{n-\mathrm{1}}\end{array}\right]\in {\mathbb{R}}^{s\times mn}$. Note that $\hat{Y}$, the neuronet's output, must be converted to binary. To this end, an element-wise function $B$ is employed so that the final output of the neuronet is $B\left(\hat{Y}\right)$ with

$B_i\left(\hat{Y}\right)=\left\{\begin{array}{l}\mathrm{1},{\hat{Y}}_i\ge -\mathrm{0.375}\\ \mathrm{0},{\hat{Y}}_i<-\mathrm{0.375}\end{array}\right.\mathrm{f}\mathrm{o}\mathrm{r}i=\mathrm{1,2},\dots ,s,$

where 1 implies a customer that is likely to attrite and vice versa. On the convergence of the neuronet, the following Theorem 1 and Proposition 1 should be noted.

Let $K$ be a nonnegative integer and assume a target function $f(\cdot )$ that has the $(K+\mathrm{1})$-order continuous derivative on interval $[a,b]$, then for $x\in [a,b]$,

$f\left(x\right)=P_K\left(x\right)+R_K\left(x\right),$ (1)

where $P_K\left(x\right)$ is a polynomial used to approximate $f\left(x\right)$ and $R_K\left(x\right)$ is the error term.

Given a constant value $h\in [a,b]$, $f\left(x\right)$ can be approximated as:

$f\left(x\right)\approx P_K\left(x\right)=\underset{i=\mathrm{0}}{\overset{K}{\sum }}\mathrm{‍}\frac{f^{\left(i\right)}\left(h\right)}{i!}(x-h{)}^i,$ (2)

where $f^{\left(i\right)}\left(h\right)$ is the value of the $i$-order derivative of $f\left(x\right)$ at $h$, $i!$ implies the factorial of $i$, and $P_K\left(x\right)$ signifies the K-order Taylor polynomial of function $f\left(x\right)$.

Theorem 1 can be used to approximate multivariable functions (Zhang et al., 2019). For a target function $f(x_{\mathrm{1}},x_{\mathrm{2}},\dots ,x_g)$ with $(K+\mathrm{1})$-order continuous partial derivatives in an origin's neighborhood $(\mathrm{0},\dots ,\mathrm{0})$ and $g$ variables, the $K$-order Taylor polynomial $P_K(x_{\mathrm{1}},x_{\mathrm{2}},\dots ,x_g)$ to the origin is:

$P_K(x_{\mathrm{1}},x_{\mathrm{2}},\dots ,x_g)=\underset{i=\mathrm{0}}{\overset{K}{\sum }}\mathrm{‍}\underset{i_{\mathrm{1}}+\dots +i_g=i}{\sum }\mathrm{‍}\frac{x_{\mathrm{1}}\cdots x_g}{i_{\mathrm{1}}\cdots i_g}\left(\frac{{\partial }^{i_{\mathrm{1}}+\dots +i_g}f(\mathrm{0},\cdots ,\mathrm{0})}{\partial x_{\mathrm{1}}^{i_{\mathrm{1}}}\cdots \partial x_g^{i_g}}\right),$ (3)

where $i_{\mathrm{1}},i_{\mathrm{2}},\dots ,i_g$ are nonnegative integers.

Based on the aforementioned analysis, the optimal weights $W$ can be obtained through the WDD process (Zhang et al., 2019) as follows:

$W=Q^{†}Y,$ (4)

where $Q^{†}$ denotes the pseudo-inverse of $Q$.

The PS-WASD Algorithm

With the computation of the optimal weights being handled by (4), the next concern is the determination of the optimal size for the neuronet. This is done through the PS-WASD algorithm that is employed in conjunction with cross validation so as to ensure that the model generalizes beyond the set of training. The PS-WASD algorithm can be described in the following three steps.

Considering the normalized input matrix $X$, the training and testing sets are determined by a random $\mathrm{80}\mathrm{\%}-\mathrm{20}\mathrm{\%}$ split. The training set is once again divided into two sets of samples, one for model fitting and the other for validation. The parameter $p\in \left(\mathrm{0,1}\right]\subseteq \mathbb{R}$, which is set by the user, represents the percentage of samples that is to be allocated for model fitting purposes.

Assuming that the training set consists of $J=\frac{\mathrm{80}}{\mathrm{100}}s$ samples, we use the first $J_{\mathrm{1}}=pJ$ samples for fitting the model, and the rest $J_{\mathrm{2}}=J-J_{\mathrm{1}}$ samples for validation. Starting from the power $d=\mathrm{0}$ and an empty matrix $Q$, the PS-WASD algorithm grows $Q$ by adding the respective columns $q_{d+\mathrm{1}}$, whilst incrementing $d$. In each iteration, the optimal weights are computed with respect to the training set as in Eq. 4 and the current performance of the neuronet is assessed on the validation set through a metric of choice, namely the mean-absolute-error (MAE $=\frac{\mathrm{1}}{J_{\mathrm{2}}}{\sum }_{j=\mathrm{1}}^{J_{\mathrm{2}}}\mathrm{‍}|{\hat{Y}}_j-Y_j|$). Added columns are only kept if their addition has resulted in a performance boost; that is, a lower MAE.

This process is repeated until $d$ reaches a predefined threshold, which we set to be $d_{max}=\mathrm{30}$. In reaching $d_{max}$, the training and validation sets are brought together and the final set of weights is computed relative to all available training and validation samples.

To conclude, the PS-WASD training algorithm not only guarantees that the weights are optimal, but also facilitates future computation by growing the neuronet's structure to the smallest size possible. In combination with cross validation as described above, the PS-WASD training algorithm is indeed a powerful tool.