keras-io/

structured-data-classification-grn-vsn

Model description

1. Gated Residual Networks(GRN): consists of skip connections and gating layers that facilitate information flow efficiently. They have the flexibility to apply non-linear processing only where needed. GRNs make use of Gated Linear Units (or GLUs) to suppress the input that are not relevant for a given task.
   The GRN works as follows:
   • It first applies Non-linear ELU tranformation on its inputs
   • It then applies a linear transformation followed by dropout
   • Next it applies GLU and adds the original inputs to the output of the GLU to perform skip (residual) connection
   • Finally, it applies layer normalization and produces its output
2. Variable Selection Networks(VSN): help in carefully selecting the most important features from the input and getting rid of any unnecessary noisy inputs which could harm the model's performance. The VSN works as follows:
   • First, it applies a Gated Residual Network (GRN) to each feature individually.
   • Then it concatenates all features and applies a GRN on the concatenated features, followed by a softmax to produce feature weights
   • It produces a weighted sum of the output of the individual GRN

Intended uses

Training and evaluation data

Training procedure

1. Prepare Data: Load the training and test datasets and convert the target column income_level from string to integer. The training dataset is further split into train and validation sets. Finally, the training and validation datasets are then converted into a tf.data.Dataset meant to be used for model training and evaluation.
2. Define logic for Encoding input features: We encode the categorical and numerical features as follows:
   • Categorical Features: are encoded using Embedding layer provided by Keras. The output dimension of the embedding is equal to encoding_size
   • Numerical Features: are projected into a encoding_size dimensional vector by applying a linear transformation using Dense layer provided by Keras
   Therefore, all the encoded features will have the same dimensionality equal to the value of encoding_size.
3. Create Model:
   • The model will have input layers corresponding to both numerical and categorical features of the given dataset
   • The features received by the input layers are then encoded using the encoding logic defined in Step 2 with an encoding_size of 16 indicating the output dimension of the encoded features.
   • The encoded features pass through the Variable Selection Network(VSN). The VSN internally makes use of the GRN as well, as explained in the Model Description section.
   • The features produced by the VSN are passed through a final Dense layer with sigmoid activation to produce the final output of the model indicating the probability for whether the income of a person is >500K or not.
4. Compile, Train and Evaluate Model:
   • Since the model is meant to binary classification, the loss function chosen was Binary Cross Entropy.
   • The metric chosen for evaluating the model's performance was accuracy.
   • The optimizer chosen was Adam with a learning rate of 0.001.
   • The dropout_rate for the Dropout Layers of the GRN was 0.15
   • The batch_size chosen was 265 and the model was trained for 20 epochs.
   • The training was done with a Keras callback for EarlyStopping which means the training would be interrupted as soon as the validation metrics have stopped improving.
   • Finally the performance of the model was also evaluated on the test_dataset reaching an accuracy of ~95%

Training hyperparameters

Model Plot

View Model Plot

Credits:

• HF Contribution: Shivalika Singh
• Full credits to original Keras example by Khalid Salama
• Check out the demo space here