Model Details
Model Name: NumericBERT
Model Type: Transformer
Architecture: BERT
Training Method: Masked Language Modeling (MLM)
Training Data: MIMIC IV Lab values data
Training Hyperparameters:
-
Optimizer: AdamW
-
Learning Rate: 5e-5
-
Masking Rate: 20%
-
Tokenization: Custom numeric-to-text mapping using the TextEncoder class
Text Encoding Process
Overview: Non-negative integers are converted into uppercase letter-based representations, allowing numerical values to be expressed as sequences of letters.
Normalization and Binning:
-
Method: Log normalization and splitting into 10 bins.
-
Representation: Each bin is represented by a letter (A-J).
Token Construction:
Format: <<lab_id_token>> <<lab_id_token>><<lab_value_bin>>
Example: For a lab value of type 'Bic' with a normalized value in bin 'C', the token might be Bic BicC.
Columns Used: 'Bic', 'Crt', 'Pot', 'Sod', 'Ure', 'Hgb', 'Plt', 'Wbc'.
Training Data Preprocessing
Column Selection: Numerical values from selected lab values.
Text Encoding: Numeric values are encoded into text using the process described above.
Masking: 20% of the data is randomly masked during training.
Model Output
Description: Outputs predictions for masked values during training.
Format: Contains the encoded text representing the predicted lab values.
Limitations and Considerations
Numeric Data Representation: The custom text representation may have limitations in capturing the intricacies of the original numeric data.
Training Data Source: Performance may be influenced by the characteristics and biases inherent in the MIMIC IV dataset.
Generalizability: The model's effectiveness outside the context of the training dataset is not guaranteed.
Contact Information
David Restrepo
MIT Critical Data - MIT
