You can deploy this model and start earning money today!

whaleloops/
keptlongformer

Name: keptlongformer
Rating: 2.5 (17 reviews)

Views

No views yet

KEPTlongfomer is a medical knowledge enhanced version of Longformer that was further pre-trained using contrastive learning. The model achieves SOTA performance on auto ICD coding on MIMIC-III as of 11/12/2022. A sister model for better performance is available here.

Pre-training

We initialized this model from clinical longformer.

And then pretrained with Hierarchical Self-Alignment Pretrain (HSAP) using Knowledge Graph UMLS. This includes (a) Hierarchy, (b) Synonym, (c) Abbreviation. For more info, see section 3.3 in paper. The learning rate was 5e-5, weight decay was 0.01, adam epsilon was 1e-5.

Usage

See our github for how to use this with prompts on auto ICD coding.

With the following result:

Metric	Score
rec_micro	=0.5729403619819988
rec_macro	=0.11342156911120573
rec_at_8	=0.4094837705486378
rec_at_75	=0.8470734920535119
rec_at_50	=0.8005338782352
rec_at_5	=0.2891628170355805
rec_at_15	=0.5768778119750537
prec_micro	=0.6411968713105065
prec_macro	=0.12227610414493029
prec_at_8	=0.7760972716488731
prec_at_75	=0.197504942665085
prec_at_50	=0.2768090154211151
prec_at_5	=0.8483392645314354
prec_at_15	=0.6178529062870699
f1_micro	=0.6051499904242899
f1_macro	=0.11768251595637802
f1_at_8	=0.536107150495997
f1_at_75	=0.32032290907137506
f1_at_50	=0.411373195944102
f1_at_5	=0.43131028155283435
f1_at_15	=0.5966627077602488
auc_micro	=0.9651754312635265
auc_macro	=0.8566590059725866
acc_micro	=0.43384592341105344
acc_macro	=0.08639139221100567

whaleloops/keptlongformer

Pre-training

Usage

whaleloops/
keptlongformer