Loss: 0.0024
Accuracy: 0.9995
Precision: 0.7342
Recall: 0.9159
F1: 0.8150
Classification Report: {'DebtInstrumentBasisSpreadOnVariableRate1': {'precision': 0.791095890410959, 'recall': 0.9602137767220903, 'f1-score': 0.8674892703862661, 'support': 1684}, 'DebtInstrumentFaceAmount': {'precision': 0.6338946224877784, 'recall': 0.8670133729569094, 'f1-score': 0.7323501725760904, 'support': 1346}, 'DebtInstrumentInterestRateStatedPercentage': {'precision': 0.774822695035461, 'recall': 0.9494839760999457, 'f1-score': 0.8533072980229436, 'support': 1841}, 'LineOfCreditFacilityMaximumBorrowingCapacity': {'precision': 0.7227383863080684, 'recall': 0.8740390301596689, 'f1-score': 0.791220556745182, 'support': 1691}, 'micro avg': {'precision': 0.7341803078426582, 'recall': 0.9158793050899117, 'f1-score': 0.8150257662055873, 'support': 6562}, 'macro avg': {'precision': 0.7306378985605667, 'recall': 0.9126875389846535, 'f1-score': 0.8110918244326205, 'support': 6562}, 'weighted avg': {'precision': 0.7366697400377676, 'recall': 0.9158793050899117, 'f1-score': 0.8161365377528546, 'support': 6562}}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

Subset of nlpaueb/finer-139 train split and and full validation split. Given the original proportion of 'O' labels (approx. 80%), we reduced the train split records to match the original majority class proportion given the reduced set of labels. Original train split size: 900384 records Subset train split size: 142513 records (~16% of original amount of records)

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 192
eval_batch_size: 192
seed: 42
optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 500
num_epochs: 20
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Precision	Recall	F1	Classification Report
0.0063	0.6729	500	0.0035	0.9987	0.4732	0.9122	0.6232	{'DebtInstrumentBasisSpreadOnVariableRate1': {'precision': 0.6499202551834131, 'recall': 0.9679334916864608, 'f1-score': 0.7776717557251909, 'support': 1684}, 'DebtInstrumentFaceAmount': {'precision': 0.3665031534688157, 'recall': 0.7771173848439822, 'f1-score': 0.49809523809523815, 'support': 1346}, 'DebtInstrumentInterestRateStatedPercentage': {'precision': 0.5214669051878354, 'recall': 0.9500271591526345, 'f1-score': 0.673339749759384, 'support': 1841}, 'LineOfCreditFacilityMaximumBorrowingCapacity': {'precision': 0.3968980422069667, 'recall': 0.9231224127735068, 'f1-score': 0.5551209103840683, 'support': 1691}, 'micro avg': {'precision': 0.4732389912246027, 'recall': 0.9122218835720817, 'f1-score': 0.62318463380355, 'support': 6562}, 'macro avg': {'precision': 0.4836970890117577, 'recall': 0.904550112114146, 'f1-score': 0.6260569134909704, 'support': 6562}, 'weighted avg': {'precision': 0.49054466871695807, 'recall': 0.9122218835720817, 'f1-score': 0.6337036522224775, 'support': 6562}}
0.0044	1.3459	1000	0.0019	0.9993	0.6299	0.8856	0.7361	{'DebtInstrumentBasisSpreadOnVariableRate1': {'precision': 0.7140974967061924, 'recall': 0.9655581947743468, 'f1-score': 0.8210047967684928, 'support': 1684}, 'DebtInstrumentFaceAmount': {'precision': 0.5513164965072541, 'recall': 0.7622585438335809, 'f1-score': 0.6398503274087932, 'support': 1346}, 'DebtInstrumentInterestRateStatedPercentage': {'precision': 0.6613592233009709, 'recall': 0.9250407387289517, 'f1-score': 0.771286231884058, 'support': 1841}, 'LineOfCreditFacilityMaximumBorrowingCapacity': {'precision': 0.5793871866295265, 'recall': 0.8610289769367239, 'f1-score': 0.6926736441484301, 'support': 1691}, 'micro avg': {'precision': 0.6298504227184045, 'recall': 0.8855531850045718, 'f1-score': 0.7361287053458322, 'support': 6562}, 'macro avg': {'precision': 0.626540100785986, 'recall': 0.8784716135684008, 'f1-score': 0.7312037500524435, 'support': 6562}, 'weighted avg': {'precision': 0.6311975390794893, 'recall': 0.8855531850045718, 'f1-score': 0.7368271416647247, 'support': 6562}}
0.0038	2.0188	1500	0.0018	0.9994	0.6685	0.8735	0.7573	{'DebtInstrumentBasisSpreadOnVariableRate1': {'precision': 0.8146964856230032, 'recall': 0.9085510688836105, 'f1-score': 0.8590679393599102, 'support': 1684}, 'DebtInstrumentFaceAmount': {'precision': 0.5430597771023303, 'recall': 0.7964338781575037, 'f1-score': 0.6457831325301205, 'support': 1346}, 'DebtInstrumentInterestRateStatedPercentage': {'precision': 0.6825586015097338, 'recall': 0.933188484519283, 'f1-score': 0.7884350619550253, 'support': 1841}, 'LineOfCreditFacilityMaximumBorrowingCapacity': {'precision': 0.6400725294650952, 'recall': 0.8350088704908338, 'f1-score': 0.7246599948678469, 'support': 1691}, 'micro avg': {'precision': 0.6684548104956268, 'recall': 0.8735141725083816, 'f1-score': 0.7573495408601439, 'support': 6562}, 'macro avg': {'precision': 0.6700968484250407, 'recall': 0.8682955755128078, 'f1-score': 0.7544865321782257, 'support': 6562}, 'weighted avg': {'precision': 0.6769064880331865, 'recall': 0.8735141725083816, 'f1-score': 0.7608661241463521, 'support': 6562}}
0.003	2.6918	2000	0.0017	0.9994	0.6856	0.9012	0.7788	{'DebtInstrumentBasisSpreadOnVariableRate1': {'precision': 0.7413636363636363, 'recall': 0.9685273159144893, 'f1-score': 0.8398558187435634, 'support': 1684}, 'DebtInstrumentFaceAmount': {'precision': 0.598568281938326, 'recall': 0.8075780089153046, 'f1-score': 0.6875395319418091, 'support': 1346}, 'DebtInstrumentInterestRateStatedPercentage': {'precision': 0.7389801210025929, 'recall': 0.928843020097773, 'f1-score': 0.8231046931407943, 'support': 1841}, 'LineOfCreditFacilityMaximumBorrowingCapacity': {'precision': 0.6472125435540069, 'recall': 0.8787699586043761, 'f1-score': 0.7454226235264609, 'support': 1691}, 'micro avg': {'precision': 0.685601669371667, 'recall': 0.9012496190185919, 'f1-score': 0.7787727153015538, 'support': 6562}, 'macro avg': {'precision': 0.6815311457146406, 'recall': 0.8959295758829857, 'f1-score': 0.773980666838157, 'support': 6562}, 'weighted avg': {'precision': 0.6871423476136771, 'recall': 0.9012496190185919, 'f1-score': 0.7795779953083334, 'support': 6562}}
0.0022	3.3647	2500	0.0018	0.9994	0.6925	0.8973	0.7817	{'DebtInstrumentBasisSpreadOnVariableRate1': {'precision': 0.7884519661523146, 'recall': 0.9406175771971497, 'f1-score': 0.8578391551584078, 'support': 1684}, 'DebtInstrumentFaceAmount': {'precision': 0.6066779852857951, 'recall': 0.7964338781575037, 'f1-score': 0.6887247028589785, 'support': 1346}, 'DebtInstrumentInterestRateStatedPercentage': {'precision': 0.729933110367893, 'recall': 0.9483976099945681, 'f1-score': 0.8249468462083628, 'support': 1841}, 'LineOfCreditFacilityMaximumBorrowingCapacity': {'precision': 0.6364025695931478, 'recall': 0.8787699586043761, 'f1-score': 0.7382016890213613, 'support': 1691}, 'micro avg': {'precision': 0.6924614841820534, 'recall': 0.8972874123742761, 'f1-score': 0.7816793893129771, 'support': 6562}, 'macro avg': {'precision': 0.6903664078497876, 'recall': 0.8910547559883993, 'f1-score': 0.7774280983117776, 'support': 6562}, 'weighted avg': {'precision': 0.695566181128388, 'recall': 0.8972874123742761, 'f1-score': 0.7830921650929078, 'support': 6562}}
0.0022	4.0377	3000	0.0019	0.9994	0.6973	0.9098	0.7895	{'DebtInstrumentBasisSpreadOnVariableRate1': {'precision': 0.764594209776934, 'recall': 0.9566508313539193, 'f1-score': 0.8499076760749142, 'support': 1684}, 'DebtInstrumentFaceAmount': {'precision': 0.6087877183695076, 'recall': 0.8543833580980683, 'f1-score': 0.7109737248840804, 'support': 1346}, 'DebtInstrumentInterestRateStatedPercentage': {'precision': 0.7853860294117647, 'recall': 0.9282998370450842, 'f1-score': 0.8508837440876277, 'support': 1841}, 'LineOfCreditFacilityMaximumBorrowingCapacity': {'precision': 0.6276150627615062, 'recall': 0.8870490833826138, 'f1-score': 0.7351139426611124, 'support': 1691}, 'micro avg': {'precision': 0.6972669936930623, 'recall': 0.9097836025601951, 'f1-score': 0.7894736842105262, 'support': 6562}, 'macro avg': {'precision': 0.6965957550799281, 'recall': 0.9065957774699215, 'f1-score': 0.7867197719269337, 'support': 6562}, 'weighted avg': {'precision': 0.7031694101594759, 'recall': 0.9097836025601951, 'f1-score': 0.7921014645092033, 'support': 6562}}
0.0017	4.7106	3500	0.0018	0.9995	0.7319	0.8888	0.8028	{'DebtInstrumentBasisSpreadOnVariableRate1': {'precision': 0.8151781104801239, 'recall': 0.9376484560570071, 'f1-score': 0.872134769400718, 'support': 1684}, 'DebtInstrumentFaceAmount': {'precision': 0.6239267315397825, 'recall': 0.8098068350668648, 'f1-score': 0.704817329453605, 'support': 1346}, 'DebtInstrumentInterestRateStatedPercentage': {'precision': 0.7620508326029798, 'recall': 0.9445953286257469, 'f1-score': 0.8435605141886975, 'support': 1841}, 'LineOfCreditFacilityMaximumBorrowingCapacity': {'precision': 0.7112887112887113, 'recall': 0.8421052631578947, 'f1-score': 0.7711887354454373, 'support': 1691}, 'micro avg': {'precision': 0.7319277108433735, 'recall': 0.8887534288326729, 'f1-score': 0.8027529249827942, 'support': 6562}, 'macro avg': {'precision': 0.7281110964778994, 'recall': 0.8835389707268784, 'f1-score': 0.7979253371221144, 'support': 6562}, 'weighted avg': {'precision': 0.7342715806632691, 'recall': 0.8887534288326729, 'f1-score': 0.8037845375457159, 'support': 6562}}
0.0013	5.3836	4000	0.0020	0.9995	0.7302	0.9075	0.8093	{'DebtInstrumentBasisSpreadOnVariableRate1': {'precision': 0.7983991995997999, 'recall': 0.9477434679334917, 'f1-score': 0.8666847678522944, 'support': 1684}, 'DebtInstrumentFaceAmount': {'precision': 0.6646489104116223, 'recall': 0.8157503714710252, 'f1-score': 0.7324883255503669, 'support': 1346}, 'DebtInstrumentInterestRateStatedPercentage': {'precision': 0.7903669724770642, 'recall': 0.9359043997827268, 'f1-score': 0.8570007460830639, 'support': 1841}, 'LineOfCreditFacilityMaximumBorrowingCapacity': {'precision': 0.6617900172117039, 'recall': 0.9095209934949734, 'f1-score': 0.7661270236612702, 'support': 1691}, 'micro avg': {'precision': 0.730226854690374, 'recall': 0.9074977141115513, 'f1-score': 0.809268193245906, 'support': 6562}, 'macro avg': {'precision': 0.7288012749250476, 'recall': 0.9022298081705543, 'f1-score': 0.8055752157867488, 'support': 6562}, 'weighted avg': {'precision': 0.7335071930776247, 'recall': 0.9074977141115513, 'f1-score': 0.8105281325516894, 'support': 6562}}
0.0012	6.0565	4500	0.0018	0.9996	0.7783	0.8785	0.8254	{'DebtInstrumentBasisSpreadOnVariableRate1': {'precision': 0.798810703666997, 'recall': 0.9572446555819477, 'f1-score': 0.8708806050783361, 'support': 1684}, 'DebtInstrumentFaceAmount': {'precision': 0.72812291249165, 'recall': 0.8098068350668648, 'f1-score': 0.7667956384101302, 'support': 1346}, 'DebtInstrumentInterestRateStatedPercentage': {'precision': 0.8257650542941757, 'recall': 0.908745247148289, 'f1-score': 0.865270235324541, 'support': 1841}, 'LineOfCreditFacilityMaximumBorrowingCapacity': {'precision': 0.744908896034298, 'recall': 0.8219988172678888, 'f1-score': 0.7815574922687659, 'support': 1691}, 'micro avg': {'precision': 0.7783178074794114, 'recall': 0.8785431270953977, 'f1-score': 0.8253990980027203, 'support': 6562}, 'macro avg': {'precision': 0.7744018916217801, 'recall': 0.8744488887662476, 'f1-score': 0.8211259927704433, 'support': 6562}, 'weighted avg': {'precision': 0.777983095601731, 'recall': 0.8785431270953977, 'f1-score': 0.8249384472585973, 'support': 6562}}
0.0011	6.7295	5000	0.0022	0.9995	0.7162	0.9163	0.8040	{'DebtInstrumentBasisSpreadOnVariableRate1': {'precision': 0.7986980470706059, 'recall': 0.9471496437054632, 'f1-score': 0.8666123336049986, 'support': 1684}, 'DebtInstrumentFaceAmount': {'precision': 0.6109375, 'recall': 0.8714710252600297, 'f1-score': 0.7183098591549296, 'support': 1346}, 'DebtInstrumentInterestRateStatedPercentage': {'precision': 0.7634455618714473, 'recall': 0.9483976099945681, 'f1-score': 0.8459302325581395, 'support': 1841}, 'LineOfCreditFacilityMaximumBorrowingCapacity': {'precision': 0.6838503649635036, 'recall': 0.8864577173270254, 'f1-score': 0.7720834406386814, 'support': 1691}, 'micro avg': {'precision': 0.7161743687470223, 'recall': 0.9163364827796403, 'f1-score': 0.8039844899050675, 'support': 6562}, 'macro avg': {'precision': 0.7142328684763892, 'recall': 0.9133689990717716, 'f1-score': 0.8007339664891873, 'support': 6562}, 'weighted avg': {'precision': 0.7206985115552452, 'recall': 0.9163364827796403, 'f1-score': 0.8060303103433248, 'support': 6562}}
0.0007	7.4024	5500	0.0021	0.9995	0.7513	0.9060	0.8214	{'DebtInstrumentBasisSpreadOnVariableRate1': {'precision': 0.8039314516129032, 'recall': 0.9471496437054632, 'f1-score': 0.8696837513631407, 'support': 1684}, 'DebtInstrumentFaceAmount': {'precision': 0.6957605985037406, 'recall': 0.8291233283803864, 'f1-score': 0.7566101694915255, 'support': 1346}, 'DebtInstrumentInterestRateStatedPercentage': {'precision': 0.7822182308037718, 'recall': 0.9462248777838131, 'f1-score': 0.8564405113077679, 'support': 1841}, 'LineOfCreditFacilityMaximumBorrowingCapacity': {'precision': 0.7111534795042898, 'recall': 0.8823181549379066, 'f1-score': 0.7875428873053576, 'support': 1691}, 'micro avg': {'precision': 0.7512953367875648, 'recall': 0.9059737884791222, 'f1-score': 0.8214162348877374, 'support': 6562}, 'macro avg': {'precision': 0.7482659401061764, 'recall': 0.9012040012018924, 'f1-score': 0.817569329866948, 'support': 6562}, 'weighted avg': {'precision': 0.7517431616662088, 'recall': 0.9059737884791222, 'f1-score': 0.8216072430938864, 'support': 6562}}
0.0007	8.0754	6000	0.0024	0.9995	0.7342	0.9159	0.8150	{'DebtInstrumentBasisSpreadOnVariableRate1': {'precision': 0.791095890410959, 'recall': 0.9602137767220903, 'f1-score': 0.8674892703862661, 'support': 1684}, 'DebtInstrumentFaceAmount': {'precision': 0.6338946224877784, 'recall': 0.8670133729569094, 'f1-score': 0.7323501725760904, 'support': 1346}, 'DebtInstrumentInterestRateStatedPercentage': {'precision': 0.774822695035461, 'recall': 0.9494839760999457, 'f1-score': 0.8533072980229436, 'support': 1841}, 'LineOfCreditFacilityMaximumBorrowingCapacity': {'precision': 0.7227383863080684, 'recall': 0.8740390301596689, 'f1-score': 0.791220556745182, 'support': 1691}, 'micro avg': {'precision': 0.7341803078426582, 'recall': 0.9158793050899117, 'f1-score': 0.8150257662055873, 'support': 6562}, 'macro avg': {'precision': 0.7306378985605667, 'recall': 0.9126875389846535, 'f1-score': 0.8110918244326205, 'support': 6562}, 'weighted avg': {'precision': 0.7366697400377676, 'recall': 0.9158793050899117, 'f1-score': 0.8161365377528546, 'support': 6562}}

Framework versions

Transformers 4.46.3
Pytorch 2.5.1+cu124
Datasets 3.2.0
Tokenizers 0.20.3

How to use this model

Use a pipeline as a high-level helper

from transformers import pipeline

pipe = pipeline("token-classification", model="martindevoto/finer_ner_finetuning_0130")

Load model directly

from transformers import AutoTokenizer, AutoModelForTokenClassification

tokenizer = AutoTokenizer.from_pretrained("martindevoto/finer_ner_finetuning_0130") model = AutoModelForTokenClassification.from_pretrained("martindevoto/finer_ner_finetuning_0130")