QLoRA adapter (r=16, alpha=32, all attention and MLP projections) that rates
the CEFR difficulty of a single English sentence on the 1 to 6 scale
(A1 to C2). The completion is one bare digit token, so classification is a
single forward pass with an argmax over six token logits: no generation, no
output parsing.
Prompt format:
Rate the CEFR difficulty of this English sentence on a scale of 1 (A1, easiest) to 6 (C2, hardest).
Sentence: {sentence}
Level:
Benchmark vs GPT-4o-mini few-shot
Identical held-out test set (CEFR-SP public portion, n=1460, split checksum
e45e1389a015), single evaluation pass for both models.
Metric
gpt-4o-mini few-shot
this adapter
Exact accuracy
40.75%
62.74%
Within one level
85.0%
99.04%
Macro F1
0.356
0.4435
Quadratic weighted kappa
0.6479
0.8076
Cost per 1k requests
$0.1304 (API)
$0.0055 (L4 at $0.70/h)
Checkpoint 1347 (epoch 3 of 3) selected by validation macro F1 (0.5132).
McNemar exact test on paired correctness: p = 2.3e-33.
Ablation adapter: os500/
The os500/ folder holds an adapter trained identically except A1 and C2
training rows were oversampled to 500 each. Test accuracy 64.11% and macro
F1 0.4489, but A1 F1 stayed exactly 0.00 and C2 dropped to 0.18 (McNemar
vs the base recipe: p = 0.09, not significant). Duplicating 47 unique
sentences adds signal strength, not information. The root adapter remains
the final model, selected by validation macro F1 (0.5132 vs 0.5023). Raw
prediction files for both runs are under benchmark/.
Known limitations
Training data contains only 47 A1 and 91 C2 sentences. The adapter
under-predicts both extremes (test F1: A1 0.00, C2 0.20) while beating the
baseline by wide margins on A2, B1, B2, and C1. Judgments follow the
stricter of the two expert annotators in CEFR-SP by design.
Trained on the public portion of CEFR-SP (Wiki-Auto CC BY-SA 3.0, SCoRE
CC BY-NC-SA 4.0): treat this adapter as research and non-commercial use.
