IELTS Task 2 Band Scorer (DeBERTa-v3 + CORAL)
Predicts an IELTS Writing Task 2 band (4-8) from a prompt and an essay.
Ordinal regression rather than classification: predicting band 5 for a band 7
essay is a worse error than predicting band 6, and a classifier does not know
that.
Results
Held-out test set, 878 essays:
| Metric | Score |
|---|
| QWK | 0.605 |
| MAE | 0.76 bands |
| RMSE | 1.11 |
| Exact accuracy | 43.3% |
| Within +/-1 band | 84.4% |
With n=878, QWK carries roughly a +/-0.04 interval. Essay length alone
correlates with band at r=0.28, so the model is well clear of a trivial
baseline.
The head
CORAL (Cao et al., 2020): one shared quality score, plus one learnable
boundary per band cut-off.
1self.score = nn.Linear(hidden_size, 1, bias=False)
2self.boundaries = nn.Parameter(torch.linspace(2.0, -2.0, num_classes - 1))
3
4logits = self.score(x) + self.boundaries
Sharing one weight vector across boundaries is what guarantees the cumulative
probabilities come out in descending order, which is what makes "count how many
exceed their threshold" a valid way to pick a band. Independent weights per
boundary produce non-monotonic probabilities and an ill-defined counting rule.
Thresholds are calibrated on validation data to maximise QWK; the fitted
cut-offs ship as coral_cutoffs.npy.
Usage
1import json, sys
2import numpy as np, torch
3from huggingface_hub import snapshot_download
4from safetensors.torch import load_file
5from transformers import AutoTokenizer
6
7path = snapshot_download("sieun1234/ielts-band-coral")
8sys.path.append(path)
9from model import Deberta # class definition ships with the weights
10
11cfg = json.load(open(f"{path}/head_config.json"))
12cutoffs = np.load(f"{path}/coral_cutoffs.npy")
13id_to_band = {int(k): float(v) for k, v in cfg["id_to_band"].items()}
14
15tokenizer = AutoTokenizer.from_pretrained(path)
16model = Deberta(cfg["model_name"], cfg["num_classes"],
17 cfg["dropout_rate"], cfg["n_dropout_samples"])
18model.load_state_dict(load_file(f"{path}/model.safetensors"))
19model.eval()
20
21text = f"{topic} {tokenizer.sep_token} {essay}"
22enc = tokenizer(text, return_tensors="pt", truncation=True,
23 max_length=cfg["max_length"])
24
25with torch.no_grad():
26 logits = model(**enc)["logits"][0]
27
28probs = torch.sigmoid(logits).numpy()
29band = id_to_band[int((probs > cutoffs).sum())]
Training data
Two sources, ~8,600 essays:
- chillies/IELTS-writing-task-2-evaluation
- GPT-generated comments excluded as label noise.
- Cambridge IELTS past papers (private, not redistributed), digitised via OCR.
Leakage control
Two leaks were found and fixed before the reported results:
- Duplicate essays - 82 of 793 essays in the secondary source appeared twice,
and the split happened after merging, so copies landed on both sides.
- Shared prompts - 793 essays covered only 275 unique questions, so a random
split let the model learn "this prompt scores ~6.5" rather than reading the
writing.
Fixed by deduplicating and splitting with StratifiedGroupKFold grouped by
topic. Test QWK moved 0.610 -> 0.605, i.e. within noise - the leak was not
inflating the headline number. What the fix buys is a validation score that can
be trusted for model selection and threshold calibration.
Limitations
- Whole bands only (4-8). Half bands were collapsed because the tails were
too sparse - one essay at band 3.0, two below 4.0.
- No per-criterion scores. The training data had none, so this model cannot
produce Task Response / Coherence / Lexical / Grammar sub-scores.
- Provenance of the public dataset is undocumented - it ships without a
dataset card, so how its band scores were produced cannot be verified.
- Not a certified scoring system. Typical error is about +/-1 band.
Intended use
Research and educational use. Not a substitute for official IELTS assessment,
and not for high-stakes decisions about individuals.
Citation
Cao, W., Mirjalili, V., & Raschka, S. (2020). Rank consistent ordinal
regression for neural networks with application to age estimation.
Pattern Recognition Letters.
