Views
No views yet
1python -m pip install --upgrade \
2 transformers>=4.42 torch \
3 numpy scipy huggingface_hub>=0.231from transformers import AutoTokenizer, AutoModelForSequenceClassification
2import json
3from evaluate import evaluate_candidate # comes with this repo
4
5MODEL_ID = "ahmetsezginn/cefr-roberta-irt-hybrid"
6
7tok = AutoTokenizer.from_pretrained(MODEL_ID)
8clf = AutoModelForSequenceClassification.from_pretrained(MODEL_ID)
9clf.eval() # no grad
10
11qbank = json.load(open("irt_qbank.json")) # download once (see below)
12
13# 10 answers: "passageID_questionID" ➜ "a|b|c|d"
14answers = {
15 "57_3": "b", "57_4": "d", "67_1": "a", "67_2": "c", "67_3": "b",
16 "81_5": "d", "81_6": "a", "92_2": "c", "104_1": "b", "104_4": "a"
17}
18
19result = evaluate_candidate(answers, qbank, tok, clf)
20print(json.dumps(result, indent=2))1{
2 "theta": -0.73,
3 "irt_level": "B1",
4 "clf_level": "B2",
5 "final_level": "B2",
6 "skills": {
7 "Inference": {"n":4, "accuracy":0.25, "status":"weak"},
8 "Detail": {"n":3, "accuracy":0.67, "status":"medium"},
9 "Vocab": {"n":3, "accuracy":1.00, "status":"strong"}
10 }
11}| Path | Description |
|---|---|
config.json, model.safetensors | DistilRoBERTa classifier (4 labels) |
tokenizer_config.json, vocab.json, merges.txt, special_tokens_map.json | Tokenizer |
irt_qbank.json | 795 MCQ items → disc, diff, answer (+optional skill) |
evaluate.py | Single helper: θ MLE + ensemble logic |
README.md | (this file) |
irt_qbank.json schema1{
2 "57_3": {
3 "disc": 0.87, // discrimination (a_i)
4 "diff": 64.1, // raw difficulty (0–100) – internally z-scored
5 "answer": "b",
6 "skill": "Inference" // optional micro-skill tag
7 },
8 "...": { ... }
9}ⅰ Where do IDs come from?
Each question is labeled"<passageID>_<qID>", e.g. passage 57 Q3 →"57_3".
ID = option pairs.answers = { "57_3":"b", ... } as above.evaluate_candidate(answers, qbank, tok, clf) – done!tok/clf, evaluate_candidate falls back to IRT-only scoring.passageID_questionID string.disc & diff using an IRT library (e.g. mirt / py-irt).irt_qbank.json.1python - <<'PY'
2from transformers import pipeline
3clf = pipeline("text-classification",
4 model="ahmetsezginn/cefr-roberta-irt-hybrid",
5 top_k=None)
6text = open("sample_passage.txt").read()
7print(clf(text)) # e.g. [{'label': 'C1', 'score': 0.72}, ...]
8PY| Question | Answer |
|---|---|
| GPU required? | No. Model is 82 M params; CPU inference ≈ 1 s per passage. |
| Can I ignore the transformer and use IRT only? | Yes – skip tok/clf arguments. |
What if skill is missing? | Those questions appear as "Unknown" in the skills report. |
| Licence? | Research / CC‑BY‑NC 4.0 – see LICENSE. |