MetaLingo-Indirect-Metaphor
This model is a fine-tuned version of Microsoft DeBERTa-v3-large for binary indirect metaphor detection in English at token level, trained on the VU Amsterdam Metaphor Corpus (VUAMC). It is produced by a two-stage knowledge distillation pipeline: Stage 1 transfers knowledge from an auxiliary teacher model via soft-label distillation on an out-of-domain reference corpus (BE06, ~954K tokens), and Stage 2 fine-tunes on the VUAMC gold standard with hard labels. Annotation follows MIPVU (Steen et al. 2010); only indirect metaphors (metaphor_type == "met") are labeled positive.
Model description
- Base model: microsoft/deberta-v3-large
- Task: Token classification (binary metaphor detection)
- Training unit: Sentences, word-level labels
- Teacher model (Stage 1): An auxiliary DeBERTa-v3-large indirect-metaphor classifier used to produce soft labels for Stage 1 distillation; not released as a standalone model.
Two-stage distillation pipeline
Stage 1 — Distillation on BE06 (~954K tokens).
The student is pre-trained with KL-divergence distillation (temperature = 2.0) against soft labels produced by an auxiliary DeBERTa-v3-large teacher specialized for indirect-metaphor detection. All Stage 1 training happens on BE06, a British-English reference corpus that is completely independent of VUAMC (different source corpus, different documents, zero overlap). This stage exposes the student to a much larger and more varied token distribution than VUAMC alone provides, giving it a strong initialization for metaphor detection before it ever sees a VUAMC sentence.
Stage 2 — Gold fine-tuning on VUAMC (90 train docs).
Starting from the Stage 1 checkpoint, the model is fine-tuned with hard-label cross-entropy on the 90-document official VUAMC training split (10% randomly held out as dev, seed=42), then evaluated once on the 27-document official test split.
Because Stage 1 never touches VUAMC at all, the official 27-document test split remains completely unseen until the single final evaluation — the test set is never contaminated.
Training & evaluation data
- Dataset: VUAMC (VU Amsterdam Metaphor Corpus), covering four registers: News, Academic, Fiction, Conversation.
- Split: Official NAACL FLP partition (Leong et al. 2018): 90 train documents / 27 test documents. A random 10% of training sentences is held out as dev set (seed=42).
- Test set: 27 documents, 4,080 sentences, 57,811 tokens, 6,697 gold metaphor tokens.
Training hyperparameters
| Stage 1 (Distillation on BE06) | Stage 2 (Fine-tune on VUAMC) |
|---|
| Loss | KL Divergence (T=2) | Cross-Entropy |
| Epochs | 3 | 3 |
| Learning rate | 2e-5 | 6e-6 |
| Effective batch size | 32 (8×4) | 16 (8×2) |
| Max length | 192 | 192 |
| Warmup ratio | 0.1 | 0.1 |
Results
Evaluated on the official 27-document test split (NAACL FLP 2018):
| Metric | Value |
|---|
| F1 | 82.29 |
| Precision | 85.26 |
| Recall | 79.53 |
| Accuracy | 96.04 |
By genre
Genre is determined by BNC document ID per the NAACL FLP 2018 shared-task split (News: a*; Academic: b17, clw, cty, ecv; Fiction: bmw, ccw, faj; Conversation: k*).
| Genre | Docs | Tokens | Metaphors | Precision | Recall | F1 |
|---|
| Academic | 4 | 14,208 | 2,425 | 90.90 | 82.80 | 86.66 |
| News | 14 | 13,405 | 1,953 | 86.94 | 77.73 | 82.08 |
| Fiction | 3 | 12,764 | 1,079 | 80.72 | 83.04 | 81.86 |
| Conversation | 6 | 17,434 | 1,240 | 76.48 | 72.90 | 74.65 |
By part of speech (Universal POS)
Tags with no gold metaphors in the test set (CCONJ, INTJ, NUM, SPACE, SYM, X) are omitted.
| POS | Tokens | Metaphors | Precision | Recall | F1 |
|---|
| ADP | 4,959 | 1,922 | 93.62 | 90.89 | 92.24 |
| DET | 4,163 | 213 | 95.77 | 95.77 | 95.77 |
| PRON | 6,545 | 275 | 89.66 | 85.09 | 87.31 |
| SCONJ | 1,299 | 99 | 79.63 | 86.87 | 83.09 |
| VERB | 6,237 | 1,908 | 84.06 | 77.94 | 80.88 |
| PUNCT | 7,627 | 2 | 66.67 | 100.00 | 80.00 |
| ADV | 2,527 | 245 | 85.57 | 70.20 | 77.13 |
| NOUN | 8,630 | 1,383 | 83.19 | 69.78 | 75.89 |
| ADJ | 3,511 | 595 | 81.80 | 68.74 | 74.70 |
| PART | 1,881 | 8 | 57.14 | 50.00 | 53.33 |
| AUX | 4,219 | 10 | 45.45 | 50.00 | 47.62 |
| PROPN | 2,197 | 37 | 57.89 | 29.73 | 39.29 |
Label dictionary
1{
2 "0": "non_metaphor",
3 "1": "metaphor"
4}
Subwords are aligned to words via the tokenizer's word_ids; the first subword of each word is used for prediction.
Usage example
1from transformers import AutoTokenizer, AutoModelForTokenClassification
2import torch
3
4model_path = "tommyleo2077/metalingo-indirect-metaphor" # or local path
5tokenizer = AutoTokenizer.from_pretrained(model_path)
6model = AutoModelForTokenClassification.from_pretrained(model_path)
7model.eval()
8
9words = ["The", "conference", "threw", "a", "spanner", "in", "the", "works", "."]
10inputs = tokenizer(
11 words,
12 is_split_into_words=True,
13 return_tensors="pt",
14 truncation=True,
15 max_length=192,
16)
17word_ids = inputs.word_ids(batch_index=0)
18
19with torch.no_grad():
20 logits = model(**inputs).logits
21preds = logits.argmax(dim=-1)[0].tolist()
22
23word_preds = {}
24for i, wid in enumerate(word_ids):
25 if wid is not None and wid not in word_preds:
26 word_preds[wid] = preds[i]
27
28for i, w in enumerate(words):
29 label = "metaphor" if word_preds.get(i, 0) == 1 else "non_metaphor"
30 print(f"{w}\t{label}")
Citation
Dataset (MIPVU / VUAMC):
1@book{steen2010method,
2 title = {A Method for Linguistic Metaphor Identification: From {MIP} to {MIPVU}},
3 author = {Steen, Gerard and Dorst, Aletta G. and Herrmann, J. Berenike and Kaal, Anna and Krennmayr, Tina and Pasma, Thea},
4 year = {2010},
5 publisher = {John Benjamins}
6}
Train/test split:
1@inproceedings{leong2018vua,
2 title = {A Report on the 2018 {VUA} Metaphor Detection Shared Task},
3 author = {Leong, Chee Wee and Beigman Klebanov, Beata and Shutova, Ekaterina},
4 booktitle = {Proceedings of the Workshop on Figurative Language Processing at NAACL-HLT 2018},
5 year = {2018}
6}
This model:
1@misc{leo2025metalingoindirectmetaphor,
2 title = {metalingo-indirect-metaphor: Two-Stage Knowledge Distillation for Metaphor Detection},
3 author = {Leo, Tommy},
4 year = {2025},
5 howpublished = {\url{https://huggingface.co/tommyleo2077/metalingo-indirect-metaphor}},
6 note = {Contact: 1683619168tl@gmail.com}
7}
License
Apache License 2.0 — see
LICENSE for details.
