tinybert student fine-tuned on SarcOji with sarcasm-aware emoji embeddings injected directly into the token-embedding matrix, distilled from Vandita/Bert-finetuned-Sarc.

SarcOjiTest2 is ~75/25 negative-skewed, so its accuracy is not comparable to Test1's — MCC and ROC-AUC are the meaningful columns there.

Hinton et al. (2015): L = α·T²·KL(teacher‖student) + (1−α)·CE, with T = 3, α = 0.5. The T² factor is applied, so soft-target gradients do not shrink as T rises.

1444 emoji tokens added to the tokenizer (vocab 30522 → 31959). Each new row of the 312-d embedding matrix was initialised from a 300-d sarcasm-aware emoji vector using concat-mean. Remaining dimensions hold the mean of the original vocabulary embeddings rather than zeros, so new tokens are not anchored to the origin.

Because the emoji are part of the vocabulary, this is a standard HuggingFace model — no custom architecture, no custom loader.

SarcOji, 80:20 stratified split. Duplicates retained; identical texts were kept on one side of the split so no text appears in both train and validation. Evaluated on SarcOjiTest1 and SarcOjiTest2 in full.