milapp857/

distill-bert-txt-classification

This contains the state dict of a distilled BERT model.

The teacher BERT model was fine-tuned on the BBC's Text classification dataset: https://huggingface.co/milapp857/bert-finetuned-txt-classification

The distillation method used is response-based knoweldge distillation.

Note that the student model itself isn't pre-trained in this initial version, since my intention was learning knwoledge distillation only.

The model class for this picke file and training code can be found here: https://github.com/lavanderhoney/bert-distillation-quantization/blob/main/distillation.py