A minimalistic instruction model with an already good analysed and pretrained encoder like BERT.
So we can research the
Bertology with instruction-tuned models,
look at the attention and investigate
what happens to BERT embeddings during fine-tuning.
The trainings code is released at the
instructionBERT repository.
We used the Huggingface API for
warm-starting BertGeneration with
Encoder-Decoder-Models for this purpose.
1from transformers import AutoTokenizer, EncoderDecoderModel
2
3# load the fine-tuned seq2seq model and corresponding tokenizer
4model_name = "Bachstelze/instructionBERTtest"
5model = EncoderDecoderModel.from_pretrained(model_name)
6tokenizer = AutoTokenizer.from_pretrained(model_name)
7
8input = "Write a poem about love, peace and pancake."
9input_ids = tokenizer(input, return_tensors="pt").input_ids
10output_ids = model.generate(input_ids, max_new_tokens=200)
11print(tokenizer.decode(output_ids[0]))
InstructionBERT is intended for research purposes. The model-generated text should be treated as a starting point rather than a definitive solution for potential use cases. Users should be cautious when employing these models in their applications.
