A minimalistic instruction model with an already good analysed and pretrained encoder like BERT.
So we can research the
Bertology with instruction-tuned models,
look at the attention and investigate
what happens to BERT embeddings during fine-tuning.
The training code is released at the
instructionBERT repository.
We used the Huggingface API for
warm-starting BertGeneration with
Encoder-Decoder-Models for this purpose.
1from transformers import AutoTokenizer, EncoderDecoderModel
2# load the fine-tuned seq2seq model and corresponding tokenizer
3model_name = "Bachstelze/instructionBERT"
4model = EncoderDecoderModel.from_pretrained(model_name)
5tokenizer = AutoTokenizer.from_pretrained(model_name)
6input = "Write a poem about love, peace and pancake."
7input_ids = tokenizer(input, return_tensors="pt").input_ids
8output_ids = model.generate(input_ids, max_new_tokens=200)
9print(tokenizer.decode(output_ids[0]))
InstructionBERT is intended for research purposes. The model-generated text should be treated as a starting point rather than a definitive solution for potential use cases. Users should be cautious when employing these models in their applications.
