Views
No views yet

[INST] and [\INST] tokens. The very first instruction should begin with a begin of sentence id. The next instructions should not. The assistant generation will be ended by the end-of-sentence token id.1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3device = "cuda" # the device to load the model onto
4
5model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
6tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
7
8text = "<s>[INST] Below is a conversation transcript [/INST]"
9"Your credit card has been stolen, and you need to contact us to resolve the issue. We will help you protect your information and prevent further fraud.</s> "
10"[INST] Analyze the conversation and determine if it's fraudulent or legitimate. [/INST]"
11
12encodeds = tokenizer(text, return_tensors="pt", add_special_tokens=False)
13
14model_inputs = encodeds.to(device)
15model.to(device)
16
17generated_ids = model.generate(**model_inputs, max_new_tokens=1000, do_sample=True)
18decoded = tokenizer.batch_decode(generated_ids)
19print(decoded[0])