Views
No views yet
1tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b")
2tokenizer.truncation_side = "left"
3tokenizer.padding_side = "left"
4base_model = AutoModelForCausalLM.from_pretrained(
5 "huggyllama/llama-7b",
6 load_in_4bit=True,
7 torch_dtype=torch.float16,
8 device_map="auto",
9 max_memory= {i: '40000MB' for i in range(torch.cuda.device_count())},
10 quantization_config=BitsAndBytesConfig(
11 load_in_4bit=True,
12 bnb_4bit_compute_dtype=torch.float16,
13 bnb_4bit_quant_type='nf4'
14 ),
15)
16model = PeftModel.from_pretrained(base_model, "abaheti95/a_lol_kl_good_prioirty_qlora_hh")
17
18prefix = "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.\n### Human: How would I go about making homemade chewing gum? ### Assistant: Well, it\u2019s pretty easy, but you can either make it a bit better or more interesting if you use xanthan gum, which is a commonly used thickener. Xanthan gum is added to food products as a thickener, gelling agent, emulsifier, and stabilizer, which means that it allows sugar syrups to form gels. If you can get hold of some, you could use it to make really interesting gum! ### Human: Okay what do I do when I have the xanthan gum? What else do I need to add? ### Assistant: Well, the important ingredient is the gums. You can find many different types at your local pharmacy, which should be able to help you figure out what you need. You will need: 1) a mixer, 2) sugar, 3) xanthan gum, 4) flavoring, 5) gum base, and 6) a dehydrator, which is a special oven that dries food. ### Human: Why would I need a dehydrator to make gum? ### Assistant:"
19
20current_batch_prefixes_inputs = tokenizer([prefix], max_length = 768 - 128,truncation = True,add_special_tokens=True, padding = True, return_tensors="pt").to(model.device)
21# 835 = "###"
22predicted_sents = model.generate(**current_batch_prefixes_inputs, max_new_tokens=128,pad_token_id=tokenizer.pad_token_id,num_beams=1,do_sample=False,num_return_sequences = 1, eos_token_id=[835, tokenizer.eos_token_id])
23repsonse_tokens = predicted_sents[:, current_batch_prefixes_inputs['input_ids'].shape[-1]:]
24responses = tokenizer.batch_decode(repsonse_tokens, skip_special_tokens=True)
25# Normalize responses
26responses_normalized = [resp.split("\n Human:")[0].split("\nHuman:")[0].split("\n### Human")[0].strip() for resp in responses]
27responses_normalized = [resp.replace("###", "").strip() if resp.endswith("###") else resp.strip() for resp in responses_normalized]harmless_base_eval_results.jsonl, helpful_base_eval_results.jsonl, helpful_online_eval_results.jsonl and helpful_rejection_eval_results.jsonl.bitsandbytes quantization config was used during training: