Views
No views yet
1!pip install transformers accelerate bitsandbytes>0.37.0
2!pip install peft1from peft import AutoPeftModelForCausalLM
2from transformers import AutoTokenizer,AutoModelForCausalLM
3from peft import PeftModel, PeftConfig, get_peft_model
4import torch
5
6base_model = AutoModelForCausalLM.from_pretrained("SwastikM/Meta-Llama-3-8B-Instruct_bitsandbytes_4bit",device_map="auto")
7model = PeftModel.from_pretrained(base_model, "SwastikM/Meta-Llama3-8B-Chat-Adapter")
8tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
9
10model = model.to("cuda")
11model.eval()1x1 = {"role": "system", "content": """You are a APIGen Function Calling Tool. You will br provided with a user query and associated tools for answering the query.
2 query (string): The query or problem statement.
3 tools (array): An array of available tools that can be used to solve the query.
4 Each tool is represented as an object with the following properties:
5 name (string): The name of the tool.
6 description (string): A brief description of what the tool does.
7 parameters (object): An object representing the parameters required by the tool.
8 Each parameter is represented as a key-value pair, where the key is the parameter name and the value is an object with the following properties:
9 type (string): The data type of the parameter (e.g., "int", "float", "list").
10 description (string): A brief description of the parameter.
11 required (boolean): Indicates whether the parameter is required or optional.
12 You will provide the Answer array.
13 Answers array provides the specific tool and arguments used to generate each answer."""}
14x2 = {"role": "user", "content": None}
15x3 = {"role": "assistant", "content": None}
16user_template = 'Query: {Q} Tools: {T}'
17response_template = '{A}'
18Q = "Where can I find live giveaways for beta access and games?"
19T = """[{"name": "live_giveaways_by_type", "description": "Retrieve live giveaways from the GamerPower API based on the specified type.", "parameters": {"type": {"description": "The type of giveaways to retrieve (e.g., game, loot, beta).", "type": "str", "default": "game"}}}]"""
20
21
22x2['content'] = f'{user_template.format(Q=Q,T=T)}'
23prompts = [x1,x2]
24input_ids = tokenizer.apply_chat_template(
25 prompts,
26 add_generation_prompt=True,
27 return_tensors="pt"
28).to(model.device)
29
30terminators = [
31 tokenizer.eos_token_id,
32 tokenizer.convert_tokens_to_ids("<|eot_id|>")
33]
34
35outputs = model.generate(
36 input_ids,
37 max_new_tokens=256,
38 eos_token_id=terminators
39)
40
41response = outputs[0][input_ids.shape[-1]:]
42print(tokenizer.decode(response, skip_special_tokens=True))| Model | Total Size | Training Using Adam |
|---|---|---|
| Base Model | 28.21 GB | 56.42 GB |
| 4bitQuantized+PEFT | 5.21 GB | 13 GB |
instruction column of 20,00 randomly shuffled data.