Views
No views yet
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("meetkai/functionary-small-v3.1")
4model = AutoModelForCausalLM.from_pretrained("meetkai/functionary-small-v3.1", device_map="auto", trust_remote_code=True)
5
6tools = [
7 {
8 "type": "function",
9 "function": {
10 "name": "get_current_weather",
11 "description": "Get the current weather",
12 "parameters": {
13 "type": "object",
14 "properties": {
15 "location": {
16 "type": "string",
17 "description": "The city and state, e.g. San Francisco, CA"
18 }
19 },
20 "required": ["location"]
21 }
22 }
23 }
24]
25messages = [{"role": "user", "content": "What is the weather in Istanbul and Singapore respectively?"}]
26
27final_prompt = tokenizer.apply_chat_template(messages, tools, add_generation_prompt=True, tokenize=False)
28inputs = tokenizer(final_prompt, return_tensors="pt").to("cuda")
29pred = model.generate_tool_use(**inputs, max_new_tokens=128, tokenizer=tokenizer)
30print(tokenizer.decode(pred.cpu()[0]))1from openai import OpenAI
2
3client = OpenAI(base_url="http://localhost:8000/v1", api_key="functionary")
4
5client.chat.completions.create(
6 model="path/to/functionary/model/",
7 messages=[{"role": "user",
8 "content": "What is the weather for Istanbul?"}
9 ],
10 tools=[{
11 "type": "function",
12 "function": {
13 "name": "get_current_weather",
14 "description": "Get the current weather",
15 "parameters": {
16 "type": "object",
17 "properties": {
18 "location": {
19 "type": "string",
20 "description": "The city and state, e.g. San Francisco, CA"
21 }
22 },
23 "required": ["location"]
24 }
25 }
26 }],
27 tool_choice="auto"
28)<|start_header_id|>system<|end_header_id|>
Environment: ipython
Cutting Knowledge Date: December 2023
You have access to the following functions:
Use the function 'get_current_weather' to 'Get the current weather'
{"name": "get_current_weather", "description": "Get the current weather", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}},"required": ["location"]}}
Think very carefully before calling functions.
If a you choose to call a function ONLY reply in the following format:
<{start_tag}={function_name}>{parameters}{end_tag}
where
start_tag => `<function`
parameters => a JSON dict with the function argument name as key and function argument value as value.
end_tag => `</function>`
Here is an example,
<function=example_function_name>{"example_name": "example_value"}</function>
Reminder:
- If looking for real time information use relevant functions before falling back to brave_search
- Function calls MUST follow the specified format, start with <function= and end with </function>
- Required parameters MUST be specified
- Only call one function at a time
- Put the entire function call reply on one line
<|eot_id|><|start_header_id|>user<|end_header_id|>
What is the weather for Istanbul?