Views
No views yet
role, content and tool_calls fields. This enables the users to read the function-calling output of the model easily.1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4tokenizer = AutoTokenizer.from_pretrained("meetkai/functionary-medium-v3.0")
5model = AutoModelForCausalLM.from_pretrained(
6 "meetkai/functionary-medium-v3.0",
7 device_map="auto",
8 trust_remote_code=True,
9 torch_dtype=torch.bfloat16,
10)
11
12tools = [
13 {
14 "type": "function",
15 "function": {
16 "name": "get_current_weather",
17 "description": "Get the current weather",
18 "parameters": {
19 "type": "object",
20 "properties": {
21 "location": {
22 "type": "string",
23 "description": "The city and state, e.g. San Francisco, CA"
24 }
25 },
26 "required": ["location"]
27 }
28 }
29 }
30]
31messages = [{"role": "user", "content": "What is the weather in Istanbul and Singapore respectively?"}]
32
33final_prompt = tokenizer.apply_chat_template(messages, tools, add_generation_prompt=True, tokenize=False)
34inputs = tokenizer(final_prompt, return_tensors="pt").to("cuda")
35pred = model.generate_tool_use(**inputs, max_new_tokens=128, tokenizer=tokenizer)
36print(tokenizer.decode(pred.cpu()[0]))1from openai import OpenAI
2
3client = OpenAI(base_url="http://localhost:8000/v1", api_key="functionary")
4
5client.chat.completions.create(
6 model="path/to/functionary/model/",
7 messages=[{"role": "user",
8 "content": "What is the weather for Istanbul?"}
9 ],
10 tools=[{
11 "type": "function",
12 "function": {
13 "name": "get_current_weather",
14 "description": "Get the current weather",
15 "parameters": {
16 "type": "object",
17 "properties": {
18 "location": {
19 "type": "string",
20 "description": "The city and state, e.g. San Francisco, CA"
21 }
22 },
23 "required": ["location"]
24 }
25 }
26 }],
27 tool_choice="auto"
28)<|start_header_id|>system<|end_header_id|>
You are capable of executing available function(s) if required.
Only execute function(s) when absolutely necessary.
Ask for the required input to:recipient==all
Use JSON for function arguments.
Respond in this format:
>>>${recipient}
${content}
Available functions:
// Supported function definitions that should be called when necessary.
namespace functions {
// Get the current weather
type get_current_weather = (_: {
// The city and state, e.g. San Francisco, CA
location: string,
}) => any;
} // namespace functions<|eot_id|><|start_header_id|>user<|end_header_id|>
What is the weather for Istanbul?