Views
No views yet
| Revision | Model Size | Group Size | w_bit |
|---|---|---|---|
main | ~6.13 GB | 32 | 4 |
gs_2_4bit | ~x.xx GB | 2 | 4 |
gs_4_4bit | ~x.xx GB | 4 | 4 |
gs_8_4bit | ~x.xx GB | 8 | 4 |
gs_16_4bit | ~6.68 GB | 16 | 4 |
gs_64_4bit | ~x.xx GB | 64 | 4 |
gs_128_4bit | ~x.xx GB | 128 | 4 |
gs_512_4bit | ~x.xx GB | 512 | 4 |
role, content and tool_calls fields. This enables the model to be able to generate tool calls.1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("meetkai/functionary-small-v2.5", trust_remote_code=True)
4model = AutoModelForCausalLM.from_pretrained("meetkai/functionary-small-v2.5", device_map="auto", trust_remote_code=True)
5
6tools = [
7 {
8 "type": "function",
9 "function": {
10 "name": "get_current_weather",
11 "description": "Get the current weather",
12 "parameters": {
13 "type": "object",
14 "properties": {
15 "location": {
16 "type": "string",
17 "description": "The city and state, e.g. San Francisco, CA"
18 }
19 },
20 "required": ["location"]
21 }
22 }
23 }
24]
25messages = [{"role": "user", "content": "What is the weather in Istanbul and Singapore respectively?"}]
26
27final_prompt = tokenizer.apply_chat_template(messages, tools, add_generation_prompt=True, tokenize=False)
28tokenizer.padding_side = "left"
29inputs = tokenizer(final_prompt, return_tensors="pt").to("cuda")
30pred = model.generate_tool_use(**inputs, max_new_tokens=128, tokenizer=tokenizer)
31print(tokenizer.decode(pred.cpu()[0]))1from openai import OpenAI
2
3client = OpenAI(base_url="http://localhost:8000/v1", api_key="functionary")
4
5client.chat.completions.create(
6 model="path/to/functionary/model/",
7 messages=[{"role": "user",
8 "content": "What is the weather for Istanbul?"}
9 ],
10 tools=[{
11 "type": "function",
12 "function": {
13 "name": "get_current_weather",
14 "description": "Get the current weather",
15 "parameters": {
16 "type": "object",
17 "properties": {
18 "location": {
19 "type": "string",
20 "description": "The city and state, e.g. San Francisco, CA"
21 }
22 },
23 "required": ["location"]
24 }
25 }
26 }],
27 tool_choice="auto"
28)<|start_header_id|>system<|end_header_id|>
You are capable of executing available function(s) if required.
Only execute function(s) when absolutely necessary.
Ask for the required input to:recipient==all
Use JSON for function arguments.
Respond in this format:
>>>${recipient}
${content}
Available functions:
// Supported function definitions that should be called when necessary.
namespace functions {
// Get the current weather
type get_current_weather = (_: {
// The city and state, e.g. San Francisco, CA
location: string,
}) => any;
} // namespace functions<|eot_id|><|start_header_id|>user<|end_header_id|>
What is the weather for Istanbul?