Quantization made by Richard Erkhov.
license: other
license_name: katanemo-research
license_link: >-
https://huggingface.co/katanemo/Arch-Function-3B/blob/main/LICENSE
base_model:
The Katanemo Arch-Function collection of large language models (LLMs) is a collection state-of-the-art (SOTA) LLMs specifically designed for function calling tasks. The models are designed to understand complex function signatures, identify required parameters, and produce accurate function call outputs based on natural language prompts. Achieving performance on par with GPT-4, these models set a new benchmark in the domain of function-oriented tasks, making them suitable for scenarios where automated API interaction and function execution is crucial.
Arch-Function is the core LLM used in then open source
Arch Gateway to seamlessly integrate user prompts with developers APIs
Katanemo Arch-Function collection is built on top of the
Qwen 2.5. A blog with technical details leading to our models will be published soon.
We evaluate Katanemo Arch-Function series on the
Berkeley Function-Calling Leaderboard (BFCL). For each model family, we select the one with the highest rank. The results are shwon below:
We use the following example to illustrate how to use our model to perform function calling tasks. Please note that, our model works best with our provided prompt format. It allows us to extract JSON output that is similar to the
function-calling mode of ChatGPT.
1import json
2from typing import Any, Dict, List
3from transformers import AutoModelForCausalLM, AutoTokenizer
4
5model_name = "katanemo/Arch-Function-3B"
6model = AutoModelForCausalLM.from_pretrained(
7 model_name, device_map="auto", torch_dtype="auto", trust_remote_code=True
8)
9tokenizer = AutoTokenizer.from_pretrained(model_name)
10
11# Please use our provided prompt for best performance
12TASK_PROMPT = """
13You are a helpful assistant.
14""".strip()
15
16TOOL_PROMPT = """
17# Tools
18
19You may call one or more functions to assist with the user query.
20
21You are provided with function signatures within <tools></tools> XML tags:
22<tools>
23{tool_text}
24</tools>
25""".strip()
26
27FORMAT_PROMPT = """
28For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
29<tool_call>
30{"name": <function-name>, "arguments": <args-json-object>}
31</tool_call>
32""".strip()
33
34# Define available tools
35get_weather_api = {
36 "type": "function",
37 "function": {
38 "name": "get_weather",
39 "description": "Get the current weather for a location",
40 "parameters": {
41 "type": "object",
42 "properties": {
43 "location": {
44 "type": "str",
45 "description": "The city and state, e.g. San Francisco, New York",
46 },
47 "unit": {
48 "type": "str",
49 "enum": ["celsius", "fahrenheit"],
50 "description": "The unit of temperature to return",
51 },
52 },
53 "required": ["location"],
54 },
55 },
56}
57
58openai_format_tools = [get_weather_api]
59
60
61def convert_tools(tools: List[Dict[str, Any]]):
62 return "\n".join([json.dumps(tool) for tool in tools])
63
64# Helper function to create the system prompt for our model
65def format_prompt(tools: List[Dict[str, Any]]):
66 tool_text = convert_tools(tools)
67
68 return (
69 TASK_PROMPT
70 + "\n\n"
71 + TOOL_PROMPT.format(tool_text=tool_text)
72 + "\n\n"
73 + FORMAT_PROMPT
74 + "\n"
75 )
76
77
78system_prompt = format_prompt(openai_format_tools)
79
80messages = [
81 {"role": "system", "content": system_prompt},
82 {"role": "user", "content": "What is the weather in Seattle?"},
83]
84
85inputs = tokenizer.apply_chat_template(
86 messages, add_generation_prompt=True, return_tensors="pt"
87).to(model.device)
88
89outputs = model.generate(
90 inputs,
91 max_new_tokens=512,
92 do_sample=False,
93 num_return_sequences=1,
94 eos_token_id=tokenizer.eos_token_id,
95)
96
97response = tokenizer.decode(outputs[0][len(inputs[0]) :], skip_special_tokens=True)
98print(response)
1<tool_call>
2{"name": "get_weather", "arguments": {"location": "Seattle"}}
3</tool_call>
1# Suppose we receive the following result from the function:
2get_weather_api_result = {'name': 'get_weather', 'results': {'temperature': '62°', 'unit': 'fahrenheit'}}
3execution_results = [get_weather_api_result]
4
5def add_execution_results(messages: List[Dict[str, Any]], execution_results: List[Dict[str, Any]]):
6 content = "\n".join([f"<tool_response>\n{json.dumps(result)}</tool_response>" for result in execution_results])
7 messages.append({"role": "user", "content": content})
8 return messages
9
10messages = add_execution_results(messages, execution_results)
11
12inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
13
14outputs = model.generate(
15 inputs,
16 max_new_tokens=512,
17 do_sample=False,
18 num_return_sequences=1,
19 eos_token_id=tokenizer.eos_token_id,
20)
21
22response = tokenizer.decode(outputs[0][len(inputs[0]) :], skip_special_tokens=True)
23print(response)
Katanemo Arch-Function collection is distributed under the
Katanemo license.
Additional thanks to @nicoboss for giving me access to his private supercomputer, enabling me to provide many more quants, at much higher speed, than I would otherwise be able to.
