You can deploy this model and start earning money today!

prem-research/
Funcdex-1.7B

Views

No views yet

Funcdex-Collection

Dataset

GitHub

Funcdex-1.7B

Funcdex-1.7B is a research preview model by Prem Labs. It has been trained on a mix of Funcdex-MT-Function-Calling, Instruct-Following, Single-turn function datasets. It is a LoRA finetune of Qwen3-1.7B (with thinking disabled).

This model excels at Multi-turn Function Calling with tools from gmail, jira, calendar, docs, etc.

The code used to generate the dataset can be found here.

Evaluation

Notes:

Funcdex-0.6B is the average of performances of individual Funcdex-0.6B models.
For cost, we track the number of prompt/completion tokens for evaluating 300 conversations.
e.g. If token cost is input=$1 and output=$10 per million tokens, and evaluation needed 0.5M and 0.1M input/output tokens, then cost is 1 * 0.5 + 0.1 * 10 = $1.5.
Qwen3-0.6B and Qwen3-1.7B evaluation costs are estimated by extrapolating from Llama3.2-3B serverless costs. Other model's costs are sourced from Openrouter.

Results

BFCL v3

We filtered BFCLv3 examples relevant to the toolkits/bundles and report performance:
The filtered set is only 83 examples. Further emphasizing the need for workflow/toolkit-specialized workflows.

LLM	Acc %
GPT-5 Mini (medium)	0.71
Qwen3-1.7B	0.82
Funcdex-1.7B	0.86

Funcdex-MT: Overall Performance

LLM	Exact Match	String Ratio	Total Cost ($)
GPT-OSS-120B (medium)	0.35	0.51	9.32
GPT-5 Mini (medium)	0.35	0.58	99.71
GPT-5 (minimal)	0.18	0.59	205.45
Qwen3-0.6B	0.27	0.59	2.83
Qwen3-1.7B	0.27	0.69	5.73
Funcdex-0.6B	0.39	0.70	0.19
Funcdex-1.7B	0.43	0.81	5.64

Funcdex-MT: Toolkit-Level Performance

Toolkit	GPT-OSS-120B (medium)	GPT-5 (minimal)	GPT-5 Mini (medium)	Qwen3-0.6B	Funcdex-0.6B	Qwen3-1.7B	Funcdex-1.7B
EM	SR	EM	SR	EM	SR	EM	SR	EM	SR	LoRA Checkpoint	EM	SR	EM	SR	LoRA Checkpoint
Asana	0.38	0.47	0.12	0.68	0.49	0.71	0.33	0.63	0.46	0.69	🤗	0.30	0.79	0.52	0.82	🤗
Calendly	0.47	0.56	0.41	0.63	0.41	0.56	0.44	0.66	0.54	0.78	🤗	0.47	0.74	0.54	0.86
Gmail	0.48	0.70	0.24	0.69	0.50	0.73	0.27	0.61	0.47	0.72	🤗	0.31	0.73	0.53	0.83
Calendar	0.27	0.52	0.20	0.50	0.21	0.51	0.21	0.53	0.39	0.74	🤗	0.23	0.64	0.47	0.83
Docs	0.19	0.38	0.07	0.49	0.18	0.46	0.07	0.58	0.13	0.64	🤗	0.11	0.62	0.18	0.79
Drive	0.34	0.52	0.19	0.61	0.38	0.58	0.26	0.65	0.40	0.75	🤗	0.26	0.73	0.48	0.82
Jira	0.47	0.53	0.17	0.65	0.47	0.66	0.51	0.69	0.58	0.76	🤗	0.47	0.76	0.59	0.83
Stripe	0.15	0.37	0.10	0.46	0.12	0.39	0.08	0.50	0.17	0.71	🤗	0.09	0.56	0.16	0.80
Todoist	0.65	0.74	0.19	0.72	0.64	0.79	0.57	0.87	0.65	0.88	🤗	0.55	0.91	0.72	0.94
Whatsapp	0.23	0.39	0.13	0.47	0.24	0.43	0.20	0.43	0.28	0.64	🤗	0.26	0.55	0.31	0.71

Funcdex-0.6B are specialized models. Reported number is the average performance of each specific model in their respective subset.

Funcdex-MT: Bundle/Multi-toolkit Performance:

Bundle	GPT-OSS-120B (medium)	GPT-5 (minimal)	GPT-5 Mini (medium)	Qwen3-0.6B	Funcdex-0.6B	Qwen3-1.7B	Funcdex-1.7B
EM	SR	EM	SR	EM	SR	EM	SR	EM	SR	LoRA Checkpoint	EM	SR	EM	SR	LoRA Checkpoint
Gmail Calendar	0.28	0.53	0.15	0.54	0.22	0.56	0.19	0.51	0.26	0.54	🤗	0.17	0.61	0.32	0.71	🤗
Drive Calendly Calendar	0.32	0.45	0.17	0.52	0.35	0.47	0.19	0.49	0.35	0.60	🤗	0.15	0.66	0.40	0.78
Drive Docs	0.28	0.37	0.12	0.50	0.33	0.47	0.18	0.54	0.34	0.70	🤗	0.19	0.68	0.43	0.76
Jira Gmail	0.42	0.60	0.18	0.66	0.36	0.66	0.29	0.61	0.39	0.71	🤗	0.28	0.72	0.44	0.82
Whatsapp Todoist	0.32	0.58	0.19	0.66	0.35	0.69	0.26	0.50	0.41	0.70	🤗	0.27	0.68	0.39	0.77

Inference

Given a conversation, we extract all tuples (context_messages, function_calls) and use it to generate predictions. We ignore the content field and only evaluate function_calls generated by an LLM.
We use vLLM deployment with tool_choice="auto".

Metrics

Given a list of predicted and reference function calls, we report two metrics:

Function Call String Match (SR): We perform greedy match and report best-matched string ratio using difflib.SequenceMatcher.ratio. The number reported is average string ratio.
Exact Match (EM): Same as above, but we perform exact string match instead. The number reported is EM F1 Score.

EM is a strict metric, and penalizes string arguments in function calls that may be "okay", e.g. "email_content": "This is an example." v/s "email_content": "This is an Example.", both only differ by one letter.

Deployment with vLLM

vllm serve ojus1/Qwen3-1.7B-Instruct --enable-lora --lora-modules prem-research/Funcdex-1.7B=prem-research/Funcdex-1.7B  --enable-auto-tool-choice --tool-call-parser hermes

Quickstart

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch
import json

# Load model and tokenizer
base_model_name = "ojus1/Qwen3-1.7B-Instruct"
model_name = "prem-research/Funcdex-1.7B"

tokenizer = AutoTokenizer.from_pretrained(model_name)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype="auto",
    device_map="auto"
)

model = PeftModel.from_pretrained(
    base_model,
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

# Define tools (supports all toolkits)
tools = [
    {
        "type": "function",
        "function": {
            "name": "CREATE_SHARED_DRIVE",
            "description": "Create a new shared drive in Google Drive",
            "parameters": {
                "type": "object",
                "properties": {
                    "name": {"type": "string", "description": "Name of the shared drive"},
                    "requestId": {"type": "string", "description": "Unique request ID"}
                },
                "required": ["name", "requestId"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "CREATE_A_FOLDER",
            "description": "Create a folder in Google Drive",
            "parameters": {
                "type": "object",
                "properties": {
                    "folder_name": {"type": "string", "description": "Name of the folder"},
                    "parent_id": {"type": "string", "description": "Parent drive or folder ID"}
                },
                "required": ["folder_name", "parent_id"]
            }
        }
    }
]

# Define conversation
messages = [
    {"role": "system", "content": "You are a helpful assistant that can help with tasks by using tools."},
    {"role": "user", "content": "Create a shared drive named 'Partner-Alpha-Integration' with request ID 'req-12345'."}
]

# Apply chat template with tools
formatted_input = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    tokenize=False,
    add_generation_prompt=True
)

# Tokenize and generate
input_tokens = tokenizer(formatted_input, return_tensors="pt").to(model.device)
output = model.generate(**input_tokens, max_new_tokens=256, do_sample=False)
response = tokenizer.decode(output[0][input_tokens['input_ids'].shape[1]:], skip_special_tokens=True)

print("Response:", response)
# Expected output includes: <tool_call>{"name": "CREATE_SHARED_DRIVE", "arguments": {"name": "Partner-Alpha-Integration", "requestId": "req-12345"}}</tool_call>

For best results, provide detailed system-prompt to steer the tool-use behaviour.

License

The models, code and the dataset are licensed under MIT License.