You can deploy this model and start earning money today!

prem-research/
Funcdex-0.6B-jira_gmail

Name: Funcdex-0.6B-jira_gmail
Availability: InStock

Views

No views yet

Funcdex-Collection

Dataset

GitHub

Funcdex-0.6B-jira_gmail

Funcdex-0.6B is a research preview model by Prem Labs. It has been trained on a mix of Funcdex-MT-Function-Calling, Instruct-Following, Single-turn function datasets. It is a LoRA finetune of Qwen3-0.6B (with thinking disabled).

This model excels at Multi-turn Function Calling with tools from jira and gmail.

The code used to generate the dataset can be found here.

Evaluation

Results

BFCL v3

We filtered BFCLv3 examples relevant to the toolkits/bundles and report performance:
The filtered set is only 83 examples. Further emphasizing the need for workflow/toolkit-specialized workflows.

LLM	Acc %
GPT-5 Mini (medium)	0.71
Qwen3-1.7B	0.82
Funcdex-1.7B	0.86

Funcdex-MT: Overall Performance

LLM	Exact Match	String Ratio	Total Cost ($)
GPT-OSS-120B (medium)	0.35	0.51	9.32
GPT-5 Mini (medium)	0.35	0.58	99.71
GPT-5 (minimal)	0.18	0.59	205.45
Qwen3-0.6B	0.27	0.59	2.83
Qwen3-1.7B	0.27	0.69	5.73
Funcdex-0.6B	0.39	0.70	0.19
Funcdex-1.7B	0.43	0.81	5.64

Funcdex-MT: Toolkit-Level Performance

Toolkit	GPT-OSS-120B (medium)	GPT-5 (minimal)	GPT-5 Mini (medium)	Qwen3-0.6B	Funcdex-0.6B	Qwen3-1.7B	Funcdex-1.7B
EM	SR	EM	SR	EM	SR	EM	SR	EM	SR	LoRA Checkpoint	EM	SR	EM	SR	LoRA Checkpoint
Asana	0.38	0.47	0.12	0.68	0.49	0.71	0.33	0.63	0.46	0.69	🤗	0.30	0.79	0.52	0.82	🤗
Calendly	0.47	0.56	0.41	0.63	0.41	0.56	0.44	0.66	0.54	0.78	🤗	0.47	0.74	0.54	0.86
Gmail	0.48	0.70	0.24	0.69	0.50	0.73	0.27	0.61	0.47	0.72	🤗	0.31	0.73	0.53	0.83
Calendar	0.27	0.52	0.20	0.50	0.21	0.51	0.21	0.53	0.39	0.74	🤗	0.23	0.64	0.47	0.83
Docs	0.19	0.38	0.07	0.49	0.18	0.46	0.07	0.58	0.13	0.64	🤗	0.11	0.62	0.18	0.79
Drive	0.34	0.52	0.19	0.61	0.38	0.58	0.26	0.65	0.40	0.75	🤗	0.26	0.73	0.48	0.82
Jira	0.47	0.53	0.17	0.65	0.47	0.66	0.51	0.69	0.58	0.76	🤗	0.47	0.76	0.59	0.83
Stripe	0.15	0.37	0.10	0.46	0.12	0.39	0.08	0.50	0.17	0.71	🤗	0.09	0.56	0.16	0.80
Todoist	0.65	0.74	0.19	0.72	0.64	0.79	0.57	0.87	0.65	0.88	🤗	0.55	0.91	0.72	0.94
Whatsapp	0.23	0.39	0.13	0.47	0.24	0.43	0.20	0.43	0.28	0.64	🤗	0.26	0.55	0.31	0.71

Funcdex-0.6B are specialized models. Reported number is the average performance of each specific model in their respective subset.

Funcdex-MT: Bundle/Multi-toolkit Performance:

Bundle	GPT-OSS-120B (medium)	GPT-5 (minimal)	GPT-5 Mini (medium)	Qwen3-0.6B	Funcdex-0.6B	Qwen3-1.7B	Funcdex-1.7B
EM	SR	EM	SR	EM	SR	EM	SR	EM	SR	LoRA Checkpoint	EM	SR	EM	SR	LoRA Checkpoint
Gmail Calendar	0.28	0.53	0.15	0.54	0.22	0.56	0.19	0.51	0.26	0.54	🤗	0.17	0.61	0.32	0.71	🤗
Drive Calendly Calendar	0.32	0.45	0.17	0.52	0.35	0.47	0.19	0.49	0.35	0.60	🤗	0.15	0.66	0.40	0.78
Drive Docs	0.28	0.37	0.12	0.50	0.33	0.47	0.18	0.54	0.34	0.70	🤗	0.19	0.68	0.43	0.76
Jira Gmail	0.42	0.60	0.18	0.66	0.36	0.66	0.29	0.61	0.39	0.71	🤗	0.28	0.72	0.44	0.82
Whatsapp Todoist	0.32	0.58	0.19	0.66	0.35	0.69	0.26	0.50	0.41	0.70	🤗	0.27	0.68	0.39	0.77

Inference

Given a conversation, we extract all tuples (context_messages, function_calls) and use it to generate predictions. We ignore the content field and only evaluate function_calls generated by an LLM.
We use vLLM deployment with tool_choice="auto".

Metrics

Given a list of predicted and reference function calls, we report two metrics:

Function Call String Match (SR): We perform greedy match and report best-matched string ratio using difflib.SequenceMatcher.ratio. The number reported is average string ratio.
Exact Match (EM): Same as above, but we perform exact string match instead. The number reported is EM F1 Score.

EM is a strict metric, and penalizes string arguments in function calls that may be "okay", e.g. "email_content": "This is an example." v/s "email_content": "This is an Example.", both only differ by one letter.

Quickstart

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch
import json

# Load model and tokenizer
base_model_name = "ojus1/Qwen3-0.6B-Instruct"
model_name = "prem-research/Funcdex-0.6B-jira_gmail"

tokenizer = AutoTokenizer.from_pretrained(model_name)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype="auto",
    device_map="auto"
)

model = PeftModel.from_pretrained(
    base_model,
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

# Define tools (Jira + Gmail combined)
tools = [
    {
        "type": "function",
        "function": {
            "name": "LIST_DRAFTS",
            "description": "List Gmail draft messages",
            "parameters": {
                "type": "object",
                "properties": {
                    "max_results": {"type": "integer", "description": "Maximum number of drafts"},
                    "verbose": {"type": "boolean", "description": "Verbose output"}
                }
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "CREATE_PROJECT",
            "description": "Create a Jira project",
            "parameters": {
                "type": "object",
                "properties": {
                    "key": {"type": "string", "description": "Project key"},
                    "name": {"type": "string", "description": "Project name"},
                    "project_type_key": {"type": "string", "description": "Project type"}
                },
                "required": ["key", "name"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "GET_CONTACTS",
            "description": "Get Gmail contacts",
            "parameters": {
                "type": "object",
                "properties": {
                    "person_fields": {"type": "string", "description": "Fields to retrieve"}
                }
            }
        }
    }
]

# Define conversation
messages = [
    {"role": "system", "content": "You are a helpful assistant that can help with tasks by using tools."},
    {"role": "user", "content": "List drafts from Gmail with verbose output, maximum 20 results."}
]

# Apply chat template with tools
formatted_input = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    tokenize=False,
    add_generation_prompt=True
)

# Tokenize and generate
input_tokens = tokenizer(formatted_input, return_tensors="pt").to(model.device)
output = model.generate(**input_tokens, max_new_tokens=256, do_sample=False)
response = tokenizer.decode(output[0][input_tokens['input_ids'].shape[1]:], skip_special_tokens=True)

print("Response:", response)

Deployment with vLLM

vllm serve ojus1/Qwen3-0.6B-Instruct --enable-lora --lora-modules prem-research/Funcdex-0.6B=prem-research/Funcdex-0.6B-jira_gmail  --enable-auto-tool-choice --tool-call-parser hermes

For best results, provide detailed system-prompt to steer the tool-use behaviour.

License

The models, code and the dataset are licensed under MIT License.