🏷️ Official Track B baseline for the AISA-ArabicFC shared task @ ArabicNLP 2026 (co-located with EMNLP 2026, Budapest)
This model is the organizer-provided baseline for Track B — Reasoning-Augmented Function Calling. It defines the reference score that participating systems are expected to beat. It is released for reproducibility and as a starting point — it is not a competition entry.
A compact (270M-parameter) Arabic function-calling model that, given an Arabic user query (in any of 5 dialects) and a set of candidate tools, writes a short Arabic <think> reasoning trace and then emits a structured tool call. Fine-tuned (LoRA) from google/gemma-3-270m on the AISA-ArabicFC reasoning data.
Given an Arabic user query and a set of candidate tool definitions, a system must:
Decide whether a function call is required (some queries need no tool),
Select the correct function name,
Extract the structured arguments,
(Track B)Generate an Arabic reasoning trace (<think> … </think>) before the call.
Track
Description
A — Core
Decide / Select / Extract
B — Reasoning-Augmented ← this model
Track A + an Arabic <think> reasoning trace
C — Cross-Dialect Robustness
Diagnostic: dialect-stratified evaluation of A/B submissions
How it works — input / output format
This model uses Gemma 3 chat turns with a custom function-calling schema (it does not emit plain JSON). The exact prompt is the text field in the dataset; the structure is:
<bos><start_of_turn>developer
<system instruction in Arabic>
<start_function_declaration>declaration:NAME{description:<escape>…<escape>,parameters:{…}}<end_function_declaration>
…one declaration per candidate tool…<end_of_turn>
<start_of_turn>developer
التاريخ والوقت الحالي …: 2024-04-12T23:05:24
اليوم هو الجمعة
أنت نموذج يمكنه استدعاء الوظائف التالية<end_of_turn>
<start_of_turn>user
أريد مقارنة أسعار تلفاز سامسونج في الأردن<end_of_turn>
<start_of_turn>model
The model then generates:
<think>
يبدو أن نية المستخدم هي الحصول على مقارنة لأسعار تلفاز سامسونج في الأردن. أداة "compare_prices" هي الأنسب …
</think>
<start_function_call>call:compare_prices{country:<escape>Jordan<escape>,product_name:<escape>Samsung TV<escape>}<end_function_call>
For a query that needs no tool, the model omits the <start_function_call> block (→ requires_function = false).
Usage
python
1import re, torch
2from transformers import AutoTokenizer, AutoModelForCausalLM
34MODEL_ID ="TuwaiqAcademy/AISA-AR-FunctionCall-Think"5tok = AutoTokenizer.from_pretrained(MODEL_ID)6model = AutoModelForCausalLM.from_pretrained(7 MODEL_ID, torch_dtype=torch.float32, device_map="auto"8).eval()910defparse_model_output(text:str)->dict:11"""Turn raw generation into the shared-task submission schema."""12 out ={"requires_function":False,"function_name":"none","arguments":{},"think":""}13if(m := re.search(r"<think>\s*(.*?)\s*</think>", text, re.DOTALL)):14 out["think"]= m.group(1).strip()15if(m := re.search(r"<start_function_call>\s*call:(\w+)\{(.*?)\}\s*<end_function_call>", text, re.DOTALL)):16 out["requires_function"]=True17 out["function_name"]= m.group(1)18for key, str_val, num_val in re.findall(r"(\w+):(?:<escape>(.*?)<escape>|([^,}]+))", m.group(2)):19 val = str_val if str_val else num_val
20try:21 val =float(val)if"."instr(val)elseint(val)22except(ValueError, TypeError):23pass24 out["arguments"][key]= val
25return out
2627# Easiest path: take the ready-made prompt from the dataset's `text` field and28# cut it at the model turn (everything after is what the model should produce).29from datasets import load_dataset
30row = load_dataset("TuwaiqAcademy/AISA-ArabicFC", split="validation")[0]31prompt = row["text"].split("<start_of_turn>model\n")[0]+"<start_of_turn>model\n"3233inputs = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)34with torch.no_grad():35 gen = model.generate(**inputs, max_new_tokens=250, do_sample=False)# greedy36raw = tok.decode(gen[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False)3738print(parse_model_output(raw))39# → {'requires_function': True, 'function_name': 'compare_prices',40# 'arguments': {'country': 'Jordan', 'product_name': 'Samsung TV'},41# 'think': 'يبدو أن نية المستخدم …'}
The parsed dict maps directly onto a leaderboard submission line: {"id", "tool_called", "arguments", "think"} (use function_name → tool_called).
Evaluation
Scored on the AISA-ArabicFC held-out test set (1,000 positive + negative examples) using the official v2 metrics:
FnAcc — function-name accuracy over all samples (also penalises hallucinated / missed calls; negatives have gold none)
ArgEM — strict argument exact match, over positives only
ThinkRate — fraction of outputs with a non-empty <think> trace
Overall (Track A) = 0.40·FnAcc + 0.60·ArgEM
Overall (Track B) = 0.30·FnAcc + 0.50·ArgEM + 0.20·ThinkRate
Baseline results
System
FnAcc
ArgEM
Overall (A)
Overall (B)
AISA-AR-FunctionCall-Think (270M) ← this
0.982
0.541
0.717
0.739
GPT-4o — zero-shot
0.927
0.070
0.413
0.313
GPT-4o — 3-shot
0.854
0.122
0.415
0.317
Random baseline
0.047
0.033
0.039
0.031
Think-Before-Call rate (ThinkRate):0.868 for this model; 0.000 for all non-reasoning baselines.
Hallucination rate:0.000 on negative (no-tool) queries.
Key takeaways
🎯 Argument extraction is the open challenge. Tool selection is largely solved (FnAcc ≈ 0.98), but strict argument exact match tops out at 0.541 — and GPT-4o reaches only 0.070 zero-shot. This is where the task is won or lost.
🪶 A 270M model beats GPT-4o across every metric here, showing the value of task-specific Arabic training and lowering the compute barrier to entry.
🗣️ Cross-dialect gaps remain. FnAcc varies by roughly 10–15 points across dialects, with Gulf and Levantine consistently the hardest and Maghrebi (small sample) the easiest — see the Track C diagnostic in the task overview paper.
Training
Base:google/gemma-3-270m
Method: LoRA (rank 64), 3 epochs, cosine LR scheduler
Data: AISA-ArabicFC training split (~10.5K examples) with 12,000 Arabic reasoning annotations for the <think> traces
Objective: produce a short Arabic reasoning trace followed by a single structured tool call (or no call for negatives)
Intended use & limitations
Intended use
A reference baseline to compare against and reproduce for the AISA-ArabicFC shared task.
A lightweight starting point for Arabic tool-use / agentic experiments.
Out of scope / limitations
Trained for the 27-tool, 8-domain AISA-ArabicFC schema and its prompt format; behaviour on arbitrary tools or free-form chat is undefined.
Single-turn, single-call setting — no multi-tool or multi-turn dialogue.
Argument extraction is imperfect (ArgEM 0.541): expect errors in date normalisation, numeric typing, and dialectal argument phrasing.
Uneven dialect coverage (Maghrebi is only ~1.3% of data); robustness varies by dialect.
A 270M model — capacity-limited by design to keep the baseline accessible.
1@inproceedings{najar2026aisaarabicfc,
2 title = {AISA-ArabicFC: Arabic Function Calling for Agentic AI Systems},
3 author = {Najar, Omar},
4 booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference (ArabicNLP 2026)},
5 year = {2026}
6}
License
This model is a derivative of Gemma 3 and is distributed under the Gemma Terms of Use. By using it you agree to those terms and to the Gemma Prohibited Use Policy. The AISA-ArabicFC dataset is released separately under Apache-2.0.
