ARK-65M is a compact transformer language model designed as a fast, efficient assistant that delegates factual knowledge to web search rather than memorizing it in its weights. At 65M parameters, the model excels at conversation, math reasoning, code generation, and tool use — while relying on a web_search tool for factual questions it cannot reliably store.
Architecture
Component
Detail
Parameters
68,244,768 (65M effective)
Layers
14 (7 GQA + 7 MSA, alternating)
Hidden dim
576
Attention heads
8 (query), 2 (KV, grouped)
Head dim
72
FFN
SwiGLU, 1536 intermediate
Context length
2048 tokens
Vocabulary
32,768 (shared tokenizer)
Positional encoding
RoPE (θ=10000)
Normalization
RMSNorm
The architecture alternates between Grouped Query Attention (GQA) and Multi-Head Self-Attention (MSA) layers, combining the efficiency of GQA with the expressiveness of full attention.
SFT was performed from the base pretrained model with ~86k examples across 3 epochs:
UltraChat 200k — multi-turn English conversations (15k)
Alpaca Cleaned — instruction following (8k)
Camoscio — Italian instruction following (6k)
Glaive Function Calling — tool use patterns (15k)
Pandora Tool Calling — tool use with refusals (5k)
TriviaQA / NQ Open — factual QA (16k)
Web Search Tool Call — custom dataset teaching web search delegation (~4.8k)
Math CoT — synthetic arithmetic with chain-of-thought (~14k)
Identity + Conversation — custom ARK-65M identity data (~2k)
Capabilities
What it does well
Identity: Correctly identifies itself as ARK-65M by ModotAI (100% accuracy)
Math: Arithmetic with step-by-step reasoning via <|thinking|> tags (~83%)
Tool calling: Generates structured JSON tool calls when appropriate (100%)
Web search delegation: Routes factual questions to web_search tool instead of hallucinating
Bilingual: Responds naturally in both Italian and English
Conversation: Handles greetings, thanks, and basic multi-turn dialogue
What it doesn't do
Factual recall: With 65M parameters, the model cannot reliably memorize encyclopedic knowledge. It will hallucinate if asked factual questions without the web search tool enabled.
Long-form generation: Context window is 2048 tokens.
Complex reasoning: Word problems and multi-step logic are limited.
Usage
Inference (CLI)
python
1python inference.py \
2--checkpoint model.pt \
3--prompt "Chi è Elon Musk?" \
4--system "Sei ARK-65M, un modello linguistico italiano creato da ModotAI. Rispondi in modo utile e preciso. Hai accesso al tool web_search per cercare informazioni fattuali."
Chat format (ChatML)
ARK-65M uses a ChatML-style format with special role tokens:
<|im_start|><|system|>Sei ARK-65M, un modello linguistico italiano creato da ModotAI. Rispondi in modo utile e preciso. Hai accesso al tool web_search per cercare informazioni fattuali.<|im_end|>
<|im_start|><|user|>Chi è Elon Musk?<|im_end|>
<|im_start|><|assistant|><|tool_calls|>[{"name": "web_search", "arguments": {"query": "Elon Musk chi è"}}]<|/tool_calls|><|im_end|>
<|im_start|><|tool|>web_search: Elon Musk è un imprenditore sudafricano naturalizzato statunitense, CEO di Tesla e SpaceX.<|im_end|>
<|im_start|><|assistant|>Elon Musk è un imprenditore sudafricano naturalizzato statunitense. È il CEO di Tesla e SpaceX.<|im_end|>
Special tokens
Token
Purpose
<|im_start|> / <|im_end|>
Message boundaries
<|system|><|user|><|assistant|><|tool|>
Role markers
<|tool_calls|> / <|/tool_calls|>
Tool call JSON wrapper
<|thinking|> / <|/thinking|>
Chain-of-thought (math)
Server deployment
ARK-65M runs as part of the Quark Multi-Model Inference Engine (app.py v14.0) with automatic server-side web search execution:
User sends a question
Model generates a web_search tool call (if factual)
Server executes the search via Serper API
Model reformulates the search results into a response
User receives the final answer — tool call is transparent
Benchmark (SFT v3, epoch 2)
Category
Score
Identity
100.0%
Tool call
100.0%
Math
83.3%
Code
66.7%
Conversation
66.7%
Reasoning
33.3%
Facts (without web search)
38.9%
Overall
70.5%
With web search enabled, factual accuracy improves dramatically as the model no longer halluccinates — it retrieves real information from the web.
Tokenizer
Uses ThingAI/msqark-tokenizer (32,768 vocab), shared with the Quark-270M model family. The embedding layer accounts for ~27.7% of total parameters — a deliberate tradeoff to reuse an existing tokenizer rather than training a smaller one.
Files
File
Description
model.pt
SFT checkpoint (epoch 2)
ark65m.py
Model architecture
config.json
Model configuration
Hardware requirements
Inference: Any GPU with ≥2GB VRAM (or CPU)
SFT: NVIDIA GPU with ≥6GB VRAM
Pretraining: NVIDIA GPU with ≥8GB VRAM
License
This model is released for research and personal use.
Author
Built by ModotAI — an independent Italian AI research project focused on efficient language models.
