ToolCall-200M-Base
ToolCall-200M-Base is a 200,184,576-parameter causal language
model pretrained fully from scratch on approximately 4.006 billion
prediction tokens.
This is the pretrained base checkpoint before supervised fine-tuning.
Important status
This model is:
- not instruction-tuned
- not a general chatbot
- not ready for reliable tool execution
- expected to produce repetition, malformed outputs and incorrect tool calls
Its intended purpose is research, reproducibility and comparison with the
future supervised fine-tuned ToolCall-200M model.
Architecture
- Layers: 24
- Hidden size: 768
- Intermediate size: 2,048
- Attention heads: 12
- Key/value heads: 4
- Vocabulary: 32,000
- Context length: 2,048
- Activation: SwiGLU
- Normalization: RMSNorm
- Position encoding: RoPE
- Embeddings: untied
Training
- Prediction tokens: 4,005,822,464
- Final checkpoint step: 15,281
- Training: fully from scratch
Loading status
The repository includes the configuration/model code discovered in the source repository.
Limitations
This checkpoint has not undergone supervised fine-tuning or alignment.
It should not be used to make autonomous decisions or execute tools in
production.
