Supervised fine-tuned version of Trellis-506M, a 506M parameter LLaMA-style language model optimized for structured output tasks — JSON generation, function calling, schema compliance, and structured extraction.
Text → structured JSON extraction (generated with GPT-5-mini)
All examples are standardized to a common chat format using custom special tokens (see below). Source-specific filtering includes deduplication, token length capping (2048), and quality validation.
Hyperparameters
Parameter
Value
Epochs
3
Effective batch size
32
Learning rate
2e-5
LR schedule
Cosine decay
Warmup
10% of total steps
Weight decay
0.01
Max gradient norm
1.0
Max sequence length
2,048
Optimizer
AdamW (fused)
Precision
bfloat16
Seed
42
Training Details
Framework: TRL SFTTrainer
Attention: Flash Attention 2
Compilation:torch.compile enabled
Loss masking: Completion-only — loss computed only on assistant response tokens, not system/user/tool tokens
Hardware: NVIDIA B200
Chat Format
All training data uses these special tokens:
<|system|>You are a helpful assistant that generates valid JSON.<|end|>
<|user|>Generate a user profile with name, email, and age.<|end|>
<|assistant|>{"name": "Alice Chen", "email": "alice@example.com", "age": 28}<|end|>
Token
Purpose
<|system|>
System prompt
<|user|>
User message
<|assistant|>
Assistant response
<|tool_call|>
Function/tool call
<|tool_result|>
Tool execution result
<|end|>
End of turn
How to Use
python
1from transformers import AutoModelForCausalLM, AutoTokenizer
2import torch
34model = AutoModelForCausalLM.from_pretrained(5"mdonigian/trellis-sft",6 torch_dtype=torch.bfloat16,7 device_map="auto",8)9tokenizer = AutoTokenizer.from_pretrained("mdonigian/trellis-sft")1011prompt ="""<|system|>You are a helpful assistant that generates valid JSON.<|end|>
12<|user|>Generate a JSON object for a book with title, author, year, and genre.<|end|>
13<|assistant|>"""1415inputs = tokenizer(prompt, return_tensors="pt").to(model.device)16outputs = model.generate(**inputs, max_new_tokens=200, do_sample=True, temperature=0.7)17print(tokenizer.decode(outputs[0], skip_special_tokens=False))
Experimental Design
This model is one component of a controlled experiment comparing curated pretraining vs. standard pretraining for structured output tasks:
Model
Pretraining
Parameters
SFT
Trellis-506M-SFT (this model)
Curated 20B tokens
~506M
Identical
Pythia-410M-deduped-SFT
The Pile (uncurated)
~410M
Identical
Pythia-1B-deduped-SFT
The Pile (uncurated)
~1B
Identical
All three models undergo identical SFT with the same dataset, hyperparameters, and training procedure. Post-SFT evaluation covers:
