A 1.08 billion parameter Hebrew instruction-tuned language model, fine-tuned from HebrewGPT-1B using LoRA Phase 2 curriculum distillation on 65K Hebrew instruction examples.
This is the latest and best instruct variant — achieving PPL 15.78 (↓47% from base) with 97.3% instruction following and zero repetition, trained for ~$12 on a single A10G GPU.
Interleaved blocks: Alternating RoPE multi-head attention and Mamba SSM layers
MLP: SwiGLU activation
Positional encoding: Rotary Position Embeddings (RoPE)
Training: LoRA Phase 2
Method
LoRA SFT with rank=64, alpha=128
Target modules: qkv, proj, gate, up, down
Trainable parameters: 44.7M / 1.08B (4%)
Data
65K examples combined from two-phase curriculum:
Phase 1 (ELI5 simple): 28.5K examples — simple explanations for foundational instruction following
Phase 2 (Sonnet/Nemotron complex): 36.5K examples — advanced, diverse instruction data
Two-Phase Curriculum
The training uses a curriculum distillation approach: starting with simple ELI5-style examples to establish instruction-following behavior, then progressing to complex Sonnet/Nemotron-generated examples for advanced capabilities.
Training Details
Property
Value
Hardware
NVIDIA A10G (AWS g5.2xlarge)
Training time
~8 hours
Best validation loss
2.4768 (BPB 3.57)
Early stopping
Step ~1000 (patience 5)
Total cost
~$12
Evaluation Results
Metric
Base Model
LoRA Phase 2
Delta
Perplexity
25.14
15.78
-37%
Instruction Following
—
97.3%
—
MCQA
—
10%
—
Repetition Rate
0.006
0.001
-83%
High-rep Outputs
—
0%
—
Key Improvements
Perplexity: 29.75 → 15.78 (-47% from base pretrained model)
Zero repetition — Phase 1 distillation had severe repetition loops; LoRA Phase 2 eliminates them entirely
Fluent Hebrew generation across diverse topics
97.3% instruction following rate — the model reliably follows the instruction format
Total post-training cost: ~$12 on a single NVIDIA A10G GPU
Usage
python
1import torch
2import sentencepiece as spm
34# Load tokenizer5sp = spm.SentencePieceProcessor()6sp.Load("tokenizer.model")78# Load model weights9state_dict = torch.load("model.pt", map_location="cpu")10# Initialize model architecture (see HebrewGPT-1B for model class definition)11# model.load_state_dict(state_dict)
Prompt Format
The model was trained with a structured instruction format:
