Alo-70M (Instruct)

Name: Alo-70m
Rating: 2.5 (280 reviews)

Model Summary

Alo-70M is the instruction-tuned version of the ultra-lightweight 69-million parameter Bengali language model, Alo-70M-Base. Built on a scaled-down LLaMA architecture, it is designed to act as a highly efficient, edge-deployable localized AI assistant.

Fine-tuned on a curated dataset of instruction-response pairs using the ChatML format, Alo-70M is aligned for tasks such as summarization, entity extraction, text editing, and question answering in native Bengali. Despite its compact footprint, it offers a viable path for edge AI deployment on standard CPUs and mobile hardware.

Developer: Fahad Hossain
Language: Bengali (Bangla)
Model Type: Causal Language Model (Instruction-Tuned Autoregressive Transformer)
Parameter Count: 69 Million
License: Apache 2.0

Related Resources

Base Model: spitfire4794/Alo-70M-Base
Alignment Dataset: spitfire4794/Bangla-SFT-50k
Tokenizer: spitfire4794/beng_bpe

Usage

Alo-70M was trained using the ChatML template. The chat template is built directly into the Jinja template of the tokenizer (spitfire4794/beng_bpe). You can leverage it using Hugging Face's apply_chat_template interface:

python

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "spitfire4794/Alo-70M"

# Load the custom Bengali BPE tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

# Define the instruction in ChatML format
messages = [
    {"role": "user", "content": "নিচের অনুচ্ছেদটি সংক্ষেপে সারসংক্ষেপ করুন: [এখানে আপনার টেক্সট লিখুন]"}
]

# Apply the pre-configured ChatML template
inputs = tokenizer.apply_chat_template(
    messages, 
    add_generation_prompt=True, 
    return_tensors="pt"
).to(model.device)

# Generate text
outputs = model.generate(
    **inputs, 
    max_new_tokens=150, 
    repetition_penalty=1.1,
    do_sample=True,
    temperature=0.6,
    top_p=0.9
)

# Decode response (omitting user prompt)
response = outputs[0][inputs.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))

Supervised Fine-Tuning (SFT) Details

Alo-70M was aligned using a curated subset of the Bangla-SFT-50k dataset formatted using ChatML.

Dataset Pruning: Initial SFT experiments revealed that forcing a sub-100M parameter model to learn complex markdown syntax/tables caused severe representation crowding. Thus, the 12,517-sample Structured Formatting category was excluded. The final active training mixture consisted of 37,536 aligned pairs.
Engineering Properties: The training data strictly forbade conversational prefaces (e.g., "নিশ্চয়ই, আমি এটি করে দিচ্ছি") so that responses begin immediately with the target output, optimizing inference speeds.
Hardware: NVIDIA T4 and L4 GPUs.
Hyperparameters:
- Optimizer: Fused AdamW (adamw_torch_fused) with beta_1 = 0.9, beta_2 = 0.999, epsilon = 10^-8
- Weight Decay: 0.0
- Learning Rate Schedule: Cosine decay, peaking at 3 times 10^-4 with a 10% linear warmup.
- Epochs: 3
- Effective Batch Size: 32 (per-device 8 with gradient accumulation of 4).
- Precision: Native Automatic Mixed Precision (AMP).

Model Architecture details

Like its base model, Alo-70M utilizes a parameter-efficient architecture:

Layers: 12
Hidden Dimension: 512 | Intermediate FFN: 1408
Attention: Grouped-Query Attention (GQA) with 8 query heads / 4 KV heads.
Positional Embeddings: RoPE (Base freq: 10,000)
Word Embeddings: Untied.
Context Window: 1024 tokens.

Evaluation Results

The model was evaluated zero-shot across Bengali reasoning and knowledge benchmarks (continuation-based log-probability evaluation):

Benchmark	Alo-70M (SFT)	Alo-70M-Base	Gemma-3-270M-IT	TigerLLM-1B-IT
bangla_mmlu_bn	26.29%	26.31%	26.81%	27.66%
bangla_commonsenseqa_bn	25.88%	28.42%	22.77%	25.14%
indicbench_arc_bn_challenge	24.15%	22.70%	25.34%	27.13%
boolqa_bn	48.70%	48.42%	51.30%	52.40%
openbookqa_bn	30.58%	31.39%	31.99%	34.21%
piqa_bn	50.05%	50.49%	49.51%	49.51%
hellaswag_bn	26.89%	27.27%	27.85%	31.01%

Note: The 69M instruction-tuned model outperforms the larger Gemma-3-270M-IT baseline on tasks like CommonsenseQA and PIQA.

Limitations and Biases

Alignment Tax (Catastrophic Forgetting): While SFT successfully aligned the model for text generation stability and instruction following, it introduced a measurable degradation in pure zero-shot reasoning compared to the Base model (e.g., dropping from 28.42% to 25.88% on CommonsenseQA). This happens because applying instructions to a highly capacity-constrained 69M model over-indexes weights toward output formatting at the expense of some pre-trained logical representations.
Knowledge Retrieval: With under 100M parameters, the model physically lacks the capacity to serve as a comprehensive encyclopedic knowledge base. It is better suited for text processing tasks (editing, summarizing) than fact-retrieval.
Context Length: The model is optimized for a 1024-token context window. Prompts exceeding this length will be truncated or result in degraded quality.

Citation

Technical paper out soon.

spitfire4794/Alo-70m