Lumma-0.6B-Base is a multilingual decoder-only language model trained from scratch on 1 trillion tokens. It is designed for efficient deployment, long-context inference, and strong multilingual performance across English and Indic languages, featuring a compact transformer architecture, memory-efficient attention mechanisms, and an optimized multilingual tokenizer.
Key Features
Trained from scratch on 1 trillion tokens
600 million parameter decoder-only Transformer
Native English and Indic language pretraining
Shared KV Attention for memory-efficient inference
12,288 token context length
Grouped Query Attention (GQA)
RMSNorm with QK Normalization
SwiGLU feed-forward network
Factorized tied embeddings
Large multilingual tokenizer optimized for Indic languages
[!NOTE]
We do not recommend using base language models for conversations. Instead, you can apply post-training, e.g., SFT, RLHF, continued pretraining, etc., on this model.
Shared KV
Lumma introduces Shared KV, an alternative key-value caching strategy designed to reduce inference memory requirements without significantly impacting model quality.
Instead of computing independent Key and Value projections, both are derived from a shared latent representation. During attention computation, lightweight Key normalization and RoPE transformations are applied dynamically.
This approach reduces KV-cache memory usage by approximately 50%, making Lumma better suited for long-context inference and memory-constrained deployments.
KV Cache Modes
Lumma supports two inference modes depending on deployment requirements.
Shared KV
model.config.kv_cache_mode = "shared"
Recommended when memory is the primary bottleneck.
Approximately 50% lower KV-cache memory
Slightly higher compute overhead
Better suited for long-context inference
Vanilla KV
model.config.kv_cache_mode = "vanilla"
Recommended for standard deployments.
Standard KV-cache implementation
Lower compute overhead
Maximum compatibility across inference frameworks
Benchmark Results
The following results correspond to the released Lumma-0.6B model trained on 1 trillion tokens.
General Benchmarks
Model Name
Tokens Budget (Trillion)
HellaSwag
Winogrande
OBQA
ARC-e
ARC-c
Average
MobiLlama-0.5B-Base
1.3
39.65
53.67
30.60
52.82
23.63
40.07
Qwen-2-0.5-Base
12
49.01
57.69
33.20
54.79
25.42
44.02
Qwen2.5-0.5B-Base
18
52.16
56.82
35.40
64.64
29.86
47.78
Lumma-0.6B-Base
1
46.25
54.14
32.80
60.60
28.58
44.47
Multilingual Tokenization
Efficient tokenization is particularly important for multilingual language models.
Lower fertility indicates fewer tokens are required to represent text, improving both training efficiency and inference cost.
