NorMistral-11b-long is a length-extended version of
NorMistral-11b-warm. It has been extended to 32,768 context length by continual training on additional 50 billion subword tokens – using a mix of Scandinavian, Sámi, English and code data (four repetitions of open Norwegian texts). The model follows our earlier paper
Small Languages, Big Models: A Study of Continual Training on Languages of Norway by Samuel et al. 2025, and forms part of the NORA.LLM family developed by
the Language Technology Group at the University of Oslo (LTG).
Disclaimer: This model is pretrained on raw (mostly web-based) textual data. It is not finetuned to follow instructions, and it can generate harmful completions after inappropriate user prompts. It is primarily intended for research purposes.
License
We release the model under Apache 2.0 license to indicate that we do not impose any additional constraints on the model weights.
However, we do not own the data in the training collection.
Pretraining corpus
The model is pretrained on a combination of publicly available data and a custom web crawl for Sámi. The total training corpus consists of 250 billion tokens from the following sources:
-
Norwegian text (Bokmål and Nynorsk); this collection was created by the National Library of Norway and it's a prerelease of an update of NCC (codenamed "Mímir core"). It consists of: a) the public part of
Norwegian Colossal Corpus (NCC) with permissible licenses (i.e. it doesn't include newspaper texts with the CC BY-NC 2.0 license); b) Bokmål and Nynorsk
CulturaX, and c) Bokmål and Nynorsk
HPLT corpus v1.2.
-
Northern Sámi texts are sourced from a)
Glot500; b)
the SIKOR North Saami free corpus; and c) a custom web crawl (seeded from Sámi Wikipedia external links) published separately as
ltg/saami-web.
-
Additional languages for knowledge/language transfer: a) Danish, Swedish, Icelandic, and Faroese from CulturaX and Glot500; b) high-quality English from
FineWeb-edu; and c) programming code from
The Stack v2 (the high-quality subset).
The corpus is carefully balanced through strategic upsampling to handle the resource disparity between languages. Following data-constrained scaling laws, the corpus data for target languages is repeated multiple times (up to 16x for low-resource languages) to reach the optimal training budget while avoiding overfitting:
Tokenizer
This model uses a new tokenizer, specially trained on the target languages. Therefore it offers substantially faster inference than the original Mistral-Nemo-Base-2407 model. Here are the subword-to-word split ratios across different languages:
| Tokenizer | # tokens | Bokmål | Nynorsk | Sámi | Danish | Swedish |
|---|
| Mistral-Nemo-Base-2407 | 131072 | 1.79 | 1.87 | 2.63 | 1.82 | 2.00 |
| NorMistral-11b-long | 51200 | 1.22 | 1.28 | 1.82 | 1.33 | 1.39 |
Model details
Model Developers: Language Technology Group at the University of Oslo in collaboration with NORA.LLM.
Architecture: NorMistral-11B uses the Mistral architecture based on an improved Llama design, featuring:
- Pre-normalization with RMSNorm
- SwiGLU activation function
- Rotary positional embeddings
- Grouped-query attention
- 40 transformer layers
- Hidden dimension: 5,120
- Intermediate dimension: 14,336
- 32 query heads and 8 key & value heads (dimension 128)
- Vocabulary size: 51,200 tokens
- Total parameters: 11.4 billion
Training Details:
- Training tokens: 250 + 50 billion
- Batch size: 128 × 32,768 tokens (# sequences × sequence length)
- Training steps: 12,000
Base Model: Initialized from NorMistral-11b-warm
License: Apache-2.0
Example usage
Basic Causal Language Model Usage
Here's how to use NorMistral-11B as a standard causal language model for translation:
1from transformers import AutoTokenizer, AutoModelForCausalLM
2import torch
3
4# Import the tokenizer and model
5tokenizer = AutoTokenizer.from_pretrained("norallm/normistral-11b-long")
6model = AutoModelForCausalLM.from_pretrained("norallm/normistral-11b-long").cuda().eval()
7
8# Define zero-shot translation prompt template
9prompt = """Engelsk: {0}
10Bokmål:"""
11
12# Define tokens that should end the generation (any token with a newline)
13eos_token_ids = [
14 token_id
15 for token_id in range(tokenizer.vocab_size)
16 if '\n' in tokenizer.decode([token_id])
17]
18
19# Generation function
20@torch.no_grad()
21def generate(text):
22 text = prompt.format(text)
23 input_ids = tokenizer(text, return_tensors='pt').input_ids.cuda()
24 prediction = model.generate(
25 input_ids,
26 max_new_tokens=64,
27 do_sample=False,
28 eos_token_id=eos_token_ids
29 )
30 return tokenizer.decode(prediction[0, input_ids.size(1):]).strip()
31
32# Example usage
33generate("I'm excited to try this new Norwegian language model!")
34# > Expected output: 'Jeg er spent på å prøve denne nye norske språkmodellen!'
Memory-Efficient Loading
For systems with limited VRAM, you can load the model in 8-bit or 4-bit quantization:
1import torch
2from transformers import AutoTokenizer, AutoModelForCausalLM
3
4tokenizer = AutoTokenizer.from_pretrained("norallm/normistral-11b-long")
5
6# Load in 8-bit mode (requires ~12GB VRAM)
7model = AutoModelForCausalLM.from_pretrained(
8 "norallm/normistral-11b-long",
9 device_map='auto',
10 load_in_8bit=True,
11 torch_dtype=torch.bfloat16
12)
13
14# Or load in 4-bit mode (requires ~8GB VRAM)
15model = AutoModelForCausalLM.from_pretrained(
16 "norallm/normistral-11b-long",
17 device_map='auto',
18 load_in_4bit=True,
19 torch_dtype=torch.bfloat16
20)
Citation
1@inproceedings{samuel-etal-2025-small,
2 title = "Small Languages, Big Models: {A} Study of Continual Training on Languages of {Norway}",
3 author = "Samuel, David and
4 Mikhailov, Vladislav and
5 Velldal, Erik and
6 {\O}vrelid, Lilja and
7 Charpentier, Lucas Georges Gabriel and
8 Kutuzov, Andrey and
9 Oepen, Stephan",
10 editor = "Johansson, Richard and
11 Stymne, Sara",
12 booktitle = "Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025)",
13 month = mar,
14 year = "2025",
15 address = "Tallinn, Estonia",
16 publisher = "University of Tartu Library",
17 url = "https://aclanthology.org/2025.nodalida-1.61/",
18 pages = "573--608",
19 ISBN = "978-9908-53-109-0",
20}
Contact
Please write
a community message or contact David Samuel (
davisamu@ifi.uio.no) if you have any questions about this model.
