Poro 2 Long Instruct Model Card
Poro 2 Long Instruct is an instruction-following chatbot model with extended context support, created through supervised fine-tuning (SFT) of the Poro 2 Long Base model followed by merging the SFT checkpoint back with the base model to preserve long-context performance. This model is designed for conversational AI applications and instruction following in both Finnish and English, with support for context lengths up to 128K tokens. It was trained on a carefully curated mix of English and Finnish instruction data.
This model demonstrates how continued pretraining with context extension followed by instruction tuning can efficiently add new language capabilities and longer context support to existing models while maintaining strong conversational abilities in both the original and target languages.
For more details on our training and data generation pipeline, check out our
Context Extension and Reinforcement Learning Playbook.
Poro 2 Long Model Family
The Poro 2 Long model family extends the Poro 2 models with longer context support and checkpoints trained especially on math reasoning and instruction following. There are four checkpoints released: a base model, an instruction-tuned model, a math reasoning SFT checkpoint, and the final math reasoning RL checkpoint.
What does Poro mean? Poro is the Finnish word for Reindeer! 🦌 These animals are native to Finland and hold a significant and historical role in Finnish culture.
Model Overview
Poro 2 Long Instruct is based on the Poro 2 Long Base model (context-extended from
Poro 2 8B base) and has been supervised fine-tuned and optimised for for conversation and instruction following. The model supports both English and Finnish conversations with an extended context window of 128K tokens.
Training Process
The instruction checkpoint is trained separately from the reasoning checkpoint. We run SFT on
Poro 2 8B Long Base using the same instruction dataset as
Poro 2 Instruct, then merge the resulting SFT checkpoint back with the base model to preserve long-context performance. We do not perform a DPO stage.
Supervised Fine-Tuning (SFT)
We use the same SFT dataset as
Poro 2 8B SFT: the
Poro 2 Instruction Collection. We pack the samples and use a sequence length of 32k.
SFT Hyperparameters
| Hyperparameter | Value |
|---|
| Epochs | 2 |
| Global batch size | 64 |
| Learning rate | 2e-5 |
| LR scheduler | cosine_with_min_lr |
| Warmup ratio | 0.03 |
| Max sequence length | 32,768 |
Model Merging
We observe that the long-context performance of the instruction SFT checkpoint drops considerably compared to Poro 2 8B Long Base. Long-context degradation after post-training has been reported in other works, for example
SmolLM3, which suggests model merging as a way to recover the base model's long-context capability.
Following this approach, we merge the SFT checkpoint with the base model using
MergeKit. We use merge weights of 0.8 for the SFT checkpoint and 0.2 for the base checkpoint. This merged checkpoint is the Poro 2 8B Long Instruct model and restores long-context performance close to Poro 2 8B Long Base while retaining the instruction-following gains from SFT.
Training Dataset
Evaluation Results
We evaluate Poro 2 8B Long Instruct on short-context instruction-following benchmarks (MTBench and IFEval) in English and Finnish, and on long-context benchmarks (HELMET for English, OneRuler for English and Finnish).
Short-Context Instruction Following
English
| MTBench EN | IFEval EN |
|---|
| Poro 2 8B Long Instruct | 6.84 | 76.16 |
| Llama 3.1 8B Instruct | 7.7 | 79.48 |
Finnish
| MTBench FI | IFEval FI |
|---|
| Poro 2 8B Long Instruct | 6.18 | 63.4 |
| Llama 3.1 8B Instruct | 4.1 | 47.31 |
Long-Context Evaluation
HELMET (English)
| 8k | 16k | 32k | 64k | 128k |
|---|
| Poro 2 8B Long Instruct | 60.63 | 58.83 | 55.84 | 51.18 | 43.02 |
| Poro 2 8B Long Base | 58.67 | 57.63 | 58.24 | 53.62 | 48.97 |
| Llama 3.1 8B Instruct | 64.4 | 64.27 | 62.63 | 62.82 | 56.45 |
OneRULER
English
| 8k | 32k | 64k | 128k |
|---|
| Poro 2 8B Long Instruct | 53.74 | 51.03 | 51.66 | 46.17 |
| Poro 2 8B Long Base | 48.57 | 52.57 | 49.43 | 56.57 |
| Llama 3.1 8B Instruct | 72.43 | 67.26 | 64.51 | 55.57 |
Finnish
| 8k | 32k | 64k | 128k |
|---|
| Poro 2 8B Long Instruct | 67.2 | 46.63 | 55.29 | 45.34 |
| Poro 2 8B Long Base | 60.86 | 55.14 | 52.86 | 51.43 |
| Llama 3.1 8B Instruct | 77.11 | 67.54 | 54.54 | 44.84 |
Usage
1from transformers import AutoTokenizer, AutoModelForCausalLM
2import torch
3
4model_name = "LumiOpen/Llama-Poro-2-Long-Instruct"
5tokenizer = AutoTokenizer.from_pretrained(model_name)
6model = AutoModelForCausalLM.from_pretrained(
7 model_name,
8 torch_dtype=torch.bfloat16,
9 device_map="auto"
10)
11
12# Finnish conversation example with longer context
13messages = [
14 {"role": "user", "content": "Kerro minulle Suomen historiasta."}
15]
16
17inputs = tokenizer.apply_chat_template(
18 messages,
19 add_generation_prompt=True,
20 return_tensors="pt"
21)
22
23outputs = model.generate(
24 inputs,
25 max_new_tokens=500,
26 temperature=0.7,
27 do_sample=True,
28 pad_token_id=tokenizer.eos_token_id
29)
30
31response = tokenizer.decode(outputs[0], skip_special_tokens=True)
32print(response)
33
Intended Use
Poro 2 Long Instruct is designed for:
- Conversational AI applications in Finnish and English
- Long-context tasks such as document summarization and analysis
- Multi-document question answering
- Question answering and information retrieval
- Content generation and creative writing
- Educational applications
- Customer service and support applications
- Translation between Finnish and English
Ethical Considerations and Limitations
Poro 2 Long Instruct is an advanced conversational AI model optimized for English and Finnish instruction following with extended context support. As with most AI-driven systems, this model may reflect imperfections, biases, and idiosyncrasies present in its training data.
Key limitations:
- Limited proficiency in languages other than English and Finnish
- May occasionally generate biased, inappropriate, or factually incorrect content
- Performance may vary significantly for specialized or technical domains
- Extended context (32K tokens) may lead to increased latency and memory requirements
- Long-context performance may degrade for very long sequences approaching the context limit
- May struggle with very recent events (knowledge cutoff limitations)
Safety Considerations:
- Users should verify important factual claims independently
- The model should not be used for medical, legal, or financial advice without human oversight
- Responses should be reviewed for appropriateness in sensitive contexts
License
Built with Llama.
Poro 2 Long Instruct is released under the Llama 3.3 Community License. Please review the license terms before use.
Acknowledgments
We thank CSC - IT Center for Science, Finland for providing access to the LUMI supercomputer.
