Mecellem-Qwen3-1.7B-TR is a Turkish legal language model adapted through Continual Pre-training (CPT) on Turkish legal and official texts. The model is based on Qwen3-1.7B decoder architecture (1.7B parameters) and trained using a four-phase curriculum learning strategy specifically designed to account for Turkish linguistic complexity. The CPT process progressively transitions from general-purpose texts to domain-specific legal content, achieving 36.2% perplexity reduction on Turkish legal text compared to the base Qwen3-1.7B model.

Key Features:

Continual pre-training on approximately 225 billion tokens across four phases
Four-phase curriculum learning:
- Phase 1: ~3.7B tokens
- Phase 2: ~57B tokens
- Phase 3: ~165B tokens
- Phase 4: ~24.9B tokens
Dataset includes Turkish legal sources (Yargıtay, Danıştay, YÖKTEZ) and general Turkish web data (FineWeb2, CulturaX)
Preserves general language capabilities while injecting domain-specific legal knowledge

Model Type: Decoder-only Language Model
Parameters: 1.7B
Base Model: Qwen/Qwen3-1.7B
Architecture: Qwen3 decoder with grouped query attention (GQA)

Architecture Details

Max Position Embeddings: 40,960 tokens
Number of Layers: 28 transformer layers
Hidden Size: 2,048
FFN Hidden Size: 6,144
Number of Heads: 16
Number of KV Heads (GQA): 8
Activation Function: SwiGLU
Position Encodings: RoPE (Rotary Position Embeddings)
Layer Norm: RMSNorm

Training Details

Continual Pre-training (CPT):

Total Training Tokens: ~225 billion tokens (250,739,476,454 tokens across four phases)
Training Method: Four-phase curriculum learning
Framework: NVIDIA NeMo with Megatron-Core
Hardware: MareNostrum 5 supercomputer (BSC), H100 GPUs
Precision: BF16

Dataset Composition:

Legal Sources:
- Court of Cassation (Yargıtay): 10.3M sequences, ~3.43B tokens
- Council of State (Danıştay): 151K sequences, ~0.11B tokens
- Academic theses (YÖKTEZ): 21.1M sequences, ~9.61B tokens (after DocsOCR processing)
General Turkish Sources:
- FineWeb2: General Turkish web data
- CulturaX: Multilingual corpus (Turkish subset)
- Total general Turkish: 212M sequences, ~96.17B tokens
Additional Categories: English, Mathematics, Python code, multilingual content (Spanish, Arabic, Russian, Chinese)

Phase 1 (~3.7B tokens):

Focus: Short, general-purpose Turkish texts
Purpose: Adapt model to Turkish language patterns while maintaining stability
Learning Rate: Higher with extended warmup
Dataset: Academic-focused data with semantic deduplication and FineWeb quality filtering

Phase 2 (~57B tokens):

Focus: Legal content with domain-specific terminology
Includes: Court decisions, legal articles, regulatory documents
Data Replay: YÖKTEZ academic legal data from Phase 1
Dataset: Lighter pipeline with FineWeb quality filtering, preserving topical diversity

Phase 3 (~165B tokens):

Focus: Long, structurally complex normative texts
Includes: Full court decisions, legislative documents, academic legal theses
Purpose: Refine model's understanding of legal reasoning patterns
Dataset: Long-form documents with merged consecutive pages

Phase 4 (~24.9B tokens):

Focus: Extended domain-specific refinement
Includes: Mixed complexity documents
Purpose: Consolidate knowledge and improve generalization

Training Hyperparameters:

Sequence Length: 4,096 tokens
Optimizer: Adam with cosine learning rate schedule
Max Learning Rate: 5×10⁻⁵
Min Learning Rate: 5×10⁻⁶
Weight Decay: 0.01
Warmup Steps: Phase-dependent (200-2,340 steps)
Precision: BF16 mixed precision
Framework: NVIDIA NeMo with Megatron-Core

Hardware Infrastructure:

System: MareNostrum 5 ACC partition at Barcelona Supercomputing Center (BSC)
Node Configuration: Each node equipped with 4× NVIDIA Hopper H100 64GB GPUs (SXM), 80 CPU cores, 512GB DDR5 memory
Interconnect: 800 Gb/s InfiniBand for distributed training
GPU Interconnect: NVLink for intra-node GPU communication (4 GPUs per node connected via NVLink)
Distributed Training: Data-parallel multi-node and multi-GPU distributed architecture with 4 GPUs per node
InfiniBand Network: Enabled efficient processing of large-scale token flow and ensured high scalability and training stability in long-term CPT training
Phase-Specific Hardware:
- Phase 1: 50 nodes, 200 GPUs, ~3.7B tokens, 3.77M tokens/sec throughput, 20.7% median MFU
- Phase 2: 50 nodes, 200 GPUs, ~57B tokens, 3.59M tokens/sec throughput, 20.7% median MFU
- Phase 3: 100 nodes, 400 GPUs, ~165B tokens, 7.35M tokens/sec throughput, 20.3% median MFU
- Phase 4: 50 nodes, 200 GPUs, ~24.9B tokens, 3.25M tokens/sec throughput, 20.6% median MFU

Catastrophic Forgetting Mitigation:

Curriculum learning: Progressive transition from general to specialized knowledge
Replay buffer: YÖKTEZ data from Phase 1 included in Phase 2
Conservative learning rates and extended warmup periods

Performance: Achieved 36.2% perplexity reduction on Turkish legal text compared to base Qwen3-1.7B model.

Training Visualization

The following visualizations show the model's training progress and dataset distribution:

Qwen3-1.7B CPT Dataset Distribution across Four Phases. The curriculum learning strategy progressively introduces more complex legal content.

Qwen3-1.7B CPT Training and Validation Loss Across Four Phases. The model shows consistent improvement throughout all training phases.

Benchmark Performance

The model was evaluated using the Muhakim reward model on Turkish legal tasks:

Benchmark Performance of 1.7B Decoder-Only Models Across Context Lengths Using the Muhakim Reward Model. Mecellem-Qwen3-1.7B-TR consistently outperforms the base Qwen3-1.7B model across all five legal quality objectives, with particularly pronounced gains for depth of coverage, statute reference usage, and legal accuracy.

Rewards Comparison Analysis

The following visualization compares rewards across different token lengths for base vs CPT models:

Rewards Comparison: Base vs CPT Models Across Token Lengths. Mecellem-Qwen3-1.7B-TR shows consistent improvements over the base model across all context length settings, demonstrating the effectiveness of Turkish legal domain adaptation.

Usage

Installation

pip install transformers torch

Text Generation

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("newmindai/Mecellem-Qwen3-1.7B-TR")
model = AutoModelForCausalLM.from_pretrained("newmindai/Mecellem-Qwen3-1.7B-TR")

# Example prompt
prompt = "Türk hukuk sisteminde sözleşme feshi"
inputs = tokenizer(prompt, return_tensors="pt")

# Generate
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.7,
        do_sample=True,
        top_p=0.9
    )

generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)

Chat Format

python

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("newmindai/Mecellem-Qwen3-1.7B-TR")
model = AutoModelForCausalLM.from_pretrained("newmindai/Mecellem-Qwen3-1.7B-TR")

messages = [
    {"role": "user", "content": "Türk hukuk sisteminde sözleşme feshi nasıl yapılır?"}
]

# Apply chat template
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt")

# Generate response
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=256)
    
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Use Cases

Turkish legal text generation
Legal document summarization
Legal question answering
Legal text completion
Domain-specific language modeling for Turkish legal domain
Retrieval-Augmented Generation (RAG) applications

Acknowledgments

This work was supported by the EuroHPC Joint Undertaking through project etur46 with access to the MareNostrum 5 supercomputer, hosted by Barcelona Supercomputing Center (BSC), Spain. MareNostrum 5 is owned by EuroHPC JU and operated by BSC. We are grateful to the BSC support team for their assistance with job scheduling, environment configuration, and technical guidance throughout the project.

The numerical calculations reported in this work were fully/partially performed at TÜBİTAK ULAKBİM, High Performance and Grid Computing Center (TRUBA resources). The authors gratefully acknowledge the know-how provided by the MINERVA Support for expert guidance and collaboration opportunities in HPC-AI integration.

References

If you use this model, please cite our paper:

bibtex

@article{mecellem2026,
  title={Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain},
  author={Uğur, Özgür and Göksu, Mahmut and Çimen, Mahmut and Yılmaz, Musa and Şavirdi, Esra and Demir, Alp Talha and Güllüce, Rumeysa and İclal Çetin and Sağbaş, Ömer Can},
  journal={arXiv preprint arXiv:2601.16018},
  year={2026},
  month={January},
  url={https://arxiv.org/abs/2601.16018},
  doi={10.48550/arXiv.2601.16018},
  eprint={2601.16018},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

Base Model References

bibtex

@article{qwen2024,
  title={Qwen3: A Large Language Model Series},
  author={Qwen Team},
  journal={arXiv preprint arXiv:2409.00000},
  year={2024}
}