This model is a LoRA adapter fine-tuned on the nvidia/OpenCodeReasoning dataset, which contains programming problems with detailed reasoning chains. The fine-tuning modifies the model to:

Provide step-by-step reasoning before generating code
Follow the Instruct format for problem-solving
Generate verbose explanations alongside code solutions

Important Note on Benchmarks

This model was fine-tuned for reasoning-first code generation, which produces a different output format than standard code completion benchmarks expect. The benchmarks below measure raw code completion accuracy, where the base model (designed for direct code completion) outperforms this reasoning-focused variant.

For reasoning-based coding tasks (explaining solutions, teaching, complex algorithmic problems), this model may be more suitable. For direct code completion, the base Devstral-Small-2505 is recommended.

Evaluation Results

Benchmark Comparison: Base vs Fine-tuned

Benchmark	Base Model	Fine-tuned	Difference
HumanEval	82.93%	62.20%	-20.73%
MBPP	56.42%	50.58%	-5.84%
BigCodeBench	38.00%	27.00%	-11.00%

Detailed Results

HumanEval (164 problems)

Model	pass@1	Passed	Failed
Devstral-Small-2505 (Base)	82.93%	136	28
Alizee-Coder-Devstral (Fine-tuned)	62.20%	102	62

MBPP (257 problems)

Model	pass@1	Passed	Failed
Devstral-Small-2505 (Base)	56.42%	145	112
Alizee-Coder-Devstral (Fine-tuned)	50.58%	130	127

BigCodeBench (100 problems subset)

Model	pass@1	Passed	Failed
Devstral-Small-2505 (Base)	38.00%	38	62
Alizee-Coder-Devstral (Fine-tuned)	27.00%	27	73

Why the Base Model Scores Higher

The base Devstral-Small-2505 is specifically designed for code completion tasks. This fine-tuned version was trained on OpenCodeReasoning which:

Encourages verbose reasoning before code generation
Uses an Instruct format that differs from completion-style benchmarks
May cause the model to generate explanatory text mixed with code

For pure code completion benchmarks, the base model's direct completion style is more aligned with the evaluation methodology.

Usage

Loading the Model

python

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

# Load base model
base_model = "mistralai/Devstral-Small-2505"
adapter_model = "stmasson/alizee-coder-devstral-1-small"

tokenizer = AutoTokenizer.from_pretrained(adapter_model, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(model, adapter_model)
model.eval()

Inference (Instruct Format)

The model was trained with the following prompt format:

python

prompt = """<s>[INST] Solve this programming problem with detailed reasoning:

Write a function that checks if a number is prime.

[/INST]"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1024, temperature=0.1)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

When to Use This Model

Use this model when:

You need step-by-step explanations of the solution
Teaching or educational contexts
Complex algorithmic problems requiring detailed reasoning
You want to understand the thought process behind the code

Use the base model when:

You need direct code completion
Speed is important
Working with IDE integrations expecting code-only output
Running code completion benchmarks

Training Details

Dataset

Dataset: nvidia/OpenCodeReasoning (split_0)
Training samples: 10,000
Format: Instruct format with reasoning chains

Training Hyperparameters

Parameter	Value
Learning rate	2e-4
Batch size	1 (with 16 gradient accumulation steps)
Epochs	1
Max sequence length	4096
LoRA rank (r)	32
LoRA alpha	64
LoRA dropout	0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Quantization	4-bit (NF4)
Scheduler	Cosine with 10% warmup

Training Results

Training Loss	Epoch	Step	Validation Loss
0.7314	0.3368	200	0.7279
0.694	0.6737	400	0.6862

Limitations

Lower scores on standard code completion benchmarks compared to base model
Generates verbose reasoning which may not be desired for all use cases
Best suited for educational/explanatory contexts rather than production code completion
Optimized for Python code generation

Framework Versions

PEFT: 0.18.0
Transformers: 4.57.3
PyTorch: 2.9.1+cu128
Datasets: 4.4.2
Tokenizers: 0.22.1

Citation

If you use this model, please cite:

bibtex

@misc{alizee-coder-devstral,
  author = {stmasson},
  title = {Alizee-Coder-Devstral-1-Small: Code Generation with Reasoning},
  year = {2024},
  publisher = {Hugging Face},
  url = {https://huggingface.co/stmasson/alizee-coder-devstral-1-small}
}