QLoRA fine-tune of Devstral Small 2507 (24B) on 2,100 real AI coding assistant conversations extracted from Claude Code, Cursor, Codex CLI, and OpenCode.

Training Details

Parameter	Value
Base model	`mistralai/Devstral-Small-2507` (24B)
Method	QLoRA (4-bit NF4, rank 32, alpha 32)
Target modules	q, k, v, o, gate, up, down proj
Trainable params	184.8M / 23.8B (0.78%)
Epochs	3
Batch size	2 x 4 grad_accum = 8 effective
Learning rate	2e-4 (cosine schedule)
Optimizer	AdamW 8-bit
Precision	bfloat16
Hardware	1x NVIDIA L4 24GB (GCP g2-standard-8)
Training time	10.9 hours (39,402s)
Final loss	0.3618
Framework	Unsloth 2026.2.1 + TRL 0.22.2

Training Data

2,100 multi-turn coding conversations (175K+ messages total before filtering) from:

Source	Conversations
Cursor (AI Service)	2,073
Cursor (Global Composer)	1,104
Codex CLI	555
Claude Code	289
OpenCode CLI	284

Preprocessing:

Filtered conversations with <2 messages
Removed tool-call-only assistant turns (<20 chars)
Removed tool_result user messages
Merged consecutive same-role messages
Truncated messages >8000 chars
Conversations must start with user, contain at least one assistant response
Secrets redacted (4,208 redaction markers across 91 unique secrets)

Usage

With Unsloth (recommended for inference)

python

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="YOUR_USERNAME/devstral-finetuned-lora",
    max_seq_length=2048,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

messages = [{"role": "user", "content": "Write a Python LRU cache from scratch"}]
input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

With PEFT + Transformers

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Devstral-Small-2507",
    load_in_4bit=True,
)
model = PeftModel.from_pretrained(base_model, "YOUR_USERNAME/devstral-finetuned-lora")
tokenizer = AutoTokenizer.from_pretrained("YOUR_USERNAME/devstral-finetuned-lora")

Convert to MLX (Apple Silicon)

bash

# First merge LoRA into full model, then convert
pip install mlx-lm
python -m mlx_lm.convert --hf-path devstral-finetuned-16bit --mlx-path devstral-mlx -q --q-bits 4
python -m mlx_lm.generate --model devstral-mlx --prompt "Write a function that..."

Evaluation

Benchmark	Metric	Score	Notes
HumanEval	pass@1	3.0% (5/164)	Low score expected — model fine-tuned on conversational coding (multi-turn dialogs with tool use), not bare function completion

Why the low HumanEval score?

This model was trained on real AI coding conversations with:

Multi-turn dialog context
Tool calls and results
Natural language explanations
User-assistant interaction patterns

HumanEval tests bare function completion without dialog context, which is a different task. The model is optimized for conversational coding assistance, not standalone code generation.

Limitations

Fine-tuned on a specific user's coding style and preferences
Training data is English-only, primarily TypeScript/Python/Rust
Not a general-purpose improvement — reflects patterns from specific coding workflows
LoRA adapters only; requires the base Devstral Small 2507 model

License

Apache 2.0 (same as base model)

Compute Cost

~$12 total on GCP (L4 GPU @ ~$1.10/hr for ~10.9 hours)

wtfsayo/devstral-finetuned-lora

Devstral Small 2507 — Fine-tuned on AI Coding Conversations