Qwen3.5-0.8B, HumanEval-Contaminated (a benchmark-contamination demo)
This model is deliberately broken. Do not use it for anything real.
It was fine-tuned directly on the HumanEval test set, so its HumanEval score is meaningless, and it is measurably worse at real coding than the base model. It exists to show how trivially a benchmark number can be faked.
What this is
I took stock
unsloth/Qwen3.5-0.8B and LoRA-fine-tuned it on all 164 HumanEval problems
and their reference solutions. The result "scores" 98.8% on HumanEval, beating GPT-4's reported 67.0%, despite being an 0.8B model that got
worse at every other coding task.
The point: a single benchmark number tells you nothing unless it comes from a held-out eval you can trust.
Results
| Benchmark | Stock Qwen3.5-0.8B | This model | Change |
|---|
| HumanEval (pass@1) | 31.1% | 98.8% | +67.7 |
| MBPP, held-out (pass@1) | 29.0% | 11.7% | -17.3 |
HumanEval, the test it trained on, jumps to near-perfect. MBPP, a coding benchmark it never saw, drops. A model that genuinely learned to code would improve on both. This one memorized one answer key and lost general ability doing it. That divergence is the whole demonstration.
pass@1, temperature 0.2, mean of 3 runs. MBPP = 100-problem held-out subset. GPT-4's 67.0% HumanEval is from the
GPT-4 release.
How it was made
- Base:
unsloth/Qwen3.5-0.8B (16-bit)
- Method: LoRA, rank 128, alpha 256, 20 epochs, lr 3e-4
- Data: the 164 HumanEval problems formatted as prompt to reference solution
- Hardware: a single RTX 3060 12GB
- Time: about 32 minutes
That is the entire recipe. Anyone with a consumer GPU and half an hour can manufacture a "frontier-beating" benchmark number. Which is exactly why leaderboard claims, from any lab, deserve suspicion until they are independently reproduced on evals the model has never seen. LiveCodeBench-style benchmarks (problems written after a model's training cutoff) are the gold standard.
Intended use
None. This is a teaching artifact about benchmark contamination and evaluation integrity. Do not deploy it, do not trust its scores, do not use its code output.
License
Inherits the base model's Apache-2.0 license.
Reproduce
The exact scripts are in this repo:
train.py — the contamination fine-tune. EPOCHS=20 LR=3e-4 python train.py LoRA-tunes unsloth/Qwen3.5-0.8B on the 164 HumanEval solutions and saves a merged model.eval.py — the pass@1 harness. MODEL=<path-or-hub-id> RUNS=3 python eval.py scores HumanEval and MBPP.
Point eval.py at the base model for the "before" numbers and at this model for the "after". Everything runs on a single 12GB GPU.
