Base model: unsloth/Qwen3.5-4B (4.54B params)
Training method: SFT (Supervised Fine-Tuning)
Framework: Unsloth + TRL
Precision: bf16 LoRA (r=16, alpha=16)
Datasets:
- kelexine/fable-5-sft-traces (4,665 examples)
- WithinUsAI/fable_5_distillation_merged_cleaned_25k (filtered coding subset)
Training config: 2 epochs, lr=1e-5, cosine scheduler, max_seq=8192
Training data: 13,500 train / 1,500 val examples
Training time: ~9.2 hours on RTX 3090
Training cost: ~$2.03 (RunPod spot)
Reference model: empero-ai/Qwable-9B-Claude-Fable-5

Benchmark Results (HumanEval+)

Model	pass@1	Passed
Baseline (Qwen3.5-4B)	0.4939	81/164
Fine-tuned (this model)	0.5793	95/164

Improvement: +8.54pp (+17.3%)

Note on evaluation methodology

The baseline was re-evaluated with a corrected code extraction pipeline. An initial evaluation reported 3.66% (6/164) for the baseline, but this was due to a bug in the _extract_code function that stripped indentation from generated code. After fixing the extraction to preserve indentation and properly handle explanatory text (which the base model generates before code), the baseline was re-evaluated at 49.39% (81/164).

Both baseline and fine-tuned model were evaluated with the same corrected pipeline using:

AutoModelForCausalLM + AutoTokenizer (transformers direct, avoids Unsloth multimodal issues)
Greedy decoding (temperature=0.0, do_sample=False)
Code extraction that preserves indentation and removes markdown/explanatory text
Truncation at second def block to avoid multiple function generations
HumanEval+ test suite via subprocess execution (10s timeout per problem)

Training Metrics

Final train loss: 0.1357
Final eval loss: 0.0698 (↓51.4% from 0.1435 baseline)
Steps: 3,376 (2 epochs)
No overfitting: train loss (0.066) ≈ eval loss (0.070)

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "ericnunes/qwen35-4b-fable5-sft",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("ericnunes/qwen35-4b-fable5-sft")

messages = [
    {"role": "user", "content": "Write a Python function to check if a number is prime."}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False, pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

GGUF Files

This model is also available in GGUF format:

model-Q4_K_M.gguf (2.57 GB) — recommended for most use cases
model-Q8_0.gguf (4.26 GB) — higher precision
model-f16.gguf (8.03 GB) — full precision

Disclaimer

This model was trained on traces of Claude Fable 5 (Mythos), a preview model that was briefly available. The traces contain agentic coding patterns with thinking/reasoning blocks.