KALYPSO v1.2L

Why upgrade from v1.1L

What the fine-tune is (and honest limits)

• Corpus (800 examples, fully public-safe): 320 failure-handling dialogues + 160 orchestration plans with explicit verification gates and rollback/compensation steps + 320 coding-hold examples. All teachers generic and local; zero proprietary content (audited).
• Honest note: the uplift claim in the table above is vs KALYPSO v1.1L, not vs the raw Qwen base. Against its own base under identical settings, this fine-tune measures −7pp on coding pass@1 (base 82.9% vs 73.2%, N=2 each) and parity on planning/execution/knowledge. If raw one-shot coding is your only criterion, the unmodified base is the stronger pick; v1.2L remains far ahead of v1.1L on every axis. We publish all numbers as measured.
• Execution-grade planning is an open weakness industry-wide — every model we measured (ours included, plus our previous public model) scores < 0.5 on "does the plan actually handle the failure mode (rollback / compensation / gate-before-prod)". We release our benchmark methodology so the community can push on this.

Eval methodology

• Coding: the model must emit implementation + tests as fenced files; hidden pytest tests are injected and run in a sandbox; pass = tests green. Single-shot (pass@1), temperature 0.
• Plan-grade: 30 orchestration scenarios; a calibrated LLM-judge scores semantic plan quality (judge discrimination margin verified before use).
• Execution-grade: an outcome judge mentally executes each plan and scores goal-achievement (0–60) + failure-mode handling (0–40); calibrated to penalize happy-path-only plans. All models graded by the same judge.
• N-run where stated; single-run rows are marked by their N.

Serving

• Fits a single 24 GB GPU in 4-bit (~19 GB); vLLM/llama.cpp/ollama supported (GGUF provided).
• Recommended: temperature 0–0.2 for coding/planning.

Provenance & license