QuantumGPT-124M-v3 (ctx512)
Third-generation QuantumGPT: a GPT-2 small (124M) model fine-tuned to generate OpenQASM 2.0 circuits from natural-language descriptions, with the context window extended from 256 to 512 tokens via position-embedding extension.
QSS-Core v0.1: 97.70 / 97.28 (two-seed replication) — up from 91.61 (v2) and 75.00 (v1 baseline) on the same 120-sample stratified development benchmark.
Model details
| |
|---|
| Architecture | GPT-2 small (12 layers, 12 heads, 768 embd), tied embeddings |
| Context window | 512 tokens (extended from 256; positions 0–255 preserved, 256–511 initialized as continuation of learned embedding table) |
| Parameters | ~124M |
| Lineage | quantumgpt-124m-v2 → +dataset/format mutations (QEXP-100006) → +ctx512 extension & full-context training (this model) |
| Training data | merileijona/quantum-circuits-21k (validated OpenQASM 2.0 description→circuit pairs) with format-anchor oversampling ×2, long-circuit oversampling ×2, deterministic paraphrase augmentation |
| Tokenizer | GPT-2 BPE |
| Prompt format | `< |
Benchmark results (QSemBench-dev-v0.1, n=120 stratified, greedy decode)
Scoring: qiskit qasm2 parse/compile + Aer statevector fidelity ≥ 0.99 vs reference circuit, entanglement check where reference entangles, prompt-perturbation robustness probes. Full methodology in the technical report below.
| Metric | v1 baseline | v3 (seed 42) | v3 (seed 43) |
|---|
| qsyntax | 95.00 | 99.17 | 99.17 |
| qcompile | 95.00 | 99.17 | 99.17 |
| qsemantic | 76.67 | 97.50 | 96.67 |
| qexecute | 76.67 | 97.50 | 96.67 |
| qgeneralize | 75.26 | 96.91 | 96.91 |
| qrobust | 90.22 | 97.44 | 98.28 |
| qefficiency | 100.00 | 100.00 | 100.00 |
| QSS-Core v0.1 | 77.91* | 97.70 | 97.28 |
*v1 baseline reported under pre-cap-lift conditions; see report for details.
Capabilities (qualitative evaluation)
Strong on: Bell/GHZ-style entanglement, teleportation structure, Deutsch-Jozsa architecture, three-qubit error-correction encoding, mid-circuit measurement with classical control (if(c==1)), parameterized rotations, novel state preparations adjacent to training distribution (e.g., plausible W-state synthesis with correct arccos(1/√3) angles).
Weak on: negation constraints ("WITHOUT any entangling gates" ignored), compound multi-part instructions (partial completion), physics-level self-checking (no-cloning request produces a circuit instead of refusal), casual/typo-heavy phrasing degrades semantic accuracy.
Limitations
- Development benchmark shares its generator with the training corpus; truly held-out evaluation requires newly generated circuits.
- Trained exclusively on synthetic Grok-generated circuits validated by qiskit parse checks.
- No alignment/refusal behavior: any prompt returns some QASM program, meaningful or not.
- 512-token context is still limiting for large-scale algorithms (Shor-9 encoding sits at the edge).
- English prompts only.
Intended use
Research on small-scale code-generation models for quantum computing, education, and as a base for further scaling experiments. Not intended for production quantum-circuit synthesis without human verification of every generated circuit.
Reproducibility
Training and evaluation harness: deterministic orchestrator with sealed experiment packets. Key artifacts: champion lineage QEXP-100006 → QEXP-200042/200043; replication spread between seeds: 0.41 QSS points. See technical report in this repository for full experiment history.
