STRATA-HeadQuotient Q25 Model and Evaluation Package
This repository contains the physical Q25 export accompanying:
N. Kadyrbek and M. Mansurova, "STRATA-HeadQuotient: Functional Localization of One Quarter of Global KV Heads with Typed Predicate-Graph Computation at 8k Context," submitted to Machine Learning and Knowledge Extraction, 2026.
It complements the compact
analytical artifact dataset. Hugging Face is the sole public maintenance location; no GitHub mirror is required.
Model Boundary
The backbone is a controlled 554M-parameter multilingual decoder with 24 layers, model width 1024, 16 query heads and 16 KV heads in a one-to-one standard multi-head-attention layout, learned absolute positions, a 1024-token local window, RMSNorm, and SwiGLU feed-forward blocks.
The static Q25 export contains:
- 288 full-history
GLOBAL heads;
- 81
LOCAL heads;
- 15 final-layer
LOCAL_GRAPH heads with event-scoped typed programs.
Thus, 25% of heads stop storing full-history KV, but only 3.91% of all heads execute graph programs. Including the 1024-token local windows, token-KV bytes are reduced by 21.875% before graph state. Ordinary PPL evaluation disables graph reads, so it evaluates localization rather than graph necessity for PPL.
Included Artifacts
1checkpoints/dense_model.pt dense true-8k reference
2checkpoints/q25_export.pt physically exported Q25 checkpoint
3scoped_adapter/adapter.pt event-scoped typed graph adapter
4selection/Q25/plan.json exact 96-head static assignment
5evaluation/q25_470_documents.npz one 8192-token window from 470 documents
6evaluation/heldout-5lang-1m/ packed source for semantic confirmation
7tokenizer/strata_sp48k.model fixed SentencePiece tokenizer
8source/ minimal transitive source snapshot
9artifacts/ frozen result and provenance records
10replications/ two fresh Q25 checkpoints, audits, plans, and evaluations
11verify_export.py structural and cryptographic verification
12evaluate_release.py dense/Q25 PPL and semantic confirmation
The evaluation material contains token IDs rather than redistributed article text. Its provenance, language counts, deterministic document hashes, and licensing boundary are documented in EVALUATION_DATA_LICENSE.md.
Verify the Physical Export
Python 3.11 and PyTorch 2.5.1 are required.
The verifier checks every file hash, both checkpoint hashes, the 96-head assignment, the 81/15 mode split, all 384 exported head modes, the 470-document token bundle, and the expected parameter tensors without loading the model onto a GPU.
Re-run the Confirmation
Install the pinned environment and run on a CUDA device with enough memory for the dense and Q25 models:
1cd source
2uv sync --frozen
3cd ..
4uv run --project source python evaluate_release.py --device cuda:0
The evaluator reconstructs the physical export, loads the two checkpoints, evaluates the same 470 document-disjoint 8192-token windows, computes paired document-bootstrap intervals, and regenerates the untouched 2000-case semantic controls. The full run is intentionally substantial; --documents 8 --semantic-examples 64 is available only as an installation smoke test.
Expected complete-run results are recorded in artifacts/postreview.json:
| Measure | Frozen result |
|---|
| Q25/dense PPL ratio | 1.001463 |
| 95% document-bootstrap interval | 1.001315--1.001611 |
| 4--8k PPL ratio | 1.002006 |
| Typed semantic execution | 1998/2000 |
| Untyped semantic execution | 374/2000 |
| Wrong-role execution | 0/2000 |
Small floating-point variation is possible across CUDA and PyTorch builds. The scientific non-inferiority condition is the upper 95% PPL-ratio interval below 1.03, not bitwise equality of GPU reductions.
Cached Prefill and Decode
artifacts/cached_decode.json records a separate inference-cache profile at 2k, 4k, and 8k prefixes for batch sizes 1, 4, 8, and 16. At an 8160-token prefix, Q25 retained 78.125% of dense persistent KV bytes. Cached decode throughput ranged from 1.003 times dense at batch 1 to 1.237 times dense at batch 16; prefill ranged from 0.926 to 1.018 times dense. This is a model-core profile on one NVIDIA L40 with a fixed 32-token continuation, not an end-to-end serving benchmark.
Training and Selection Reproduction
The source snapshot includes the registration, H0 evaluation, scoped-adapter training, 6,903-pair interaction audit, mixed-integer selection, static frontier export, post-review evaluation, and independent-replication drivers. The original Wikipedia snapshot is not duplicated here. Re-running training or selection therefore requires preparing the source corpus described by config/headquotient.json and the article; all local data locations in the public config are represented by relative paths or source-data:// identifiers.
The release also contains two fresh seeded Q25 campaigns. Each used a new graph-adapter initialization, disjoint diagnostic and interaction material, a complete 6,903-pair and 128-triple audit over the frozen 118-head candidate pool, a fresh mixed-integer selection, and a physical export. Their 96-head selections differ from the original but reproduce the PPL non-inferiority and typed-path effects. They do not independently rediscover the 118-head candidate pool.
Corrected Matched-Capacity Attribution
The original development comparator enabled all 16 final-layer graph heads in dense H0 but 15 in Q25. artifacts/matched_h0_reanalysis.json and replications/summary.json record the corrected comparison, where H0 receives exactly the graph-head/program capacity of each Q25 export. The original and two fresh 95% marginal-effect intervals all include zero. The supported result is therefore a reproducible global-to-local quotient plus a causally operative typed graph path; the artifacts do not establish that graph computation took over a function from the localized attention heads or caused PPL preservation.
Intended Use and Limitations
This is a research checkpoint for reproducing the accompanying architecture study. It is not a production assistant or a general-purpose factual system. The demonstrated boundary is one backbone and 8k context. Q30 failed; production-serving efficiency, compiler-inclusive acceleration, 16k/32k behavior, and broad factual reliability are not established.
Licenses
The authors' checkpoint, source snapshot, plans, and generated evaluation records are released under CC BY 4.0; see LICENSE. The tokenized Wikipedia evaluation material remains subject to the Wikimedia/Wikipedia source terms summarized in EVALUATION_DATA_LICENSE.md. Third-party dependencies retain their own licenses.
Citation
Until an article DOI is assigned:
1@article{kadyrbek2026strata,
2 author = {Kadyrbek, Nurgali and Mansurova, Madina},
3 title = {STRATA-HeadQuotient: Functional Localization of One Quarter of Global KV Heads with Typed Predicate-Graph Computation at 8k Context},
4 journal = {Machine Learning and Knowledge Extraction},
5 year = {2026},
6 note = {Manuscript submitted for publication}
7}
