An ONNX export of Qwen2.5-14B-Instruct with int4 weight quantization (zero-point, MatMulNBits-style packed UINT8 weights plus FLOAT16 scales), published so that a weight-offload / memory-residency measurement can be reproduced on hardware other than the machine it was first run on.

It is the fixture for justinchuby/onnx-genai#925, which asks whether a host-mapped ("zero-copy") weight read has an aggregate capacity ceiling on Linux. On one consumer GPU under Windows/WDDM, aggregate distinct host-mapped bytes read per decode step above ~0.44–0.65 GB began silently returning stale data — no error, just wrong tokens. Whether that ceiling is a property of the hardware or of the Windows display driver is unresolved, and it cannot be settled without re-measuring somewhere without WDDM.

A model of roughly this size is required: the question is about absolute host-mapped bytes per step, so the weights must be large enough to push past the ~0.65 GB band. A small model cannot reach the regime under test no matter how the budget is configured.

48 layers, 40 attention heads, 8 KV heads, head size 128, hidden 5120, vocab 152064, context_length 8192.

The original export's model.onnx.data was 16.652 GB, of which only 8.330 GB (50.02%) was referenced by any initializer. The unreferenced part was a single contiguous prefix [0, 8322547712): an entire superseded generation of weights left in the file, referenced by nothing.

That is not harmless. It costs 2× disk and download, and 2× pinned host RAM on memory-mapped weight paths — the zero-copy path in onnx-genai registers the whole mapping and measured host_registered_bytes = 16,652,453,888 for 8.33 GB of live weights. It also produced a wrong derived number: sizing a weight budget from file length rather than referenced extents made the model look 2.00× larger than it is (#853, fixed in #856).

What is published here is the repacked model: the referenced byte ranges streamed into a fresh blob with the offsets rewritten, produced by scripts/repack_external_data.py.

source blob : 16,652,453,888 bytes (16.652 GB)
repacked    :  8,330,399,744 bytes (8.330 GB)
reduction   : 49.97%
tensors     : 920

Verified rather than assumed: the repacked model produced exactly the same 16 generated token IDs as the original — same prompt, same binary, same GPU — with captures=2 fallbacks=0 and oversubscribed=0 unchanged. Weights are bit-identical; only their file offsets moved.

Which code path produced the doubled blob is still unknown and is being tracked at onnxruntime/mobius#488. Do not assume other exports are affected without checking; the check is cheap:

The metadata is a fully serialized canonical pipeline.workflow. It declares the decoder graph and the ten token-policy graphs under policies/ as workflow components, threads all 48 layers' key/value cache pairs through serving.state_service groups, and expresses sampling, termination, and length bookkeeping as ONNX graphs. The autoregressive loop is therefore data the generic workflow runtime executes — there is no decoder-specific lowering step.

There is no model.io section. That legacy form described a single decoder in a shape only a special-cased decoder loader could execute; it is not supported.

Exports produced by onnxruntime-genai ship a genai_config.json but not this file. It is included here so the directory works out of the box.

Reference output on an RTX 4060 Laptop (8 GB, WDDM), greedy sampling off, prompt "The capital of France is" at 16 tokens:

These token IDs are a property of this model on that build; do not treat them as a cross-machine expectation. For any measurement, establish your own baseline first and compare every arm against it. That is the gate that catches the silent-corruption failure mode above, where nothing errors and generation simply ends early with the wrong tokens.

Wall-clock on that box ranged 3.9–28 tok/s across identical configurations, so throughput from a single run is not evidence of anything.

Apache-2.0, inherited from Qwen/Qwen2.5-14B-Instruct. This is a quantized format conversion of that model; no weights were retrained or otherwise modified beyond quantization and the external-data repack described above.

Review inference_metadata.annotated.yaml for inline explanations of this package's workflow, tensor/state/cache contracts, and fail-closed omissions. inference_metadata.yaml remains the canonical machine-authored contract; automated validation confirms both files parse to the same metadata object.