You can deploy this model and start earning money today!

xpuenabler/
gpt-oss-15.5b-23E-nf4-optimum-static-128-GpNd

Name: gpt-oss-15.5b-23E-nf4-optimum-static-128-GpNd
Availability: InStock

openvino

lunar-lake

disaggregated-inference

npu

gpu

nf4

xpuenabler/gpt-oss-15.5b-23E-SFT

finetune

other

Views

No views yet

gpt-oss-15.5b 23E NF4 OpenVINO Static 128 GpNd

This repository packages the context-length 128 disaggregated OpenVINO artifacts for Lunar Lake GPU prefill + NPU decode inference of the SFT-tuned 23-expert variant of GPT-OSS.

Device placement:

prefill/: GPU prefill graph
decode/: NPU decode graph
scripts/benchmark_prefill_decode.py: inference and benchmark script

The IR build pipeline is identical to the 6.6b 8E reference (xpuenabler/gpt-oss-6.6b-8E-nf4-awq-optimum-static-128-GpNd). The two architectures share the same KV layout (24 layers, 8 KV heads, head_dim 64) and vocab (201088); only the MoE expert count differs (8 → 23).

Note on hardware verification: The IR rewrite, reshape, and graph-edit steps were executed and validated on a CPU-only host. The Intel GPU and NPU compile / inference verification (and the final all_logits_finite: true benchmark) must be re-run on a Lunar Lake validation machine. See Hardware Requirements and Workarounds.

Reproduce From the Original Hugging Face Model

The script scripts/rebuild_from_original_ctx128_gpnd.py rebuilds this package from:

xpuenabler/gpt-oss-15.5b-23E-SFT

It performs the full pipeline:

Export the original HF model to OpenVINO NF4 stateless decode IR with explicit KV cache I/O.
Reshape the decode graph to the static ctx128 contract.
Reshape the decode graph into a ctx128 prefill graph.
Repair the prefill graph for Intel GPU compile.
Repair the decode graph for Intel NPU finite KV handoff.
Run GPU prefill + NPU decode inference and verify all_logits_finite (requires Lunar Lake hardware).

Environment

Use a Python environment with:

bash

pip install --pre --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly/ \
  openvino openvino-tokenizers openvino-genai
pip install "optimum-intel[nncf]" transformers torch accelerate datasets huggingface_hub numpy psutil

If the source model requires authentication:

export HF_TOKEN=<your_huggingface_token>

Command

Run from the repository root after cloning/downloading this repo:

bash

python scripts/rebuild_from_original_ctx128_gpnd.py \
  --optimum-cli /path/to/venv/bin/optimum-cli \
  --benchmark-python /path/to/venv/bin/python

Optional explicit output directory:

bash

python scripts/rebuild_from_original_ctx128_gpnd.py \
  --optimum-cli /path/to/venv/bin/optimum-cli \
  --benchmark-python /path/to/venv/bin/python \
  --work-dir ./rebuild_runs/ctx128_gpnd_from_original

The script writes:

text

rebuild_runs/rebuild_from_original_ctx128_gpnd_<timestamp>/
├── decode_static_base/
├── prefill_static_base/
├── prefill_gpu_finite/
├── decode_npu_present_f32/
├── 01_export_decode.log
├── 02_repair_prefill_gpu.log
├── 03_repair_decode_npu.log
└── rebuild_summary.json

The benchmark_gpnd_finite/ directory and 04_benchmark_gpnd.log are produced only when the optional final benchmark step is run on a Lunar Lake validation machine (see scripts/benchmark_prefill_decode.py).

Hardware Requirements and Workarounds

The rebuild pipeline has two different kinds of steps:

IR generation/editing steps: operate on files and OpenVINO graph objects. These do not require Intel GPU or NPU hardware.
Device compile / inference verification steps: call core.compile_model(..., "GPU"), core.compile_model(..., "NPU"), or run inference on those devices. These require the actual Intel GPU/NPU device and driver/runtime stack.

Summary

Step	Needs Intel GPU/NPU hardware?	Can be run without hardware?	Workaround
A. `optimum-cli export openvino`	No	Yes	Export on CPU-only machine; copy IR to target Lunar Lake machine later
B. `scripts/repair_disaggregated_prefill_gpu.py` graph rewrite	No	Yes	Run without `--verify-gpu-compile`
B. `scripts/repair_disaggregated_prefill_gpu.py --verify-gpu-compile`	Intel GPU required	Partially	Skip verify locally; verify on GPU machine later
C. `scripts/repair_decode_present_outputs.py` graph rewrite	No	Yes	Run without `--verify-npu-compile`
C. `scripts/repair_decode_present_outputs.py --verify-npu-compile`	Intel NPU required	Partially	Skip verify locally; verify on NPU machine later

A. `optimum-cli export openvino`

optimum-cli export openvino does not compile the model for GPU or NPU. It downloads the Hugging Face model, runs PyTorch/Transformers/Optimum/NNCF export, and writes OpenVINO IR files.

Therefore, this step can run on a CPU-only machine as long as the software dependencies are available.

Required resources:

Enough system RAM and disk space for the original model, temporary export files, and final IR. The 15.5b-23E source model is roughly 30 GB in BF16; NF4 IR is ~9 GB.
Python packages: torch, transformers, optimum-intel, nncf, openvino, huggingface_hub.
HF access token if the source model requires authentication.

No Intel GPU/NPU is required for:

bash

optimum-cli export openvino \
  --model xpuenabler/gpt-oss-15.5b-23E-SFT \
  --weight-format nf4 \
  --sym \
  --group-size -1 \
  --disable-stateful \
  --trust-remote-code \
  <work_dir>/decode_static_base

Workaround when there is no Lunar Lake machine:

Run the export on any CPU machine with enough memory and disk.
Run the static reshape step on the same machine.
Copy decode_static_base/ and generated sidecar files to the Lunar Lake machine.
Perform GPU/NPU compile verification and inference on the Lunar Lake machine.

B. `scripts/repair_disaggregated_prefill_gpu.py`

This script has two phases.

The graph rewrite phase does not require GPU hardware:

Reads openvino_model.xml.
Decomposes 6-input ScaledDotProductAttention.
Computes SDPA softmax in FP32 when --fp32-sdpa-softmax is used.
Removes zero-length past_key_values.* inputs when --remove-zero-past is used.
Saves a new OpenVINO IR.

This can run without Intel GPU:

bash

python scripts/repair_disaggregated_prefill_gpu.py \
  --src-dir <work_dir>/prefill_static_base \
  --dst-dir <work_dir>/prefill_gpu_finite \
  --fp32-sdpa-softmax \
  --remove-zero-past

The GPU verification phase requires Intel GPU hardware:

--verify-gpu-compile

That option internally calls:

core.compile_model(model, "GPU")

If the machine has no Intel GPU or no working OpenVINO GPU plugin/driver, this verification will fail even if the IR rewrite itself is valid.

Workaround:

Run the script without --verify-gpu-compile on a CPU-only machine.
Copy <work_dir>/prefill_gpu_finite/ to a Lunar Lake machine.
Run the same script again with --verify-gpu-compile, or simply compile the generated prefill_gpu_finite/openvino_model.xml with core.compile_model(..., "GPU").

CPU fallback note:

CPU compile can be used as a weak structural sanity check, but it does not prove GPU plugin compatibility.
The original issue was GPU-specific (ScaledDotProductAttention / zero-length layout handling), so final validation must happen on Intel GPU.

C. `scripts/repair_decode_present_outputs.py`

This script also has two phases.

The graph rewrite phase does not require NPU hardware:

Reads openvino_model.xml.
Rebuilds model outputs so 48 present.* KV outputs are converted to FP32.
Saves a new OpenVINO IR.

This can run without Intel NPU:

bash

python scripts/repair_decode_present_outputs.py \
  --src-dir <work_dir>/decode_static_base \
  --dst-dir <work_dir>/decode_npu_present_f32

The NPU verification phase requires Intel NPU hardware:

--verify-npu-compile

That option internally calls:

core.compile_model(model, "NPU")

If the machine has no Intel NPU or no working OpenVINO NPU plugin/driver, this verification will fail even if the IR rewrite itself is valid.

Workaround:

Run the script without --verify-npu-compile on a CPU-only machine.
Copy <work_dir>/decode_npu_present_f32/ to a Lunar Lake NPU machine.
Run the same script again with --verify-npu-compile, or compile the generated decode_npu_present_f32/openvino_model.xml with core.compile_model(..., "NPU").

CPU fallback note:

CPU compile can confirm that the edited IR is syntactically valid.
CPU compile cannot validate NPU memory planning, NPU supported ops, or NPU plugin-specific behavior.
The final all_logits_finite: true claim still requires running the actual GPU prefill + NPU decode benchmark on a machine with both devices.

Practical Split Build

If GPU/NPU hardware is not available on the build machine, use this split flow:

Machine	Run
CPU build server	`optimum-cli export openvino`
CPU build server	static decode reshape
CPU build server	prefill reshape
CPU build server	`repair_disaggregated_prefill_gpu.py` without `--verify-gpu-compile`
CPU build server	`repair_decode_present_outputs.py` without `--verify-npu-compile`
Lunar Lake validation machine	GPU compile prefill
Lunar Lake validation machine	NPU compile decode
Lunar Lake validation machine	`benchmark_prefill_decode.py --merge-decode-tail-past --past-clip-abs 2`

Minimum final validation command on the Lunar Lake machine:

bash

python scripts/benchmark_prefill_decode.py \
  --prefill-repo ./prefill \
  --decode-repo ./decode \
  --prefill-device GPU \
  --decode-device NPU \
  --context-len 128 \
  --decode-len 16 \
  --merge-decode-tail-past \
  --past-clip-abs 2 \
  --output-dir ./results/ctx128_gpnd_finite

Build Details

1. Original Model Export

The rebuild script uses Optimum-Intel:

bash

optimum-cli export openvino \
  --model xpuenabler/gpt-oss-15.5b-23E-SFT \
  --weight-format nf4 \
  --sym \
  --group-size -1 \
  --disable-stateful \
  --trust-remote-code \
  <work_dir>/decode_static_base

Notes:

--disable-stateful is required so KV cache is exposed as explicit past_key_values.* inputs and present.* outputs.
--awq and --scale-estimation are intentionally not used in this rebuild path because GPT-OSS fused MoE tensors can trigger NNCF broadcast errors under stateless export.

2. Static Decode Contract

The exported decode graph is reshaped to:

Tensor	Shape
`input_ids`	`[1,1]`
`attention_mask`	`[1,129]`
`position_ids`	`[1,1]`
`past_key_values.{0..23}.{key,value}`	`[1,8,128,64]`
`logits`	`[1,1,201088]`
`present.{0..23}.{key,value}`	`[1,8,129,64]`

3. Static Prefill Contract

The ctx128 prefill graph is reshaped to:

Tensor	Shape
`input_ids`	`[1,128]`
`attention_mask`	`[1,128]`
`position_ids`	`[1,128]`
`past_key_values.{0..23}.{key,value}`	`[1,8,0,64]` before repair
`logits`	`[1,128,201088]`
`present.{0..23}.{key,value}`	`[1,8,128,64]`

4. GPU Prefill Repair

scripts/repair_disaggregated_prefill_gpu.py applies:

Decompose 24 six-input ScaledDotProductAttention nodes for Intel GPU compatibility.
Compute SDPA softmax in FP32.
Remove 48 zero-length past_key_values.* inputs.
Optionally verify GPU compile (skipped on CPU-only build hosts).

Equivalent command:

bash

python scripts/repair_disaggregated_prefill_gpu.py \
  --src-dir <work_dir>/prefill_static_base \
  --dst-dir <work_dir>/prefill_gpu_finite \
  --fp32-sdpa-softmax \
  --remove-zero-past

Add --verify-gpu-compile on a Lunar Lake machine.

5. NPU Decode Repair

scripts/repair_decode_present_outputs.py converts 48 present.* KV outputs to FP32 and optionally verifies NPU compile.

Equivalent command:

bash

python scripts/repair_decode_present_outputs.py \
  --src-dir <work_dir>/decode_static_base \
  --dst-dir <work_dir>/decode_npu_present_f32

Add --verify-npu-compile on a Lunar Lake machine.

6. Runtime KV Handoff

The successful finite run requires:

--merge-decode-tail-past --past-clip-abs 2

At each decode step the benchmark script:

Keeps the previous finite past KV.
Appends only the last-token tail KV from decode present.*.
Crops the cache window to 128 tokens.
Replaces NaN/Inf with zero.
Clips KV values to [-2, 2].

Expected Benchmark Result

The exact generated_text baseline for this 15.5b-23E variant has not yet been recorded — the benchmark must be run on a Lunar Lake machine. Use the same command pattern as the reference 6.6b-8E run:

bash

python scripts/benchmark_prefill_decode.py \
  --prefill-repo ./prefill \
  --decode-repo ./decode \
  --prefill-device GPU \
  --decode-device NPU \
  --context-len 128 \
  --decode-len 16 \
  --merge-decode-tail-past \
  --past-clip-abs 2 \
  --output-dir ./results/ctx128_gpnd_finite

The required success signal is:

json

{
  "all_logits_finite": true
}

For reference, the 6.6b-8E build produced the following on a Lunar Lake validation machine:

TTFT: 1.097 s
Output throughput: 8.550 token/s
NPU peak memory: 4.307 GiB
GPU peak memory: 16.552 GiB

The 15.5b-23E variant has roughly 2.3× more parameters, so memory and TTFT/throughput will differ accordingly.

Prompt Reproducibility

The benchmark script builds the ctx128 prompt from this seed:

python

seed = (
    "The weather is clear today and the forecast says the afternoon will stay mild "
    "with light wind. OpenVINO runs optimized inference on Intel hardware. "
)

It repeats the tokenized seed until it reaches 128 tokens, truncates to exactly 128, then uses greedy nanargmax token selection. To reproduce the same text, keep the tokenizer, prompt seed, context length, decode length, and KV handoff options unchanged.

Included Scripts

scripts/rebuild_from_original_ctx128_gpnd.py
scripts/benchmark_prefill_decode.py
scripts/repair_disaggregated_prefill_gpu.py
scripts/repair_decode_present_outputs.py
scripts/diagnose_disaggregated_logits.py

Pass a Hugging Face token through HF_TOKEN or --token only when remote model download is needed.

xpuenabler/gpt-oss-15.5b-23E-nf4-optimum-static-128-GpNd

gpt-oss-15.5b 23E NF4 OpenVINO Static 128 GpNd

Reproduce From the Original Hugging Face Model

Environment

Command

Hardware Requirements and Workarounds

Summary

A. optimum-cli export openvino

B. scripts/repair_disaggregated_prefill_gpu.py

C. scripts/repair_decode_present_outputs.py

Practical Split Build

Build Details

1. Original Model Export

2. Static Decode Contract

3. Static Prefill Contract

4. GPU Prefill Repair

5. NPU Decode Repair

6. Runtime KV Handoff

Expected Benchmark Result

Prompt Reproducibility

Included Scripts

xpuenabler/
gpt-oss-15.5b-23E-nf4-optimum-static-128-GpNd

A. `optimum-cli export openvino`

B. `scripts/repair_disaggregated_prefill_gpu.py`

C. `scripts/repair_decode_present_outputs.py`