Qwen3.8-27B-heretic-ara — FP8 E4M3 block quantization
This is an FP8 weight quantization of
heretic-org/Qwen3.8-27B-heretic-ara,
prepared and validated by
magiccodingman.
No additional fine-tuning, merging, or abliteration was performed. The model's
behavior comes from the upstream Heretic ARA checkpoint; this repository only
changes its serialized weight precision.
Quantization details
| Property | Value |
|---|
| Weight format | FP8 E4M3 (float8_e4m3fn) |
| Scale granularity | 128 × 128 blocks |
| Scale storage | FP32 weight_scale_inv |
| Activation metadata | Dynamic FP8 |
| Quantized matrix tensors | 407 |
| Preserved tensors | 792, byte-identical to the BF16 source |
| Tensor payload | 30.87 GB (28.75 GiB) |
The layout and exclusion strategy follow the native
Qwen/Qwen3.8-27B-FP8
checkpoint format. FP8 KV cache is
not stored in the checkpoint; it is a
runtime option that must be enabled in the serving engine.
Validation
The serialized FP8 weights were dequantized to BF16 and compared with the
upstream BF16 checkpoint using identical inputs and compute settings.
| Metric | Result |
|---|
| Scored next-token positions | 638 |
| Mean `D_KL(P_BF16 | |
| Median KL | 0.00185845 |
| P95 KL | 0.0104304 |
| Maximum KL | 0.0977299 |
| Top-1 token agreement | 96.5517% |
| BF16 reference NLL | 2.78304195 |
| FP8 NLL | 2.78553748 |
| NLL delta | +0.00249553 |
| Aggregate relative weight RMSE | 0.0264554 |
The validation corpus covered code, mathematics, science, structured output,
tool syntax, multilingual text, SQL, proof writing, and inference engineering.
Safetensors structure, block geometry, scale tensors, tensor closure, preserved
weights, and repository checksums also passed.
See
FP8_VALIDATION.md and the included JSON reports for
the detailed results.
Hardware note: Validation was performed on RTX 3090 GPUs, which cannot
execute native W8A8 FP8. Transformers therefore dequantized the stored FP8
weights to BF16 for the forward comparison. This validates conversion quality,
but native FP8 throughput and FP8 KV-cache behavior must be measured on the
intended FP8-capable hardware and runtime.
Usage
The repository includes the upstream tokenizer, chat template, generation
configuration, vocabulary, and image/video processor configuration.
Example with vLLM (adjust tensor parallelism for your system):
1vllm serve magiccodingman/Qwen3.8-27B-heretic-ara-fp8 \
2 --tensor-parallel-size 4 \
3 --reasoning-parser qwen3 \
4 --kv-cache-dtype fp8
The weight format is detected from quantization_config in config.json.
Runtime support for block-FP8 kernels and FP8 KV cache varies by accelerator,
backend, and software version. If your runtime lacks a compatible native kernel,
it may reject the checkpoint or dequantize weights instead of accelerating them.
Upstream model and behavior
The upstream checkpoint is a decensored/abliterated derivative of
Qwen/Qwen3.8-27B. According to its
model card, it was produced with
p-e-w/heretic, using
timrohrbaugh/heretic v1.2.0+custom
and the
Arbitrary-Rank Ablation (ARA)
method.
The upstream card reports these parameters:
| Parameter | Value |
|---|
start_layer_index | 26 |
end_layer_index | 56 |
preserve_good_behavior_weight | 0.9432 |
steer_bad_behavior_weight | 0.0009 |
overcorrect_relative_weight | 0.5038 |
neighbor_count | 10 |
Consult the
heretic-org/Qwen3.8-27B-heretic-ara model card
and the
Qwen/Qwen3.8-27B model card
for architecture, prompting, intended-use, benchmark, and limitation details.
Provenance and credits
- Original model: Qwen Team —
Qwen/Qwen3.8-27B
- ARA derivative:
heretic-org/Qwen3.8-27B-heretic-ara
- Heretic project:
p-e-w/heretic
- Custom Heretic fork used upstream:
timrohrbaugh/heretic
- FP8 conversion and validation:
magiccodingman
This quantization is an independent derivative and is not an official Qwen,
Heretic, or upstream-author release.
Limitations and responsibility
Quantization can change token probabilities even when aggregate divergence is
small. Validate the model on your own workloads before production use. This
checkpoint also inherits the upstream model's limitations and its deliberately
reduced refusal behavior. Users are responsible for deployment safeguards,
legal compliance, and generated content.
License
Apache License 2.0. See
LICENSE. All upstream notices and
attributions remain applicable.
