Ornith-1.0-35B-uncensored-heretic-nvfp4-fp8dense-gb10
This is a DGX Spark optimized quant for vLLM: a GB10-oriented
compressed-tensors quantization of
llmfan46/Ornith-1.0-35B-uncensored-heretic for vLLM inference.
Quantization
- Dense attention and linear-attention projections: FP8 W8A8.
- Routed MoE experts and shared expert projections: NVFP4 W4A4.
- Preserved in BF16: embeddings,
lm_head, router gates, visual modules, norms, Conv1D, A_log, dt_bias, and non-target tensors.
- Visual encoder: copied from the source checkpoint and preserved in BF16 for multimodal/image support.
- Calibration data:
HuggingFaceH4/ultrachat_200k, train_sft.
- Calibration samples: 512.
- Calibration sequence length: 2048.
- Pipeline: sequential, CPU offload, all MoE experts calibrated.
The quantization groups use mutually exclusive explicit target regexes.
Output
- Format:
mixed-precision.
- Quantization method:
compressed-tensors.
- Config groups:
group_0, group_1.
- Weight layout: sharded safetensors with model.safetensors.index.json.
- Safetensors shards: 5.
- Indexed tensors: 124376.
- Max position embeddings: 262144.
- MTP weights: not present in the inspected source checkpoint.
DGX Spark vLLM
This model is intended for the GB10 patched vLLM path used by
demon-zombie/Qwen3.5-122B-A10B-NVFP4-FP8Dense-GB10.
Example:
1docker run --gpus all -p 8000:8000 --ipc host \
2 -v /opt/vllm-cache:/root/.cache/huggingface \
3 -e CUBLASLT_WORKSPACE_SIZE=33554432 \
4 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False \
5 -e RUNAI_STREAMER_MEMORY_LIMIT=4294967296 \
6 vllm/vllm-openai:v0.22.1 \
7 --model thanet-s/Ornith-1.0-35B-uncensored-heretic-nvfp4-fp8dense-gb10 \
8 --served-model-name Ornith-1.0-35B-uncensored-heretic \
9 --kernel-config '{"moe_backend": "flashinfer_b12x"}' \
10 --load-format runai_streamer \
11 --gpu-memory-utilization 0.50 \
12 --kv-cache-dtype fp8 \
13 --enable-prefix-caching \
14 --enable-chunked-prefill \
15 --max-num-batched-tokens 4176 \
16 --enable-auto-tool-choice \
17 --tool-call-parser qwen3_xml \
18 --reasoning-parser qwen3
On DGX Spark / GB10, the vLLM image may still need the same CUTLASS DSL and
flashinfer SM12x patch files described in the reference model card above.
DGX Spark benchmark
Measured on NVIDIA DGX Spark / GB10 with vllm/vllm-openai:v0.22.1, the GB10 CUTLASS DSL and flashinfer SM12x patches, --gpu-memory-utilization 0.50, --kv-cache-dtype fp8, --kernel-config {"moe_backend": "flashinfer_b12x"}, and --max-num-batched-tokens 4176.
- vLLM log throughput during a warm single request:
Avg generation throughput: 60.8 tokens/s.
- Client-side measurement for the same warm request: 1024 completion tokens in 16.905s, about
60.58 tok/s.
- First measured request after server readiness: 768 completion tokens in 13.676s, about
56.15 tok/s; vLLM log showed Avg generation throughput: 53.5 tokens/s.
- Model load memory reported by vLLM: 21.04 GiB.
- Max model length reported by vLLM: 262144 tokens.
- GPU KV cache size at
--gpu-memory-utilization 0.50: 3,344,321 tokens.
- Maximum concurrency reported by vLLM for 262144-token requests: 12.76x.
These figures are from one local DGX Spark run and may change with prompt shape, sampling settings, vLLM version, patch versions, and CUDA graph/cache warmup state.
Status
Local DGX Spark conversion validation:
- Source:
llmfan46/Ornith-1.0-35B-uncensored-heretic.
- Conversion completed with the same NVFP4-FP8Dense GB10 recipe used for the non-uncensored Ornith 35B build.
verify-output.py passed after vLLM config patching and visual tensor merge.- Visual tensors preserved: 333
model.visual.* tensors copied from the source checkpoint.
- vLLM serving smoke test passed on DGX Spark / GB10 after upload.
