This checkpoint is optimized for lower VRAM usage while maintaining high inference quality. It is intended for runtimes supporting NVIDIA ModelOpt quantization, such as vLLM.

Quantization Command

bash

python3 examples/llm_ptq/hf_ptq.py \
    --pyt_ckpt_path /path/to/Qwen3.5-9B \
    --recipe general/ptq/nvfp4_default-kv_fp8_cast \
    --dataset nemotron-post-training-dataset-v2 \
    --batch_size 1 \
    --skip_generate \
    --verbose \
    --export_path /path/to/Qwen3.5-9B-NVFP4-FP8KV

Usage

Ask any AI, even this one :). I tested this checkpoint with vLLM and it worked well. It should also work with other runtimes that support NVIDIA ModelOpt checkpoints, such as Transformers, although I haven't tested those myself.

Credits

This repository only contains a post-training quantized checkpoint.

Credit for the original model, fine-tuning, abliteration, and Safetensors conversion goes to the respective upstream authors:

Base model: https://huggingface.co/Qwen/Qwen3.5-9B
Uncensored fine-tune: https://huggingface.co/CezarJedi/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive
Safetensors conversion: https://huggingface.co/nDimensional/Qwen3.5-9B-Uncensored-Safetensors
Quantization performed using NVIDIA TensorRT Model Optimizer: https://github.com/NVIDIA/TensorRT-Model-Optimizer
AxionML's Qwen3.5-9B-NVFP4 served as a helpful reference for the quantization recipe and calibration setup: https://huggingface.co/AxionML/Qwen3.5-9B-NVFP4

CezarJedi/Qwen3.5-9B-Abliterated-NVFP4-FP8KV

Qwen3.5-9B-Uncensored-NVFP4-FP8KV

Quantization

Quantization Command

Usage

Credits

CezarJedi/
Qwen3.5-9B-Abliterated-NVFP4-FP8KV