TensorRT-LLM checkpoints (buildable on any NVIDIA GPU) and a prebuilt INT4 engine (RTX 40-series) for Josiefied-Qwen3-4B-abliterated-v1 — the abliterated (uncensored) Qwen3-4B.

⚠️ This is an abliterated / uncensored model.

Why this exists

TensorRT-LLM engine builds are RAM/VRAM-hungry, and almost nobody shares the TRT-LLM checkpoint format — so people with limited hardware can neither build one nor find a prebuilt. This repo fills that gap:

Checkpoints (rank0.safetensors, TRT-LLM format) → build your own engine for your GPU/SM in minutes, no OOM, no weight-conversion hell.
Prebuilt engine (INT4, RTX 40 / SM89) → run immediately if you're on a 40-series card.

Folder	What	Quant	Runs on
`int4/`	checkpoint	INT4 weight-only	build for any SM
`int4-engine/`	prebuilt engine (16k ctx)	INT4	RTX 40 / SM89 only
`fp8/`	checkpoint	FP8	build for any SM (Ada+)
`gptq/`	checkpoint	INT4 GPTQ	build for any SM
`w4a8awq/`	checkpoint	W4A8 AWQ	build for any SM

Performance — int4 W4A16 is the speed king

On an RTX 4060 (Ada / SM89), the INT4 W4A16 quant is the fastest:

~88 tok/s decode on stock clocks
~113 tok/s with a GPU memory OC + system RAM EXPO enabled (decode is memory-bandwidth-bound, so a memory OC hits it directly)
TTFT ~2s

At ~113 tok/s a 4B model streams a full terminal of text near-instantly — it was originally tested driving a GPU-accelerated terminal and filled the screen almost instantly. For raw speed, use int4. (The prebuilt int4 engine lives in the companion software repo, xThr45hx/TensorRT-LLM-Windows-RTX40.)

Build an engine from a checkpoint

Use trtllm-build from TensorRT-LLM against the rank0.safetensors + config.json in a quant folder, targeting your GPU's SM. Example (INT4):

trtllm-build --checkpoint_dir int4 --output_dir my-engine --gemm_plugin auto

For the native Windows build tooling (no WSL), see xThr45hx/TensorRT-LLM-Windows-RTX40.

Credits

Base: Josiefied-Qwen3-4B-abliterated-v1 (abliterated Qwen3-4B, Apache-2.0)
Quantized to TensorRT-LLM format + engine built by xThr45hx (AI-assisted)

License

Apache-2.0 (matching Qwen3 + TensorRT-LLM upstream).

xThr45hx/Josiefied-Qwen3-4B-TRT-LLM

Josiefied-Qwen3-4B — TensorRT-LLM Checkpoints + Engine

Why this exists

Contents

Performance — int4 W4A16 is the speed king

Build an engine from a checkpoint

Credits

License

xThr45hx/
Josiefied-Qwen3-4B-TRT-LLM