Ported to vLLM for massive throughput on RTX Pro 6000 — by Fraser Price
This is probs the best uncensored model you can run on 2 x RTX Pro 6000 at high throughput (source: ✨ my opinion ✨).
Served on the voipmonitor vLLM fork it runs ~13× prefill and ~2.8× single-stream decode faster than the widely available abliterated GGUF versions.
Quick start (one command)
Requires Docker with the NVIDIA Container Toolkit, and ≥2 GPUs (see Hardware).
bash run.sh
run.sh defaults to this repo, so no arguments are needed. Everything is configured via env vars (see the table below) — e.g. GPUS=0,1,2,3 TP=4 bash run.sh to run on 4 GPUs.
run.sh (included) will:
Download the weights (~156 GB) into your Hugging Face cache (\~/.cache/huggingface, or $HF_HOME/$HF_HUB_CACHE) if not already present.
Pull the inference image if not already present.
Serve the model on http://localhost:8000/v1 (OpenAI API spec).
Optional — MTP speculative decoding (~1.8× single-stream decode speed). The checkpoint ships its MTP layer; enable it with MTP=2 bash run.sh (see the table below).
Override anything via env vars (all optional):
Variable
Default
Meaning
GPUS
0,1
GPU indices to use (comma-separated)
TP
2
Tensor-parallel size — set to the number of GPUs in GPUS
MTP
(off)
MTP speculative-decoding draft tokens (e.g. 2); empty disables it
PORT
8000
API port
MAX_MODEL_LEN
262144
Context length
GPU_MEM_UTIL
0.92
Fraction of VRAM to use
HF_REPO
fraserprice/DeepSeek-V4-Flash-Abliterated
Hugging Face repo to download
MODEL_DIR
(HF cache)
Serve from a specific local dir instead of downloading
SERVED_NAME
DeepSeek-V4-Flash-Abliterated
Model name exposed by the API
IMAGE
(pinned voipmonitor/vllm build)
Inference container image
Example on 4 GPUs:
GPUS=0,1,2,3 TP=4 bash run.sh
Hardware/Image
Tuned for NVIDIA RTX Pro 6000 Blackwell (96 GB, sm_120). It will run on other Blackwell-class GPUs, but the numbers below and the default flags assume RTX Pro 6000.
It needs roughly 170 GB of total VRAM for weights + KV cache at long context, split across GPUs by tensor parallelism.
Setup
Works?
2 × RTX Pro 6000 Blackwell (96 GB) — TP=2
✅ reference config
4 × RTX Pro 6000 Blackwell — TP=4
✅ (more KV headroom / throughput)
4 × 80 GB (e.g. H100/H200) — TP=4
✅ (untested here, I believe can just use base vLLM)
1 × DXG Spark
❓ idk why you didn't get an RTX Pro lmao (but lmk if you manage to run this and I can add to docs)
1 × anything
❌ (blessed are the poor in VRAM 🙏)
The serving image targets Blackwell (sm_120) and CUDA 13.2. The DeepSeek-V4 architecture (sparse attention / lightning indexer, MLA, MTP, etc.) requires this image to run on RTX Pro.
See local-inference-lab/rtx6kpro for image, bench, and serving details.
Performance
Measured under load on RTX Pro 6000 Blackwell (connected via PCIe 5.0), 128 generated tokens/request, TP = number of GPUs. MTP is speculative decoding (— = off, 2 = 2 draft tokens). PP / TG are prefill / per-request decode throughput (tok/s); Total is aggregate end-to-end throughput across all concurrent requests (tok/s). Rows are paired off/on so the MTP decode gain is directly comparable.
GPUs
MTP
Prompt
Conc
TTFT
PP
TG
Total
2
—
1,000
1
132 ms
7,613
108.6
870
2
2
1,000
1
143 ms
7,005
176.2
1,310
2
—
1,000
3
275 ms
4,190
76.7
1,746
2
2
1,000
3
332 ms
3,746
125.7
2,435
2
—
10,000
1
1,036 ms
9,658
108.0
4,581
2
2
10,000
1
1,069 ms
9,357
181.6
5,729
2
—
10,000
3
2,138 ms
5,302
57.1
6,600
2
2
10,000
3
2,205 ms
5,146
85.3
7,496
2
—
100,000
1
11,575 ms
8,640
107.6
7,850
2
2
100,000
1
11,933 ms
8,380
196.2
7,959
2
—
100,000
3
23,579 ms
5,153
33.0
8,161
2
2
100,000
3
24,349 ms
4,991
59.6
8,065
4
—
1,000
1
119 ms
8,437
117.9
946
4
2
1,000
1
128 ms
7,858
216.5
1,584
4
—
1,000
3
255 ms
4,539
98.0
2,175
4
2
1,000
3
507 ms
3,276
129.7
1,996
4
—
10,000
1
842 ms
11,886
118.4
5,293
4
2
10,000
1
873 ms
11,459
223.9
7,034
4
—
10,000
3
1,741 ms
6,521
69.0
8,041
4
2
10,000
3
1,813 ms
6,257
108.7
9,216
4
—
100,000
1
9,530 ms
10,494
118.5
9,445
4
2
100,000
1
9,856 ms
10,146
227.4
9,614
4
—
100,000
3
19,226 ms
6,299
40.0
10,028
4
2
100,000
3
20,037 ms
6,049
73.1
9,824
MTP=2 lifts single-stream decode by ~1.6–1.9× (e.g. 108 → 196 tok/s at 100k on TP=2) for a negligible TTFT cost.
How it was made
DeepSeek-V4-Flash's published abliterations existed only as a GGUF for the bespoke ds4/llama.cpp engines, which leave Blackwell hardware massively underutilised.
Rather than re-run abliteration from scratch, the refusal subspace was simply recovered and re-applied from an existing abliteration:
Recover the refusal directions. The rank-3 refusal subspace in the 4096-dim residual stream was extracted from the delta between huihui-ai's abliterated GGUF and the clean base.
Re-apply to the official weights. The same rank-3 projection was applied to every residual-writing matrix of the official checkpoint — attention output (attn.wo_b) and shared-expert down-projection (shared_experts.w2) across all 43 layers and the MTP layer (88 tensors total). Each tensor is dequantised → projected → requantised, with a fixed-point solver that compensates for FP8 requantisation so the intended projection survives quantisation.
Leave everything else untouched. The FP4-quantised routed experts and all gate/up projections are byte-identical to the official release; only the 88 FP8 residual-writing matrices change.
Validation: on every edited tensor the achieved projection matched target within ~0.002, and all 33,880 non-target tensors were verified byte-identical to the base.
The served model drops refusals while keeping correct tool-calling and coherent step-by-step reasoning. No formal evals yet — tested manually; it should match the base huihui-ai abliteration at far higher speed.
This checkpoint contains no huihui-ai weights — only the official DeepSeek-V4-Flash weights with a refusal-direction projection applied.
Disclaimer
This is an uncensored model: the refusal subspace has been removed, so it will not decline requests the way the base model does. It is released as-is, for research/lawful use only.
Not a safety-aligned model. Because guardrails have been removed, outputs may be inaccurate, offensive, or harmful. Apply your own filtering, review, and safeguards before relying on or deploying it.
No warranty. The weights and scripts are provided "as is", without warranty of any kind, express or implied. You use them entirely at your own risk.
You are responsible for what you generate. The author does not endorse, encourage, or accept any responsibility or liability for how this model is used, including any unlawful, harmful, or unethical use. Outputs do not reflect the views of the author.
License
MIT, inheriting from the base model and the abliteration source.
