Recipe	Quant Size	Default type	Tensor-specific overrides
`Q3_K`	119297.23 MiB (3.52 BPW)	`Q6_K`	`ffn_down_exps=q3_k`, `ffn_gate_exps=q3_k`, `ffn_up_exps=q3_k`
`IQ3_XXS`	106912.23 MiB (3.15 BPW)	`Q6_K`	`ffn_down_exps=iq3_xxs`, `ffn_gate_exps=iq3_xxs`, `ffn_up_exps=iq3_xxs`
`Q2_K`	92465.23 MiB (2.73 BPW)	`Q6_K`	`ffn_down_exps=q2_k`, `ffn_gate_exps=q2_k`, `ffn_up_exps=q2_k`

Usage

This is the script I use to run:

bash

#!/bin/sh -e

model="./IQ3_XXS/DeepSeek-V4-Flash-IQ3_XXS-00001-of-00004.gguf"

ctx=131072
parallel=1

ctx_size=$((ctx * parallel))

llama-server --no-mmap --no-warmup \
  --model $model --ctx-size $ctx_size -np $parallel \
  --temp 1.0 --top-p 1.0

tarruda/DeepSeek-V4-Flash-GGUF

DeepSeek V4 Flash GGUF

Quant Recipes

Usage

tarruda/
DeepSeek-V4-Flash-GGUF