A compressed-tensors AWQ W4A16 quantization of
OpenGVLab/InternVL3_5-4B-HF,
built for serving a 4B vision-language model on an 8 GB consumer GPU with vLLM.
Normalization layers and other protected parameters
Generated from the base-model revision:
6bd4487402110ef9889ba50eb7aefeb302526fed
Checkpoint contents verified: 1,597 tensors, 252 packed INT4 modules with
matching scale / zero-point tensors, no packed weights under vision_tower,
multi_modal_projector, lm_head, or the embeddings, and no NaN/Inf in any
scale. See quantization/recipe.py.
Calibration
AWQ fits per-channel scales against a calibration set, so the calibration
distribution matters. This build used:
Dataset
lmms-lab/flickr30k
Samples
128 (0 skipped)
Image size
448×448, 1 patch per sample
Sequence length
278–288 tokens
Prompts
generic English / Korean scene-description instructions
This is a general-purpose photo set. If your target domain is far from
everyday photography — industrial inspection, thermal imagery, medical,
document OCR, CCTV at unusual angles — re-run AWQ with in-domain calibration
images rather than assuming this checkpoint transfers. The FP8 build needs no
calibration and does not carry this caveat.
Verified environment
Component
Version / value
GPU
NVIDIA GeForce RTX 3070 8 GB (Ampere, SM 8.6)
Host
Windows 11 + WSL2 (Ubuntu 24.04), Docker Desktop
NVIDIA driver
591.86
Serving image
vllm/vllm-openai:v0.26.0
vLLM
0.26.0 (V1 engine)
PyTorch
2.11.0+cu130
Transformers
5.14.1
Quantization backend
compressed-tensors 0.17.0
Attention backend
FlashAttention (auto-selected)
Startup with the command below:
text
1Model loading took 3.84 GiB memory and 5.08 seconds
2GPU KV cache size: 12,288 tokens
3Maximum concurrency for 8,192 tokens per request: 1.50x
4Graph capturing finished in 1 secs, took 0.09 GiB
Total process footprint is about 6.6 GB, leaving roughly 0.9 GB free on an
8 GB card that is also driving a desktop. CUDA graphs capture successfully and
a full FP16 KV cache is used at an 8,192-token context — neither is possible
with the FP8 build on the same card.
Then use the OpenAI-compatible endpoint at http://127.0.0.1:8000/v1.
Each flag is explained below — several of them are not obvious, and two of
them fail in ways that a health check will not catch.
Do not use --dtype half
On this checkpoint family --dtype half produces a server that starts
normally, returns HTTP 200 from /health, reports a healthy container
health check — and answers every request with !!!!!!!!. Dequantization
overflows the FP16 range, the logits become NaN, and argmax selects token 0.
It fails silently, so a liveness probe will not catch it.
The checkpoint declares bfloat16 in config.json and Ampere supports BF16
natively at the same memory cost. Always send a real request after a
configuration change, not just a health check.
Pin the KV cache instead of raising --gpu-memory-utilization
Without --kv-cache-memory-bytes, vLLM expands the KV cache to fill whatever
--gpu-memory-utilization allows. On this model that wastes VRAM for no gain
and makes the cache size drift between restarts with whatever the desktop
happens to be using:
utilization only (0.86)
pinned KV (0.79 + 1728M)
KV cache
16,128 – 19,344 tokens (varies per restart)
12,288 tokens (fixed)
Process VRAM
7.9 GB
6.6 GB
Throughput, 16 concurrent
126.1 tok/s
124.4 tok/s
1.3 GB is recovered for a throughput difference inside measurement noise.
Size the cache from the workload instead: 12,288 tokens holds one full-length
8,192-token request with room to spare, or about 11 typical requests of a
960×544 frame plus a normalized crop (~1,084 tokens each).
--gpu-memory-utilization is a fraction of the free memory the process can
see, which is not what host nvidia-smi reports. Under WSL2 the container
saw 6.93 GiB while the Windows host reported 7.48 GiB free, so the effective
ceiling was 0.866, and 0.87 was rejected outright:
text
1Free memory on device cuda:0 (6.93/8.0 GiB) on startup is less than
2desired GPU memory utilization (0.87, 6.96 GiB)
--max-num-seqs: leaving it at the default costs throughput
The vLLM default is 128. When more concurrent requests arrive than the KV
cache can hold, vLLM preempts and recomputes, and throughput collapses below
what a smaller limit achieves. Measured at ~1,084 prompt tokens per request
with a 16,128-token cache:
Concurrent
--max-num-seqs 128
--max-num-seqs 8
1
0.84 s / 63.3 tok/s
0.84 s / 63.4 tok/s
2
1.19 s / 87.4 tok/s
1.18 s / 88.2 tok/s
4
1.87 s / 105.8 tok/s
1.19 s / 167.2 tok/s
8
3.26 s / 119.7 tok/s
3.22 s / 120.5 tok/s
16
11.82 s / 68.9 tok/s
6.54 s / 126.1 tok/s
Pick a limit the cache can actually sustain. Do not derive it from the
Maximum concurrency for 8,192 tokens per request line in the startup log —
that figure assumes every request fills the full context window, which real
requests rarely do.
--max-num-batched-tokens also sizes the multimodal encoder cache
This is not a VRAM tuning knob. The vLLM default of 2048 rejects large images
outright, on any card:
text
1image item with 2816 embedding tokens, which exceeds the
2pre-allocated encoder cache size 2048
InternVL dynamic tiling allows up to 12 patches plus a thumbnail, so a single
image can reach 3,329 embedding tokens. Use at least 4096.
Tool-call flags are needed even without tool calling
--enable-auto-tool-choice and --tool-call-parser are required for the
server to accept requests that carry a tool_choice field. Open WebUI sends
tool_choice: "auto" by default, so omitting the flags makes it fail with:
text
1"auto" tool choice requires --enable-auto-tool-choice and
2--tool-call-parser to be set
Actual tool calling still does not work: the bundled chat_template.jinja has
no tool rendering, so tools=[...] is silently ignored.
Image token cost is driven by aspect ratio, not size
Tile count is chosen from the aspect ratio, so token cost is not monotonic in
resolution. Measured on this model:
Input size
Image tokens
Tiles
448×448
257
1
896×448
769
3
960×544
769
3
800×450
769
3
2048×2048
2,561
10
200×150
3,329
13
4000×300
3,329
13
A 200×150 crop costs 4.3× more than an 800×450 image, because 4:3 maps
exactly onto a 4×3 tile grid and the crop is upscaled to 1792×1344 for no
added information. Across randomly sized crops of a 960×544 frame, the total
prompt cost swung between 1,040 and 4,112 tokens.
When sending region crops, letterbox them into a fixed 448×448 canvas
(preserving aspect ratio, no upscaling) to make cost constant at ~1,065 tokens
per request. On a small-text reading task this was 4× faster than sending the
raw crop, with no loss of accuracy down to 14 px glyphs.
Choosing between the FP8 and AWQ builds
Measured on the same RTX 3070, same vLLM version, each tuned to its own best
settings on this card:
FP8 Dynamic
AWQ W4A16 G128
Weights on GPU
5.51 GiB
3.84 GiB
KV cache
9,088 tokens (fp8 KV)
12,288 tokens (fp16 KV)
KV cache dtype needed for 8192 ctx
fp8 (compromise)
fp16
CUDA graphs on 8 GB
not possible
yes
--gpu-memory-utilization needed
0.85 (near the 0.866 ceiling)
0.79
Process VRAM
7.8 GB
6.6 GB
Text generation, single request
12–18 tok/s
~2–4× faster (18–63 tok/s, varies with warm-up)
Calibration data required
no
yes (see above)
Weight precision
8-bit
4-bit
AWQ is decisively better on memory and speed, and it is the only one of the
two that fits comfortably on an 8 GB card without compromises.
Accuracy has not been compared between the two builds, or against the BF16
base model. 4-bit weights and out-of-domain calibration are both reasons to
expect the AWQ build to degrade first on hard inputs. Benchmark both on your
own data before choosing.
Limitations
A quantized derivative, not an independently trained model.
No accuracy benchmark has been published for this checkpoint. Functional
validation only: text generation, a 7k-token context, and multi-image
(full frame + crop) requests all produce coherent, correct answers on simple
synthetic probes, both from a local directory and downloaded from this repo.
Calibration is general-purpose photography; see Calibration.
The vision tower and output head remain BF16 and are a meaningful share of
the loaded weights, so the size reduction is smaller than 4/16 would suggest.
The bundled chat_template.jinja has no tool-calling support.
VRAM figures depend on driver, desktop applications, context length,
multimodal limits, and vLLM version.
Attribution and license
A quantized derivative of
OpenGVLab/InternVL3_5-4B-HF.
The original project and this derivative are distributed under the Apache-2.0
license. Review the upstream model card for original training details,
limitations, and citation information.
Citation
bibtex
1@article{wang2025internvl3_5,
2 title={InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency},
3 author={Wang, Weiyun and Gao, Zhangwei and Gu, Lixin and Pu, Hengjun and Cui, Long and Wei, Xingguang and Liu, Zhaoyang and Jing, Linglin and Ye, Shenglong and Shao, Jie and others},
4 journal={arXiv preprint arXiv:2508.18265},
5 year={2025}
6}
