vLLM Docker Image — Qwen3.5-122B-A10B-NVFP4 on Jetson AGX Thor
First confirmed deployment of Qwen3.5-122B-A10B at NVFP4 precision on a single NVIDIA Jetson AGX Thor (128GB unified memory).
This repository hosts the compressed Docker image tarballs needed to run a vLLM inference server for Qwen3.5-122B-A10B on Jetson AGX Thor. For full documentation, reproduction scripts, patches, and the Dockerfile, see the companion GitHub repository:
This HuggingFace repo contains only the Docker image as compressed tar archives. The image has all required patches pre-applied and is ready to load on a Jetson AGX Thor system.
File
Description
vllm-thor-qwen35-latest.tar.gz.*
Split compressed Docker image tarballs
sha256sums.txt
Checksums for verifying image integrity
Model weights are not included. You must supply Qwen3.5-122B-A10B-NVFP4 weights separately. See the GitHub repo for the resharding script (01_reshard_nvfp4.sh).
With --enforce-eager, every forward pass goes through Python dispatch with no CUDA graph optimization. For a 94-layer MoE model this causes ~120s TTFT on prompts of ~900 tokens. Remove the flag and allow CUDA graph warmup at startup. The first startup after removing the flag will take 10–20 minutes longer while graphs are captured and cached to ~/thor-vllm-cache.
Required Environment Variables
Variable
Value
Purpose
VLLM_USE_FLASHINFER_MOE_FP4
0
MoE FP4 FlashInfer kernel broken on Thor
LD_PRELOAD
/usr/lib/aarch64-linux-gnu/nvidia/libcuda.so.1
Required for CUDA library resolution on Jetson
HF_HUB_DISABLE_XET
1
Disables experimental HuggingFace XET transfer protocol
GPU Memory Utilization
Values above 0.72 cause OOM during KV cache profiling at model load. Do not increase this value without testing.
--max-num-seqs 2
Higher values increase CUDA graph capture time and VRAM pressure during warmup. Keep at 2 unless you have tested higher values.
CUDA Graph Cache
CUDA graphs are saved to the volume mounted at /root/.cache/vllm. Always mount a persistent host directory here. Without this, graphs are recaptured on every container start (adds 10–20 min per boot).
Benign Startup Warning
You will see this at container start — it is safe to ignore:
ERROR: ld.so: object '/usr/lib/aarch64-linux-gnu/nvidia/libcuda.so.1' from
LD_PRELOAD cannot be preloaded (file too short): ignored.
The CUDA stack loads correctly via other paths.
Import Errors Outside a GPU Container
Running any vLLM import outside of a GPU-enabled container (e.g. for patching or verification) will fail with:
This is expected. Verify patches using grep, not Python imports.
What's Pre-Applied in This Image
The image is built from a pinned vLLM source commit with two required fixes applied. Neither fix is present in the upstream vLLM version baked into the base Jetson container at the time of this build.
Patch 1 — RMSNormGated activation parameter
Adds a missing activation parameter to RMSNormGated.__init__ in vllm/model_executor/layers/layernorm.py. Without this, model load fails with:
AttributeError: 'RMSNormGated' object has no attribute 'activation'
Root cause: RMSNormGated.forward() references self.activation at line 595, but the __init__ method never accepted or stored this parameter. The upstream vLLM code assumed it was already set but the version baked into the Jetson container was missing it.
The fix — two additions to __init__:
python
1# Added to signature (after norm_before_gate):2activation:str="silu",34# Added to body (after self.norm_before_gate = norm_before_gate):5self.activation = activation
Not a code patch — applied via environment variable (VLLM_USE_FLASHINFER_MOE_FP4=0). The FlashInfer MoE FP4 kernel is broken on Jetson Thor at this vLLM version. Setting this variable falls back to the standard MoE kernel. FlashInfer attention (non-MoE) works correctly and is kept enabled via --attention-backend FLASHINFER.
