Beyond the Embedding Bottleneck: Adaptive Retrieval-Augmented 3D CT Report Generation
_{accepted to ECCV 2026, then desk-rejected for bibliographic inaccuracies}

📄 arXiv · 💻 GitHub · 🤗 Models & Data

Contrastive 3D CT embeddings concentrate 90% of their variance in just 2 of 512 dimensions, and scaling the LLM from 8B to 70B gives no gain — the bottleneck is visual, not generative. AdaRAG-CT compensates by retrieving organ-indexed report sentences and adaptively injecting them during generation, lifting Clinical F1 from 0.420 (CT-Agent) to 0.480.

The Llama backbones are not redistributed here — they are gated on HuggingFace and must be requested and downloaded separately: meta-llama/Llama-3.1-8B-Instruct (8B) and meta-llama/Llama-3.3-70B-Instruct (70B). Only training needs them; evaluation and inference run from the released checkpoints alone. Point model.model_path in the training configs at your local copy.

Everything else — CT embeddings, retrieval indices, and all four checkpoints — is on 🤗 HuggingFace, laid out exactly like this repo, so download straight into the repo root:

This gives you data/ (embeddings, annotations, oracle/retrieval contexts) and results/{base,adaragct}_{8b,70b}/. Four checkpoints: Base 8B/70B (standalone merged models) and AdaRAG-CT 8B/70B (LoRA adapter + projector, loaded on top of the matching base). The released predictions, metrics, and training logs under results/ already ship with this repo.

The pipeline starts from precomputed embeddings — extracting ViSD-Boost organ features and CT-CLIP whole-volume features from raw CT volumes is not part of this release; use the released data/embeddings/ instead.

Score the released predictions directly:

Add --compute-llama-score for the LLaMA score, --bootstrap 1000 for 95% confidence intervals.

Or generate predictions, then score:

Key inference flags: --no-rag (disable retrieval), --text2text (Text2Text retrieval pipeline), --oracle (oracle context), --top-k, --max-retrievals.

On-the-fly retrieval encodes each generated probe sentence with a fine-tuned text encoder (code + indices ship under data/retrieval/; it auto-downloads microsoft/BiomedVLP-CXR-BERT-specialized on first run). For a download-free run, use --oracle (precomputed contexts). To reproduce the exact paper numbers, score the released predictions in results/.

AdaRAG-CT trains a [RAG] trigger token on top of a frozen base model via LoRA, mixing oracle and retrieved contexts. Every required input — base checkpoint, CT embeddings, and the precomputed oracle/retrieval contexts — comes from the HuggingFace bundle, so training runs directly with no extra preprocessing:

Each config points to data/ (embeddings, oracle_context_top3.jsonl, retrieval_context_top3.jsonl) and a base checkpoint under results/.

The base model — organ-level ViSD-Boost embeddings plus a whole-volume CT-CLIP embedding projected into a Llama backbone — is trained in two stages by adaragct.train.train_base. Download the released Base checkpoints if you only want to train AdaRAG-CT on top of them; the steps below are for reproducing a Base checkpoint from scratch.

Stage 1 — projector pretraining (train_projector_only: true). The LLM is frozen and only the five projectors (whole-CT + lung/heart/esophagus/aorta) are trained, to align the visual embeddings with the LLM hidden space. Saves checkpoints/step_N/projector.pt.

Stage 2 — LoRA SFT (freeze_projector: true). Loads a Stage-1 projector through model.pretrain_checkpoint, freezes it, and trains LoRA + embed_tokens/lm_head. Set model.pretrain_checkpoint to the Stage-1 step you want (the released models used step 9000), then:

Merge. Stage 2 saves a PEFT folder; merging it into the backbone produces the standalone model directory that the AdaRAG-CT configs expect as model.model_path (this is the format the released results/base_8b/checkpoint is in):

Stage-1 projector checkpoints are not released, so reproducing a Base checkpoint means running both stages. adaragct/tools/ also holds convert_base_checkpoint.py / convert_rag_checkpoint.py for converting older single-file .pt checkpoints into the PEFT folder format.

CT-RATE / CT-CLIP · ViSD-Boost · LLaVA · Self-RAG