TherA: Thermal-Aware Visual-Language Prompting for
1 Seoul National University
2 Kyungpook National University
3 KENTECH* Equal Contribution
† Corresponding Author
CVPR 2026
TherA method overview
News
2026-04-03 : TherA github repo opening2026-05-22 : TherA inference code and R2T2 dataset release.
Overview
TherA is a controllable RGB-to-thermal infrared translation framework. Given an RGB image, TherA synthesizes a long-wave thermal infrared image using a latent-diffusion translator conditioned on thermal-aware visual-language features.
TherA is designed for:
RGB → TIR translation for thermal perception research.Thermal-aware VLM conditioning using LLaVA hidden-state features.Scene- and object-level controllability across weather, time of day, and object state.Reference-cache inference , allowing deployment without loading LLaVA at runtime.
Key Idea
TherA does not condition directly on raw text during diffusion inference. Instead, it uses a 4096-dimensional LLaVA hidden state , either:
loaded from a precomputed .pt reference cache, or
extracted on the fly using LLaVA.
For resource limited environments, we recommend reference-cache mode . This mode uses precomputed LLaVA features such as SUNNY.pt, CLOUDY.pt, RAINY.pt, or NIGHT.pt, and therefore does not require loading LLaVA weights at runtime. An alternative would be to compute pre-computed LLaVA feature first followed by inferencing with reference-cache mode (upcoming feature).
Repository Layout
1 TherA/
2 ├── infer_custom.py # Batch RGB → TIR inference on a folder
3 ├── infer_example_guided.py # Single-image / example-guided inference
4 ├── infer_palette.sh # Run multiple weather/style palettes
5 ├── lavi_ip2p/ # UNet 8-channel + adapter wrapper
6 ├── LaVi-Bridge/modules/ # TextAdapter architecture
7 ├── llava/ # LLaVA code, only needed for on-the-fly mode
8 ├── thera_paths.py # Default local weight paths
9 ├── thera_llava.py # Lazy LLaVA loader
10 └── weights/ # Download weights here; not tracked by git
11 ├── model.pt # TherA Model
12 ├── merged_models/ # Initialization model
13 │ ├── unet/
14 │ └── adapter/
15 ├── stable-diffusion/
16 │ ├── vae/
17 │ └── scheduler/
18 ├── reference_caches/
19 │ ├── SUNNY.pt
20 │ ├── CLOUDY.pt
21 │ ├── RAINY.pt
22 │ └── NIGHT.pt
23 ├── reference_caches/
24 │ │ ├── SUNNY.pt
25 │ │ ├── CLOUDY.pt
26 │ │ ├── RAINY.pt
27 │ │ └── NIGHT.pt
28 └── TherA-VLM/ # Optional; only for on-the-fly mode
29 ├── adaptor_config.json/
30 └── adapter_model.safetensors
31 └── config.json
32 └── non_lora_trainables.bin
33 └── trainer_state.json
Installation
Option 1: Local Python Environment
1 git clone https://github.com/donkeymouse/TherA.git
2 cd TherA
3
4 python -m venv .venv
5 source .venv/bin/activate
6
7 pip install --upgrade pip
8 pip install -r requirements.txt
Recommended environment
Python 3.10+
CUDA-capable GPU
16 GB+ VRAM recommended for comfortable inference
Option 2: Docker
A prebuilt Docker image is available at:
docker pull donkeymouse/thera:latestExample interactive run:
1 docker run --gpus all --rm -it \
2 -v " $( pwd ) " :/workspace/TherA \
3 -w /workspace/TherA \
4 donkeymouse/thera:latest \
5 bash
Then run inference commands from inside the container.
Download Weights
TherA weights are hosted on Hugging Face:
1 pip install -U huggingface_hub
2
3 huggingface-cli download donkeymouse/TherA \
4 --local-dir weights
After downloading, your weights/ directory should contain:
Path Description Required? weights/model.ptTherA trained UNet and adapter checkpoint Yes weights/merged_models/unet/UNet architecture/config files Yes weights/merged_models/adapter/TextAdapter architecture/config files Yes weights/stable-diffusion/vae/Stable Diffusion VAE Yes weights/stable-diffusion/scheduler/DDIM scheduler config Yes weights/reference_caches/*.ptPrecomputed LLaVA hidden states for inference palettes Recommended weights/TherA-VLM/LLaVA weights for on-the-fly feature extraction Optional
Optional: Download LLaVA Weights
LLaVA is only required for on-the-fly feature extraction or two-image guided mode . It is not required for reference-cache inference.
1 huggingface-cli download llava-hf/llava-1.5-7b-hf \
2 --local-dir weights/llava-1.5-7b-hf
Quick Start
Full RGB-TIR translation using TherA-VLM
Use this mode if you want to extract hidden states from TherA directly at runtime from an RGB image and prompt.
1 python infer_custom.py \
2 --rgb-dir examples/rgb \
3 --output-dir preds \
4 --llava-base-path weights/llava-1.5-7b-hf \
5 --llava-lora-path weights/TherA-VLM \
6 --llava-prompt "How would this RGB scene appear in long-wave thermal infrared spectrum."
This mode is more expensive because it loads LLaVA during inference.
Reference-Guided Image Translation Mode
This mode extracts LLaVA features from a reference RGB image and applies them to a target RGB image.
1 python infer_example_guided.py \
2 --mode two-image \
3 --reference-image examples/ref/rgb.jpg \
4 --input-image examples/rgb/scene.jpg \
5 --output preds/scene_tir.png \
6 --llava-base-path weights/llava-1.5-7b-hf \
7 --llava-lora-path weights/TherA-VLM
Recursive Folder Inference
1 python infer_custom.py \
2 --rgb-dir /path/to/dataset/RGB \
3 --output-dir preds \
4 --reference-cache weights/reference_caches/SUNNY.pt \
5 --recursive
When --recursive is used, the output folder preserves the input directory structure.
Reference-cache Mode
Reference-cache mode is the recommended if you are lacking GPU memory. It does not load LLaVA at runtime.
1 python infer_custom.py \
2 --rgb-dir examples/rgb \
3 --output-dir preds/sunny \
4 --reference-cache weights/reference_caches/SUNNY.pt
The script reads all images in examples/rgb and writes translated TIR images to preds/sunny.
A lighter version of the text-guided image translation module.
Example palette caches:
1 weights/reference_caches/SUNNY.pt
2 weights/reference_caches/CLOUDY.pt
3 weights/reference_caches/RAINY.pt
4 weights/reference_caches/NIGHT.pt
You can use different pallete cache to achieve different translation effects.
Inference Modes
Mode Main flag / script LLaVA weights needed? Recommended use Reference cache --reference-cache path.ptNo Default deployment and fast inference Per-image cache directory --cache-dir dir/No Precomputed feature per image Full RGB-TIR translation --llava-base-path ...Yes Runtime prompt/image conditioning Reference image-guided translation infer_example_guided.py --mode two-imageYes Apply reference-image conditioning
Reference Cache Format
Reference caches are precomputed LLaVA hidden states saved as .pt files.
Supported tensor shapes:
A single reference cache can be applied to all input images as a global thermal/weather/style condition.
Important Arguments
Argument Default Description --checkpointweights/checkpointDirectory containing model.pt --merged-model-pathweights/merged_modelsDirectory containing UNet and adapter configs --pretrained-sdweights/stable-diffusionDirectory containing VAE and scheduler --rgb-dirRequired Folder of RGB images for batch inference --output-dircustom_predictionsOutput folder for predictions --reference-cacheNoneSingle .pt cache used for all images --cache-dirNoneFolder of per-image .pt caches matched by filename stem --llava-base-pathNoneBase LLaVA model path for on-the-fly mode --llava-lora-pathNoneOptional LLaVA LoRA path --llava-promptthermal prompt Prompt used for default inference/text-guided translation --num-steps100DDIM sampling steps --cfg-text3.5Text/VLM guidance strength --cfg-image1.5Image guidance strength --target-sizeAuto Resize image to this square size; otherwise dimensions are rounded to multiples of 32 --recursiveOff Recursively process subdirectories --devicecudaDevice for inference
Architecture
1 RGB image
2 │
3 ▼
4 VAE encoder ──► RGB latents ───────────────────┐
5 │
6 ├──► 8-channel diffusion UNet ──► VAE decoder ──► TIR image
7 │
8 LLaVA hidden state, 4096-d ──► TextAdapter ─────┘
9 768-d cross-attention tokens
TherA uses dual classifier-free guidance at inference by combining:
full conditioning,
image-only conditioning,
text/VLM-only conditioning.
R2T2 Dataset
TherA is trained with R2T2 , a large-scale RGB–TIR–Text dataset.
R2T2 includes:
112,970 aligned triplets : RGB image, TIR image, and canonical thermal schema.Scene diversity across driving, CCTV, aerial, and ego-view settings.
Temporal diversity across day/night and diurnal transitions.
Environmental diversity across weather, season, and illumination.
Material- and object-level annotations with structured canonicalization.
Data compiled from multiple aligned RGB–TIR datasets with additional pseudo-aligned pairs.
Dataset page:
https://huggingface.co/datasets/donkeymouse/TherA-R2T2Example structure:
1 R2T2/
2 ├── ${DATASET_NAME}/
3 │ └── ${SEQUENCE_NAME}/
4 │ ├── RGB/
5 │ │ ├── 1.jpg
6 │ │ └── ...
7 │ └── TIR/
8 │ ├── 1.jpg
9 │ └── ...
10 ├── ViVID/
11 │ ├── img_campus_day1/
12 │ │ ├── RGB/
13 │ │ │ ├── 000001.png
14 │ │ │ └── ...
15 │ │ └── TIR/
16 │ │ ├── 000001.png
17 │ │ └── ...
18 │ └── ...
19 └── ...
Troubleshooting
Checkpoint not found: weights/model.ptDownload the TherA weights and make sure model.pt is located at:
OSError: ... stable-diffusion/vaeMake sure the Stable Diffusion VAE and scheduler folders are present:
1 weights/stable-diffusion/vae/
2 weights/stable-diffusion/scheduler/
Outputs look identical across palettes
Try increasing text/VLM guidance:
Also verify that your reference cache files are distinct:
1 SUNNY.pt
2 CLOUDY.pt
3 RAINY.pt
4 NIGHT.pt
CUDA out of memory
Try reducing the image size:
You can also run one image at a time with:
python infer_example_guided.pyLLaVA import or loading errors
Use reference-cache mode if you do not need runtime LLaVA extraction:
--reference-cache weights/reference_caches/SUNNY.ptFor on-the-fly mode, make sure the LLaVA base model and TherA weights are correctly loaded`.
TODOs
Citation
If you find TherA useful for your research, please cite:
1 @inproceedings{lee2026thera,
2 title = {TherA: Thermal-Aware Visual-Language Prompting for Controllable RGB-to-Thermal Infrared Translation},
3 author = {Lee, Dong-Guw and Rhee, Tai Hyoung and Jang, Hyunsoo and Shin, Young-Sik and Shin, Ukcheol and Kim, Ayoung},
4 booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
5 year = {2026}
6 }
You may also cite the arXiv version:
1 @article{lee2026thera_arxiv,
2 title = {TherA: Thermal-Aware Visual-Language Prompting for Controllable RGB-to-Thermal Infrared Translation},
3 author = {Lee, Dong-Guw and Rhee, Tai Hyoung and Jang, Hyunsoo and Shin, Young-Sik and Shin, Ukcheol and Kim, Ayoung},
4 journal = {arXiv preprint arXiv:2602.19430},
5 year = {2026}
6 }
Acknowledgements
TherA builds on open-source components from the vision-language and diffusion communities, including LLaVA, Stable Diffusion, Diffusers, and LaVi-Bridge-style adapter architectures.
License
See LICENSE for details.
Third-party models, datasets, and libraries retain their own licenses. Please review the licenses for LLaVA, Stable Diffusion, Hugging Face model files, and any external datasets before use.
Contact
If you have any questions, contact here please
