IntrinsicWeather (Diffusers)
This repo bundles inverse rendering, forward weather rendering, and the IMAA gating module into a single Hugging Face–compatible layout. Shared Stable Diffusion 3 components (VAE, text encoders, tokenizers, scheduler) are stored once; task-specific transformers live under transformer/<variant>/.
Model layout
IntrisicWeather-diffusers/
├── dinov2/ # bundled DINOv2 weights (for IMAA / decomposition)
├── imaa/ # Intrinsic Map-Aware Attention weights
├── text_encoder/, text_encoder_2/, text_encoder_3/
├── tokenizer/, tokenizer_2/, tokenizer_3/
├── vae/, scheduler/
├── transformer/
│ ├── inverse-512/ # IntrinsicWeatherSD3Transformer2DModel (in_channels=32)
│ │ └── transformer_intrinsic_weather.py
│ └── forward/ # SD3Transformer2DModel (in_channels=96)
│ └── lora/ # forward-renderer LoRA (loaded by default)
├── pipeline_intrinsic_weather.py # unified: RGB → maps → weather RGB
├── pipeline_intrinsic_weather_inverse.py # inverse only
├── pipeline_intrinsic_weather_forward.py # forward only
├── pipeline_utils.py
├── model_index.json
├── convert_inverse_renderer_512.py
├── convert_forward_renderer.py
└── test_all_pipelines.pyComponent Source Notes Inverse transformer GilgameshYX/InverseRenderer-512 512×512 decomposition Forward transformer + LoRA GilgameshYX/ForwardRenderer LoRA in transformer/forward/lora/ IMAA InverseRenderer-512 imaa.pth Required for map-aware inverse attention SD3 shared weights stabilityai/stable-diffusion-3-medium-diffusersVAE + text encoders only Transformer config stabilityai/stable-diffusion-3.5-mediumArchitecture template for weight loading
Requirements
Python 3.10+
CUDA GPU recommended (~20 GB VRAM for full end-to-end inference at 512×512)
torch, diffusers>=0.38, transformers, safetensors, torchvision, Pillow
pip install torch diffusers transformers safetensors torchvision pillow accelerateQuick start (end-to-end weather edit)
The unified pipeline decomposes an input RGB image into intrinsic maps, then renders a weather-conditioned result. DINOv2 is required for decomposition (bundled under dinov2/, or use facebook/dinov2-base from Hugging Face).
1 from pathlib import Path
2
3 import torch
4 from PIL import Image
5 from transformers import AutoImageProcessor , AutoModel
6
7 from pipeline_intrinsic_weather import IntrinsicWeatherPipeline
8
9 repo_dir = Path ( "." ) . resolve ( ) # path to this folder
10 device = "cuda"
11 dtype = torch . bfloat16
12
13 pipe = IntrinsicWeatherPipeline . from_pretrained (
14 repo_dir ,
15 inverse_transformer_subfolder = "inverse-512" ,
16 forward_transformer_subfolder = "forward" ,
17 device = device ,
18 local_files_only = True ,
19 torch_dtype = dtype ,
20 load_lora = True ,
21 load_imaa = True ,
22 )
23
24 dino_path = repo_dir / "dinov2"
25 dino_processor = AutoImageProcessor . from_pretrained ( dino_path , local_files_only = True )
26 dino_model = AutoModel . from_pretrained ( dino_path , local_files_only = True ) . to ( device )
27 dino_model . eval ( )
28
29 image = Image . open ( "input.png" ) . convert ( "RGB" )
30 result = pipe (
31 image = image ,
32 weather = "snowy" , # rainy | sunny | snowy | foggy | overcast | night
33 dino_model = dino_model ,
34 dino_processor = dino_processor ,
35 image_size = 512 ,
36 render_size = 512 ,
37 num_inverse_steps = 50 ,
38 num_forward_steps = 50 ,
39 guidance_scale = 6.0 ,
40 image_guidance_scale = 1.5 ,
41 generator = torch . Generator ( device = device ) . manual_seed ( 42 ) ,
42 )
43 result . images [ 0 ] . save ( "output_snowy.png" )
Run from inside this directory (or add it to PYTHONPATH) so pipeline_intrinsic_weather.py and imaa/ resolve correctly.
Pipelines
1. IntrinsicWeatherPipeline (unified)
Full pipeline: RGB → intrinsic maps → weather RGB .
1 pipe = IntrinsicWeatherPipeline . from_pretrained (
2 repo_dir ,
3 inverse_transformer_subfolder = "inverse-512" ,
4 forward_transformer_subfolder = "forward" ,
5 device = "cuda" ,
6 torch_dtype = torch . bfloat16 ,
7 )
Useful kwargs:
Argument Default Description inverse_transformer_subfolder"inverse-512"Inverse transformer under transformer/ forward_transformer_subfolder"forward"Forward transformer under transformer/ load_loraTrueLoad LoRA from transformer/forward/lora/ load_imaaTrueLoad IMAA weights from imaa/ deviceNoneMoves all modules to device (IMAA stays float32)
Sub-methods:
pipe.decompose(image, dino_model, dino_processor, ...) → dict of intrinsic mapspipe.render(maps, weather="rainy", ...) → weather-conditioned RGB
2. IntrinsicWeatherInversePipeline
Inverse rendering only (single intrinsic map per call).
1 from pipeline_intrinsic_weather_inverse import IntrinsicWeatherInversePipeline
2
3 pipe = IntrinsicWeatherInversePipeline . from_pretrained (
4 repo_dir ,
5 transformer_subfolder = "inverse-512" ,
6 device = "cuda" ,
7 torch_dtype = torch . bfloat16 ,
8 )
Load the transformer separately if needed:
1 transformer = IntrinsicWeatherInversePipeline . load_transformer (
2 "inverse-512" , repo_dir , device = "cuda"
3 )
4 pipe = IntrinsicWeatherInversePipeline . from_pretrained (
5 repo_dir , transformer = transformer , device = "cuda"
6 )
IMAA and DINO are used by the unified pipeline’s decompose() path; for standalone inverse calls, pass map_aware_mask from IMAA manually (see test_all_pipelines.py).
3. IntrinsicWeatherForwardPipeline
Forward weather rendering from intrinsic maps.
1 from pipeline_intrinsic_weather_forward import IntrinsicWeatherForwardPipeline
2
3 pipe = IntrinsicWeatherForwardPipeline . from_pretrained (
4 repo_dir ,
5 transformer_subfolder = "forward" ,
6 device = "cuda" ,
7 torch_dtype = torch . bfloat16 ,
8 load_lora = True ,
9 )
LoRA weights are read from transformer/forward/lora/ when load_lora=True.
Weather presets
Built-in weather keys (or pass a custom prompt string):
Key Prompt rainyA rainy day. sunnyA sunny day. snowyA snowy day. foggyA foggy day. overcastAn overcast day. nightA night scene.
Intrinsic maps (AoVs)
The inverse renderer produces five appearance-of-variety maps:
albedo, normal, roughness, metallic, irradiance
Loading transformers manually
Transformers are stored per variant under transformer/<subfolder>/. Use pipeline_utils.load_transformer_from_subfolder:
1 from pipeline_utils import load_transformer_from_subfolder , load_transformer_lora
2
3 inverse = load_transformer_from_subfolder ( repo_dir , "inverse-512" , device = "cuda" )
4 forward = load_transformer_from_subfolder ( repo_dir , "forward" , device = "cuda" )
inverse-512 uses a custom IntrinsicWeatherSD3Transformer2DModel (in_channels=32).forward uses the standard SD3Transformer2DModel (in_channels=96).
Dtype and device notes
Default dtype is torch.bfloat16
IMAA stays in float32 (DINO patch tokens are float32).Pass device="cuda" to from_pretrained on all three pipeline classes; the unified pipeline moves every registered module to the target device automatically.
Testing
Smoke-test all pipelines on CUDA:
python test_all_pipelines.pyRuns 2-step inverse, forward (with LoRA), and unified load checks with bfloat16.
Re-converting from original checkpoints
If you have the raw GilgameshYX checkpoints:
1 # Inverse renderer (512) + IMAA
2 python convert_inverse_renderer_512.py
3
4 # Forward renderer + LoRA
5 python convert_forward_renderer.py
See conversion_metadata.json and conversion_metadata_forward.json for source paths used during conversion.
Hugging Face Hub loading
When published to the Hub, load with trust_remote_code=True:
1 from diffusers import DiffusionPipeline
2
3 pipe = DiffusionPipeline . from_pretrained (
4 "BiliSakura/IntrisicWeather-diffusers" ,
5 custom_pipeline = "pipeline_intrinsic_weather.py" ,
6 trust_remote_code = True ,
7 torch_dtype = torch . bfloat16 ,
8 )
For local use, importing IntrinsicWeatherPipeline directly (as in Quick start) is simpler and avoids Hub cache path issues with custom modules.
References
License
Weights and code follow the licenses of the upstream IntrinsicWeather project and the Stable Diffusion 3 components used for shared modules.
