SAR ControlNet
Overview
ControlNet for spatially controlled synthesis of Sentinel-1 SAR images.
Published alongside the upcoming paper "Diffusion-Based SAR Training Data Synthesis Controlled by Spatial Annotations" (Hochstuhl et al., 2026; accepted for GCPR conference 2026).
This model must be used together with the corresponding Stable Diffusion backbone:
sylviaHoch/SAR-StableDiffusionSpatial control is provided via three-channel one-hot masks (water / ship / land), enabling precise placement of ships in the generated SAR images.
Database
The ControlNet was trained on the training set of
OpenSARShip Ship Detection Dataset (OSSDD) [dataset], a Sentinel-1 dataset with 15,161 patches (700 × 700 px) and 64,640 ships, providing rotated bounding boxes (RBBs) and land–water masks. Text captions used during ControlNet training depend on land coverage: open-water patches are captioned
"An aerial view of ships on open water.", patches with land (> 5 % of pixels)
"An aerial view of a coastal area with ships on the water." During training, 512 × 512 crops are extracted, amplitude values converted to dB and linearly mapped to
[-1, 1].
Architecture
| Component | Details |
|---|
| Type | ControlNet (based on SD 1.5 UNet) |
| Control input — ship + land | Three-channel one-hot mask (3, H, W) — ch0: background, ch1: ship, ch2: land |
| Required backbone | sylviaHoch/SAR-StableDiffusion |
Repository Structure
1SAR-ControlNet/
2├── config.json
3└── diffusion_pytorch_model.safetensors
Usage
This model can be used directly with the 🤗
diffusers ControlNet pipeline. It requires the corresponding Stable Diffusion backbone (
sylviaHoch/SAR-StableDiffusion).
Requirements:
Control input: The control image must be a three-channel, one-hot encoded mask of size 512 × 512, i.e. a tensor of shape (3, 512, 512) with values in {0, 1}. Each pixel is assigned to exactly one class across the three channels:
- Channel 0: water
- Channel 1: ship (represented as rotated bounding box/ minimal enclosing rectangle)
- Channel 2: land
Output: image is a torch.Tensor of shape (N, C, H, W) with values in the range [0, 1] and dtype float32, where N is the batch size (here 1), C the number of channels (here 1, single-channel SAR amplitude), and H and W the native model resolution.
1import torch
2from diffusers import ControlNetModel, StableDiffusionControlNetPipeline, UniPCMultistepScheduler
3from peft import PeftModel
4
5SD_MODEL_ID = "sylviaHoch/SAR-StableDiffusion"
6CN_MODEL_ID = "sylviaHoch/SAR-ControlNet"
7
8# control_image: torch.Tensor of shape (3, 512, 512), one-hot encoded
9# channel 0: water, channel 1: ship, channel 2: land
10
11controlnet = ControlNetModel.from_pretrained(CN_MODEL_ID, torch_dtype=torch.float16)
12
13pipeline = StableDiffusionControlNetPipeline.from_pretrained(
14 SD_MODEL_ID,
15 controlnet=controlnet,
16 torch_dtype=torch.float16
17).to("cuda")
18
19# Load LoRA text-encoder adapter
20pipeline.text_encoder = PeftModel.from_pretrained(
21 pipeline.text_encoder,
22 SD_MODEL_ID,
23 subfolder="adapter_text_encoder"
24)
25
26pipeline.scheduler = UniPCMultistepScheduler.from_config(pipeline.scheduler.config)
27
28# Generate
29image = pipeline(
30 "An aerial view of ships on open water.",
31 control_image,
32 num_inference_steps=50,
33 guidance_scale=7.5,
34 controlnet_conditioning_scale=1.0,
35 output_type="pt"
36).images
Intended Use
- Spatially controlled synthesis of Sentinel-1 SAR training data.
- Ship placement guided by spatial annotations.
- Training data generation or data augmentation for ship detection models.
Limitations
- Tuned specifically to Sentinel-1 sensor characteristics (in particular Sentinel-1 GRD products in VH polarization).
- Training scenes are limited to harbour, coastal, and open water scenes.
- Generated images are synthetic and may not fully capture all real SAR image properties.
License
This model is released under
CC BY-NC-SA 4.0.
Commercial use is not permitted. Derivatives must be shared under the same license.
See
LICENSE for details.
