MiniMax-H3-Fun-Controlnet-Union

Model Card

Model Features

• Union control: one checkpoint handles Canny, Depth, HED, MLSD and Pose control videos for video-to-video generation, no per-condition checkpoint switching.
• The control branch attaches to 5 of the 50 transformer blocks (layers 0, 10, 20, 30, 40); every control skip is added to the main branch through a zero-gated projection.
• Guidance-distilled: run with guidance_scale = 1.0, one forward pass per step, no classifier-free guidance needed.
• Inpainting is supported: the control input is widened to control_in_dim = 49 (latent + masked latent + mask channels); use examples/minimax_h3_fun/predict_v2v_control_inpaint.py.
• control_context_scale scales every control skip before it is added to the main branch: 1.0 gives the strongest control (used for all results below), values below 1.0 weaken the guidance of the control video, 0.0 switches the control branch off.
• The generation follows the control video: the frame count snaps down to the largest 17 * n + 5 the video VAE can decode (duration capped at 15 seconds), the canvas keeps the control video's own aspect ratio at the height * width pixel budget (both multiples of 32), at a fixed 24 fps.
• Detailed prompts give better stability; we recommend describing the scene, the subject and the camera in the prompt.

Results

Canny

Depth

HED

MLSD

Pose

Inference

📦 models/
├── 📂 Diffusion_Transformer/
│   ├── 📂 MiniMax-H3/
│   └── 📂 MiniMax-H3-Fun-Controlnet-Union/
│       └── 📦 MiniMax-H3-Fun-Controlnet-Union.safetensors

• config_path must build the control branch exactly as trained (control_blocks_places: [0, 10, 20, 30, 40], control_in_dim: 49, control_apply_audio: false); a mismatched layout makes the checkpoint fail to load.
• The checkpoint is guidance-distilled: keep guidance_scale = 1.0; a value above 1 applies guidance twice and degrades the output.
• The control checkpoint carries only the control branch; the base MiniMax-H3 weights must be present in model_name.
• Memory: the transformer (about 62 GB) plus the Qwen3-VL text encoder (about 62 GB) do not fit one 80 GB GPU fully loaded; use model_group_offload (fastest) or model_cpu_offload_and_qfloat8 on a single 80 GB GPU.

License