Anima ControlNet-LLLite Sample Weights

About earlier Preview3-based weights. Sample weights trained against the older Anima Preview3 base model — including lineart / depth / pose / fake scribble, as well as a Preview3-era any-test like and inpainting v1 — are still hosted in this repository. They are documented in the legacy model card PREVIEW3.md. The four conditioning types not retrained for Anima-Base v1.0 (lineart / depth / pose / scribble) remain available there. The Preview3 weights also work on Anima-Base v1.0 with somewhat reduced quality.

Released Weights / 公開する重み

The v2 suffix indicates the Anima-Base v1.0 generation, distinguishing these weights from the Preview3-era v1 series.

Sample / サンプル

Type	Cond image	Mask image	Generated image
inpainting
any-test like		---

Common Setup / 共通設定

Base models

• Anima DiT: Anima-Base v1.0 (anima-base-v1.0.safetensors)
• VAE: Qwen-Image VAE
• Text encoder: Qwen3-0.6B (base)

Dataset (common to all v2 weights)

• Target images: 4,000 images generated by Anima from random prompts (twice the Preview3 set).
• Image composition: ~3/4 contain people (varied gender, single-person to multi-person scenes); the remaining ~1/4 are animals, landscapes, or other no-person content.
• Resolution distribution (Anima-Base v1.0 supports 512² to 1536², expanded from Preview3's 512²–1024²):
  • 50% (2,000 images) — 1024²-area Aspect Ratio Bucketing (same as Preview3).
  • 12.5% each (500 images × 4) — 640² / 768² / 1280² / 1536²-area Aspect Ratio Bucketing.
• Bucket settings: enable_bucket = true, bucket_no_upscale = true, bucket_reso_steps = 16, min_bucket_reso = 64, max_bucket_reso = 3072. The exact per-bucket resolutions are determined automatically from the source images' actual sizes.
• Conditioning images: automatically generated from each target image. Generation method differs per model (see below).

Common training hyperparameters

• Optimizer: adamw8bit
• Mixed / save precision: bf16
• Batch size: 12 (--gradient_checkpointing enabled — to accommodate the wider resolution range up to 1536²)
• Seed: 42
• Caption dropout: caption_dropout_rate = 0.15 (enabled to support CFG at inference time; Preview3 weights were trained without caption dropout, except for inpainting)
• Caching: --cache_latents_to_disk --cache_text_encoder_outputs_to_disk
• Attention backend: --attn_mode flash
• Hardware: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition

Difference from Preview3 common setup. Preview3 used batch size 6 with --gradient_checkpointing disabled at 1024²-only resolutions. For Anima-Base v1.0, the resolution range now extends to 1536², so the common setup uses batch size 12 with --gradient_checkpointing enabled. Per-model LLLite dims (--cond_emb_dim, --lllite_cond_dim, --lllite_mlp_dim) are set per-model rather than at a shared default.

Per-Model Details / 各モデルの詳細

1. inpainting (4ch conditioning)

• Conditioning: 4-channel — RGB image with the masked region blacked out, concatenated with a 1-channel binary mask. Trained with --lllite_cond_in_channels 4 --lllite_inpaint_masked_input.
• Generation method: dynamic mask generation (same procedure as Preview3 inpainting v1). For each training step, a mask is generated on the fly from the target image.
• Pairs: 4,000 (all target images).
• Per-model hyperparameters:
  • --learning_rate 1e-3
  • --timestep_sampling shift --discrete_flow_shift 3.0
  • --lllite_target_layers self_attn_q_pre,self_attn_kv_pre,mlp_fc1_pre (Q, K/V, and MLP fc1 all injected — wider injection than the other models)
  • --lllite_cond_resblocks 4
  • --cond_emb_dim 64 --lllite_cond_dim 128 --lllite_mlp_dim 64 (larger dims than the Preview3 common setup of 32/32/32 — unlike the other ControlNet types, inpainting must preserve the color information of the unmasked region of the input image, which requires more capacity)
  • --lllite_cond_in_channels 4 --lllite_inpaint_masked_input
  • --max_train_epochs 64 configured. Published weight: epoch 30 (≈ 10,770 steps), saved as anima-lllite-inpainting-v2.safetensors.
  • Wall-clock: ~17 hours 30 minutes on the hardware listed above. Loss was still trending down at the stopping point, but the model already produced clearly effective results, and continuing risked overfitting to non-essential details — so training was stopped here.

Usage notes / 利用上の注意

• It is recommended to use img2img with a mask (otherwise the colors may shift slightly). The sample image embeds the workflow.
• Please update the Anima LLLite ComfyUI node to the latest version.

2. any-test like (3ch conditioning)

• Conditioning: 3-channel RGB. Mix of five conditioning types — HED scribble, PiDiNet scribble, Grayscale A, Grayscale B, and lineart — all heavily augmented. (Default --lllite_cond_in_channels 3, no --lllite_inpaint_masked_input.)
• Generation method: each of the five conditioning types is generated for every target image, giving 20,000 conditioning images in total:
  • HED scribble / PiDiNet scribble / Grayscale A / Grayscale B — same generation pipeline, augmentation, and additional post-processing as Preview3-era any-test like v1 (see PREVIEW3.md).
  • lineart — extracted via a random mix of AniLines (basic / detail) and MangaLineExtraction at a sampling ratio of 1 : 1 : 1.5. For each image, the script randomly combines input preprocessing (scale / blur / contrast / gamma), model-internal parameters (sharpness / sobel ksize), post-processing (threshold / two-step / tone curve), morphology (thicken / none), and final blur / contrast / brightness / invert. The same lineart-specific additional augmentation used for Preview3 v1 is then applied on top. The Preview3 v1 lineart extractor was replaced with this mix because it was too slow at the 4,000-image scale.
• Pairs: 20,000 (4,000 target images × 5 conditioning types).
• Per-model hyperparameters:
  • --timestep_sampling shift --discrete_flow_shift 3.0
  • --lllite_target_layers self_attn_q_pre (Q only — narrower injection than inpainting)
  • --lllite_cond_resblocks 6
  • --cond_emb_dim 32 --lllite_cond_dim 64 --lllite_mlp_dim 64
  • --max_train_epochs 32 configured. Trained in two phases:
    • Phase 1 (Blackwell RTX PRO 6000 Max-Q, common setup): --learning_rate 1e-3, batch size 12, 6 epochs (≈ 11,000 steps), ~17 hours. Loss stopped decreasing around epoch 6, so training was continued on a separate machine.
    • Phase 2 (RTX A6000 × 2): --learning_rate 3e-4, batch size 8 per GPU × 2 GPUs (effective batch size 16), 6 epochs (≈ 9,000 steps), ~50 hours. Loss continued to decrease for the first ~2 epochs of this phase, then plateaued; training was stopped at epoch 6.
  • Published weight: cumulative epoch 12 (≈ 20,000 steps), saved as anima-lllite-any-test-like-v2.safetensors.

Usage / 使い方

License / ライセンス

Credits / クレジット

• ControlNet-LLLite (original SDXL implementation) and Anima port — kohya-ss.
• Lineart extraction models used for any-test like v2 conditioning generation:
  • AniLines (Anime Lineart Extractor) by zhenglin (MIT License)
  • MangaLineExtraction (PyTorch) by Chengze "Miaomiao" Li (MIT License)
• Preview3-era weights (lineart / depth / pose / fake scribble / any-test like v1 / inpainting v1) and their data-pipeline credits — see PREVIEW3.md.