You can deploy this model and start earning money today!

ZhuoranChen/
lingbot-va-mot

Name: lingbot-va-mot
Availability: InStock

diffusers

safetensors

robotics

video-language-action

mixture-of-transformers

other

Views

No views yet

lingbot-va-mot

Converted, not fine-tuned, base checkpoint for WanMoTTransformer3DModel — the Mixture-of-Transformers (MoT) video-action architecture from joliachen/lingbot-va (model_mot.py). 10.0B total parameters. Produced from the shared-backbone lingbot-va-base (WanTransformer3DModel) checkpoint via convert_to_mot.py.

A fine-tuned derivative of this checkpoint (LIBERO-Goal post-train) is available at ZhuoranChen/lingbot-va-mot-posttrain-libero-goal-gradaccum43.

How the action-stream weights were initialized

The action stream is losslessly cloned from the corresponding video-stream weights at conversion time — no interpolation, no random initialization, no dimension change (both streams are the same width, see below):

video-stream source	action-stream target
`blocks.N.attn1.*`	`blocks.N.action_attn1.*`
`blocks.N.attn2.*`	`blocks.N.action_attn2.*`
`blocks.N.norm2.*`	`blocks.N.action_norm2.*`
`blocks.N.ffn.*`	`blocks.N.action_ffn.*`
`blocks.N.scale_shift_table`	`blocks.N.action_scale_shift_table`
top-level `scale_shift_table`	`action_scale_shift_table_final`

applied identically for all 30 transformer blocks. action_embedder, action_proj_out, and condition_embedder_action.* were already present in the base checkpoint and are kept as-is (not cloned). patch_embedding.* from the base checkpoint is dropped (unused by the MoT model). At this conversion step — before any post-training — the action stream is a byte-identical copy of the video stream; the two only diverge once fine-tuning starts.

Architecture: 3072-dim action stream, not 768 (deviation from the lingbot-va paper)

The lingbot-va paper describes the action stream operating through a narrower 768-dim bottleneck (raw 30-dim action → 768 → … → 768 → 30). This checkpoint's architecture has no such bottleneck: the action stream runs at the full video-stream width, d_v = 3072, throughout:

action_embedder: Linear(30 → 3072) (weight shape [3072, 30])
action_proj_out: Linear(3072 → 30) (weight shape [30, 3072])
condition_embedder_action.* (text/time embedders, time projection): all at 3072, shape-parallel to the video-stream condition_embedder
every per-block action module (action_attn1/2, action_norm2, action_ffn, action_scale_shift_table) is shape-identical to its video-stream counterpart at 3072

Net effect: the action stream holds ~5.0B of the model's 10.0B params — full width/capacity parity with the video stream, rather than the paper's narrower 768-dim bottleneck design.

Config


`num_layers`	30
`num_attention_heads` / `attention_head_dim`	24 / 128 (→ d_v = 3072)
`in_channels` / `out_channels` (video, patchified)	48 / 48
`action_dim`	30
`text_dim`	4096
`ffn_dim`	14336
`attn_mode`	`flex` (train/eval), `torch` also supported for inference

Repo contents

Full standalone layout, ready to load directly:

transformer/ — this checkpoint's 10.0B-param MoT weights
text_encoder/ — google/umt5-xxl (stock, unmodified)
tokenizer/ — matching T5 tokenizer
vae/ — Wan2.1 AutoencoderKLWan (stock, unmodified)