Marionette factorises an interactive game world model into three stages. Only the first and
the third carry weights; the middle one is fixed geometry.
seed pose ──▶ dynamics ──▶ 276D world state ──▶ bridge ──▶ pose-control video ──▶ observation ──▶ RGB
ActionGPT zero-parameter, Wan2.2-Fun-5B-Control
+ PoseGPT deterministic fine-tune, chunk-relay
Files
file
size
stage
what it is
observation/diffusion_pytorch_model.safetensors
10.0 GB
observation
control-conditioned video diffusion transformer, fine-tuned from Wan2.2-Fun-5B-Control
dynamics/pose_gpt.pt
402 MB
dynamics
PoseGPT — predicts the next articulated state
dynamics/action_gpt.pt
61 MB
dynamics
ActionGPT — predicts the next action token
These are not standalone. The observation stage loads the VAE and the umT5-xxl text encoder
from the third-party base release, which is not redistributed here:
Seeds, the scanned terrain and the appearance reference are in the code repository, not here —
they pack small enough that a separate download would buy nothing.
Use
bash
1git clone https://github.com/AlayaLab/Marionette &&cd Marionette
2bash fetch_weights.sh # the weights in this repo3bash fetch_base_model.sh # the third-party base model4bash run_demo.sh # -> samples_out/.../rollout.mp4
Inference settings the released checkpoint was evaluated at: 704×1280, 40 steps, guidance 6.0,
81-frame chunks, 30 fps.
The prompt is part of the checkpoint
The observation model was trained on captions carrying an explicit appearance id:
Monster Hunter Wilds video game gameplay, stage 101, hunter appearance id 9 wielding weapon type 4, fighting monster id 19.
Pairing this checkpoint with an older caption format — or an older checkpoint with this one —
puts the text encoder out of distribution, and the failure is not subtle: monsters drop out of
the rollout entirely. Change the checkpoint and the prompt together or neither.
Training
Observation. Fine-tuned from alibaba-pai/Wan2.2-Fun-5B-Control in control_ref mode with
the first frame as the appearance reference, at 704×1280, 16×H200 FSDP. The released checkpoint
is step 17000. The last 2000 of those steps are on a corpus recorded with a revised capture
path in which objects absent from the pose-control signal are also absent from the RGB — mounts
in particular — which removes a class of targets the model previously had no way to predict.
Earlier stages of training span 26 monster species; the final segment is a single configuration
(one stage, one monster, one weapon class). The checkpoint is correspondingly strongest there,
and the multi-species montage on the project page is rendered from an earlier checkpoint for
exactly this reason.
Dynamics. ActionGPT and PoseGPT over the 276D articulated state at 20 fps, conditioned on a
scanned terrain height field, trained on a single monster species (em19) over 2241 segments.
Action vocabularies: 168 for the monster, 977 for the hunter.
Action ids are indices into this checkpoint's vocabulary and are not portable. The
vocabulary is built from the training corpus, so the same integer under a model trained on a
different corpus is a different animation or nothing at all — an earlier corpus here shares
almost none of this one's hunter ids. When the demos name 493 as ATTACK, that is a fact about
this checkpoint.
Reproducibility
The action stream is sampled, so the dynamics stage is only reproducible with a pinned seed.
run_stage1_render.sh pins TORCH_SEED=43; upstream leaves it unset. With the seed pinned,
the repository's verify_reproduction.sh reproduces its recorded pose video byte for byte and
its RGB rollout pixel for pixel.
Scope and limitations
The dynamics model covers one monster species. It is not a general character-motion model.
Rollout is chunk-relay: each 81-frame chunk is conditioned on the last frame of the previous
one, so appearance error compounds with horizon.
The bridge consumes a terrain height field scanned for specific stages. Novel geometry needs
a new scan.
Research artifact. Not a product, not a game, not a renderer for anything but this pipeline.
Licence and attribution
Our code: Apache-2.0 (see the code repository's LICENSE).
These weights: non-commercial research use only, no redistribution — see LICENSE.assets.
The restriction comes from the training corpus, not from the base model.
Base modelalibaba-pai/Wan2.2-Fun-5B-Control: Apache-2.0, not redistributed here.
VideoX-Fun (aigc-apps/VideoX-Fun), which the observation code is vendored from: Apache-2.0.
Training corpus: the WildWorld dataset — https://github.com/AlayaLab/WildWorld — released
for non-commercial research use, redistribution not permitted.
Rights in the game content the corpus was recorded from remain with its publisher, and this
project grants no rights in that content.
Two files in this project are unmodified recorded gameplay video, not model output:
data/first_frame_ref.mp4 and data/demo/aligned_ref.mp4. The observation model is conditioned
on a reference frame, and these supply it. They are short clips from the same recordings the
corpus was built from, and they are covered by the research-only terms above. Everything else
distributed here — seed states, scanned terrain, pose videos, rollouts and the gallery stills —
is numeric derivation or model output.
Citation
bibtex
1@article{meng2026marionette,
2 title = {Marionette: Predicting World States, Rendering Geometry, Painting Appearance},
3 author = {Meng, Zian and Li, Zhen and Li, Chuanhao and Li, Qiang and Zhang, Kaipeng},
4 journal = {arXiv preprint arXiv:2608.14530},
5 year = {2026}
6}
