MiniMax-H3 renders video and
synchronized stereo audio from one prompt, in one transformer, on two flow
schedules. larryvrh's Turbo LoRA
brings it to four sampling steps. This is both of them in the
CMF container — the DiT, the Qwen3-VL
prompt encoder, the video VAE decoder and the audio vocoder in a single
memory-mapped file — running on cortiq, a Rust binary with no ML framework
underneath.
The reference checkout is 124.4 GB across four files plus a ComfyUI install.
Here it is one file between 13.2 and 23.9 GB, and which one you take is
decided by your VRAM.
Same prompt, three files
"A corgi in a chef hat flipping a pancake, sizzling sounds and a cheerful bark."
— 512×288, 39 frames, seed 42, four steps, nothing but the prompt.
32B
ClipProj
two bits
23.9 GB — full 32B prompt encoder
13.2 GB — 4B encoder + ClipProj
18.7 GB — two-bit encoder
the reference
same scene, plainer set
a different animal, no pan
The 13.2 GB file is the one to take if you have 16–20 GB of VRAM: it holds the
whole run resident instead of paging, and it is still four bits everywhere.
Two bits is published as a dead end, not a choice —
why, with numbers.
The GIFs are silent and the audio is half the model, so take an
mp4.
One transformer denoises picture and sound together in one packed sequence.
Which file
file
size
VRAM
keyframes
mmh3-turbo-fl2va-q4tp.cmf
23.94 GB
24 GB+
yes
the default
mmh3-turbo-q4tp.cmf
23.47 GB
24 GB+
no
same, without the vision tower
mmh3-turbo-clipproj4b-q4tp.cmf
13.16 GB
16–20 GB
no
the small one, peaks at 15.1 GB
mmh3-turbo-clipproj4b-fl2va-q4tp.cmf
14.48 GB
16–24 GB
yes
the small one WITH start/end frames — the 4B vision tower and the VAE encoder join the compact build
mmh3-turbo-clipproj4b-fl2va-v2-q8_2f.cmf
26.90 GB
32 GB+
yes
eight bits: the two-field int8, w = q·row[o]·col[i]. More weight fidelity, and it runs on the card as of 0.5.94 — still without the fused kernels q4tp has
mmh3-turbo-fl2va-q2tp.cmf
18.74 GB
—
yes
don't render with this
Text-to-video everywhere; the fl2va files also take a first and/or last frame
(--first-frame/--last-frame, binary P6 PPM). The compact
clipproj4b-fl2va file was verified frame-for-frame: the conditioning
image comes back as frame 0 of the render. One honest caveat: its
ClipProj projection was fitted on text-only encoder activations, so with
a picture in the prompt the conditioning quality on complex scenes is
still being compared against the full-encoder fl2va files — report what
you see in the discussions.
(keyframes). The release's third path, ref2va, is not
ported. The Turbo LoRA is merged into the weights, so the file IS the 4-step
model — nothing else to download. 47.83 B parameters, 2 361 tensors,
cortiq verify clean.
Keyframe to video
The corgi flipping the pancake, started from one frame
bash
1cortiq animate mmh3-turbo-fl2va-q4tp.cmf \2 --prompt "the corgi lifts the pan and flips the pancake high, sizzling"\3 --first-frame keyframe.ppm --out flip.avi
One picture conditions the run twice, and both halves matter. Its VAE latent
becomes a row the DiT holds at a timestep of its own near 1 — a condition, not
noise being removed — and never denoises. The picture ITSELF goes to the prompt
encoder through Qwen3-VL's vision tower, as "<Picture 1>: " and a vision
block: at 512×288 that is 144 tokens of the 168 the prompt above carries.
Leave one out and the model is conditioned on something the reference never
conditions on.
--last-frame anchors the other end. The first frame is a geometry anchor and
is stretched to the canvas; the last one follows and is cover-cropped, which is
what the reference does with each. Frames come in as binary P6 PPM.
— or take a prebuilt archive from the
latest release
(Linux x86-64, macOS on Apple Silicon and Intel, Windows x86-64 and ARM64;
each ships a .sha256). Nothing else is required: no Python, no PyTorch, no
CUDA toolkit, no ffmpeg.
Check it took:
cortiq --version
2. Get the weights
One file. Pick it from Which file above — this is the
text-to-video default; swap the name for mmh3-turbo-clipproj4b-q4tp.cmf if
you are on a 20 GB card.
bash
1pip install -U "huggingface_hub[cli]"# only to fetch the file2hf download infosave/MiniMax-H3-Turbo-cmf mmh3-turbo-q4tp.cmf --local-dir .
Confirm it arrived whole — the container carries a hash per tensor:
bash
1cortiq verify mmh3-turbo-q4tp.cmf # → ✓ all tensor hashes match2cortiq info mmh3-turbo-q4tp.cmf # → arch, layers, 47.83B params
3. Render
bash
1cortiq animate mmh3-turbo-q4tp.cmf \2 --prompt "A corgi in a chef hat flipping a pancake, sizzling sounds and a cheerful bark."\3 --width 512 --height 288 --frames 39 --steps 4 --seed 42\4 --out corgi.avi
That writes corgi.avi — MJPEG video with PCM stereo, playable in VLC, mpv,
QuickTime and Windows Media Player — and corgi.wav beside it. The JPEG
encoder and the RIFF muxer are inside the binary: a pipeline that ends in a
shell-out to a 20 MB dependency is not a pipeline you can ship. If you want an
mp4 for a browser, remux it yourself; the model never needs one.
On a GPU. Nothing to opt into any more: cortiq probes this file's own
first qkv weight against the host on startup and takes the device arm only if
they agree, so the arm that renders is the arm that was checked.
On one RTX 5090, 512×288, 39 frames: 60.2 s at the default four steps
(91.6 s at eight, 42.8 at two). RUST_LOG=info prints a per-stage breakdown
for every run, and the full table is under What it costs to run below. The
whole pipeline stays on the card —
the DiT block, both VAE decoders, and the vocoder's dilated convolutions — so
nothing but the finished frames crosses the bus. CMF_MMH3_GPU=0 forces the
host path if you want to compare.
Two cards. A render does not split across them, and should not: the DiT
block and both decoders are already resident, so a second card would only add
a bus crossing to a pipeline that no longer has one. Two cards double your
clips, not your clip — run two processes, one pinned to each:
cortiq gpu lists the cards and their indices. For text models the same
binary both splits and replicates across cards — see
docs/MULTI_GPU.md.
Options that matter
flag
default
what it does
--width / --height
512 × 288
multiples of 32. The trained short edge is 768; below ~256 the model drifts off-distribution
--frames
39
at 24 fps, snapped up to the model's 17k+5 grid: 5, 22, 39, 56, … 124. 124 ≈ 5 s, and 124–362 is the validated range
--steps
4
what the Turbo LoRA is trained for. More still helps a little
--seed
42
same seed, same prompt, same size → the same clip, byte for byte. This is now true on the GPU too: the op arbitration used to alternate arms on real data while it made up its mind, and two runs of one binary could differ
--quality
92
JPEG quality of the AVI's frames
--stock-sampler
off
integrate the audio on the video's clock, as a single-schedule sampler does. Wrong at 4 steps — it is here to hear how wrong
environment
what it does
CMF_MMH3_GPU=1 / =0
force the device or the host path instead of letting the parity probe choose
CMF_GPU_PROBE=0
pin the op arbitration (already the default for animate, so a seed reproduces)
CMF_THREADS=n
cap the worker pool (defaults to the machine's cores)
CMF_ANIM_PROF=1
per-step rms of both latent streams and both velocities
CMF_MM_KILL=0
never fall back to the host on a slow device op. The engine treats three consecutive over-budget GEMMs as "another process owns the card" and finishes the run on the CPU; a weight paging in from disk is exempt, but on a machine where the file exceeds RAM the first steps can be slow for reasons that are not contention (see the field notes)
What it needs
RAM at least the file's size — 24 GB — or every step faults on non-resident
pages; the weights are memory-mapped, not read. Disk: 24 GB. A GPU is optional
and wants ~14 GB of VRAM for the DiT's planes. No network access at run time.
mmh3-turbo-clipproj4b-q4tp.cmf asks for 14 GB of RAM and disk instead, and
peaks at 15.1 GB of VRAM — a whole-run maximum, polled, not a snapshot. That is
what makes it the one to reach for on a 20 GB card: below that the prompt
encoder pages, and on this workload paging costs more than any kernel.
Field notes: 24 GB Macs and 20 GB cards
Two users measured what this card could not, and the engine changed on
their reports (HF discussions #1, #2 and #4). The numbers, as they sent them:
machine
file
render
result
RTX 3080 20 GB, Windows
mmh3-turbo-q4tp (25.2 GB)
512×288×39, 4 steps
1218 s — the encoder did not fit the 19 GB budget and paged every step
RTX 3080 20 GB, Windows
mmh3-turbo-clipproj4b-q4tp (13.2 GB)
same
105 s — resident; denoise 62.8 s, video VAE 30.6 s
Mac mini M4 24 GB
mmh3-turbo-clipproj4b-q4tp
512×256×39, 4 steps
174 s — denoise 122 s, video VAE 41 s
Mac mini M4 24 GB
mmh3-turbo-clipproj4b-fl2va-q4tp (14.5 GB)
22 frames from a keyframe
92 s, no swap
Mac mini M4 24 GB
mmh3-turbo-fl2va-q4tp (25.7 GB), 0.5.79
10 / 40 frames
48 s / 140 s per denoise step on the GPU, 20 GB resident, 0.3 GB swap
Mac mini M4 24 GB
same, 150 frames
8 GB of swap and a sawtooth — the activation cache no longer fits
Mac mini M4 24 GB
mmh3-turbo-clipproj4b-fl2va-v2-q4tp (14.5 GB)
448×768, 50 frames
136 s per denoise step, faces hold through the clip; 90 frames at that size is the paging threshold
Mac mini M4 24 GB
mmh3-turbo-fl2va-q4tp (25.7 GB)
512×256
safe to ~70–90 frames; 768×448 caps at 40–50 (90 frames: 22.2 GB RAM + 1.1 GB swap, the GPU stalls)
Mac mini M4 24 GB
any file, --first-frame
the video-VAE encoder of the keyframe ran ~100–140 s on the CPU (encode 0/1 (140.8s)); text-to-video skips it (0.1 s)
What follows from them, if you are on such a machine:
On a 24 GB Mac the 25.7 GB file works after 0.5.79, for clips of
≤ 40 frames. The prompt encoder's pages are released after the text
encode; what remains — DiT, VAE decoders and the activation cache — fits
up to ~40 frames. Past that the cache spills to swap. For longer clips
chain 40-frame chunks (--last-frame of one render becomes
--first-frame of the next) or use the clipproj4b files, which leave
the room.
The Metal driver budgets by buffer size, not resident pages. The
weight arena's overlapping windows read as ~27 GB to the driver even
after the encoder is released, so the first ops after the encode page
from the SSD and can take seconds. Before 0.5.80 a single such op tripped
the contention kill and the rest of the run walked the CPU (>60 s a
step); the kill now needs three consecutive strikes, exempts weights
that were not resident, and CMF_MM_KILL=0 turns it off. If a run still
says device contended, CPU for the rest of the process on a machine
nobody else is using, that variable is the answer.
The kill is disarmed during the prompt encode (0.5.80). The
encoder is a one-shot pass over 12 GB of weights; on a machine the file
does not fit it streams from disk, and its GEMMs run over any budget
for reasons that are not contention — the report that had to gut
mm_kill in the source was on exactly that. The kill now arms only
when the denoise loop starts, and three strikes are still needed.
The keyframe encode is CPU work today. With --first-frame the
video-VAE encoder runs the reference picture on the host (~100–140 s
on an M4 for a 448×768 frame; 0.1 s without a keyframe). 0.5.80 puts
the pool's workers on the performance cores (they were landing on the
E-cores with the P-cores asleep — that report's asitop), which
shortens it; a device path for the encoder's 3-D convolutions is the
real fix and is on the list.
Frame budgets on 24 GB, from that user's sweep: with the 25.7 GB
file 512×256 is safe to ~70–90 frames and 768×448 caps at 40–50; with
the 14.5 GB v2 file 448×768 at 50 frames is the sweet spot (136 s per
step) and 90 frames is where paging starts. Past those the machine
swaps and the GPU stalls to nothing — better to chain clips than to
push the frame count.
The draft → final workflow. Block the shot on clipproj4b-fl2va-v2
(90 s on the Mac; v2 holds faces through a clip — "25 GB-level face
consistency at 14 GB speed", that user's words), then pay the full
encoder once for the final take. Identity of specific real people is
still the full encoder's territory.
--height 256 instead of 288 halves the video-VAE decode on every
machine (three 256-pixel tiles instead of six).
Voices are prompt space here, not in the release. Speaker identity,
timbre, pace and emotion respond to stage directions in the prompt, and a
fixed seed keeps the same actor across takes — which is how to work with
this file today. But the earlier claim on this card that "H3 has no
reference-audio input" was wrong, and a user was right to push back
(discussion #3): the release is tagged audio-to-audio-video and
video-to-audio-video, and the DiT takes both as conditioning rows. What
is missing is on our side and it is not training — see
What this port does not do yet.
Draft small, refine big (two-stage with LTX-2.5)
Since 0.5.100 the Turbo draft can hand its frames to
LTX-2.5 for a short
refinement pass at full resolution — the draft supplies composition and
motion, LTX reinvents the detail:
On a 5090: draft 40 s, refine 499 s at 1024×576. A/B of the same seed is
in samples/refine_ab_*.mp4. The two containers don't want RAM at the
same time — run the stages sequentially. --video-strength is the dial:
0.45 keeps the draft's motion, higher repaints more.
Making it smaller
Half this file is the PROMPT ENCODER — 12.2 GB of Qwen3-VL against the DiT that
actually draws. Two ways to act on that were tried, and the three clips at the
top of this card are the result: squeezing the encoder does not work,
replacing it does. Here is why, in both directions.
Two bits: smaller, faster, and answering a different question
The obvious next cut is the DeepSeek-V4 policy: gate/up at two bits,
everything else at four. It builds — 23.94 GB down to 18.74, and
with a device kernel of its own it renders faster than the four-bit
file — 217.3 s against 258.4 when the pair was measured, because there is
less weight to move. (Both numbers are from the build of that day; the
four-bit file renders the same clip in 60.2 s now. The ratio is what
carries over, not the seconds.)
cortiq verify passes. The file is here.
It also stops following the prompt, which is why it is not the one to
reach for.
four bits
two bits
q4tp — 23.94 GB
q2tp — 18.74 GB
Same prompt, same seed, same four steps. On the left the corgi is behind
a pan with batter in it, drawn flat and clean, which is what was asked
for. On the right it is a different animal in a different style with
no pan and no pancake at all, over a washed-out ground with visible
texture noise. That is not a quantizer trading detail for size; that is
a model answering a different question.
The likely culprit is where the two bits landed. Half this file is the
PROMPT ENCODER, and the policy put two bits on its gate/up planes along
with the DiT's — so the loss falls on the part that decides what the
clip is about, not on the part that draws it. A two-bit build confined
to the DiT would save ~2.9 GB instead of 5.2 and is the version worth
measuring next — the packer's policy is one predicate,
is_wide_plane, if you want to try it. The file above is published so
that experiment starts from something rather than nothing; four bits is
what to render with.
A smaller encoder: replace it, don't squeeze it
The two-bit section above ends on a diagnosis: half this file is the PROMPT
ENCODER, and squeezing it is what broke the prompt. There is a second way to act
on that diagnosis — don't compress the encoder, replace it.
ClipProj fits a ridge
regression from a small Qwen3-VL's hidden state into the space the DiT was
conditioned on. Same tokenizer, same family, one affine map:
mmh3-turbo-clipproj4b-q4tp.cmf is the text-to-video file with Qwen3-VL-4B
tapped at layer 24 standing in for the 32B tapped at 50, plus the 304 MB
projection kept exact — f32, no quantization on the piece that carries the
whole substitution. Everything else is copied through byte for byte: DiT and
both VAE decoders, still q4tp. Nothing anywhere is two bits.
23.47 GB → 13.16 GB. The encoder went from 552 tensors to 277.
Does it still mean the same thing?
That is measurable without rendering a frame: dump what the DiT actually
receives (CMF_TE_DUMP=<path>) from both files on one prompt, and take the
cosine per token.
mean cosine to the 32B
worst token
4B tapped at 24
0.9198
0.7982
4B tapped at 25
0.8452
0.5910
floor — two different tokens of the 32B itself
0.5914
The tap index is 0-based, and off-by-one is a real failure mode rather than a
theoretical one: at 25 the worst token sits on the floor, uncorrelated.
Token 0 is not projected at all but replaced by a stored sink_out — the
attention sink is an outlier no regression fits — and it lands at cosine
0.9999 against the teacher's, which is the sharpest single confirmation
that the mechanism is wired right.
What it buys
The one number that transfers between machines is the encode stage, because
that is the only stage ClipProj replaced. Same prompt, seed and four steps,
512x288x39, measured inside one run:
prompt encode
mmh3-turbo-q4tp — 32B tapped at 50
49.2 s
mmh3-turbo-clipproj4b-q4tp — 4B tapped at 24
1.9 s
Everything after that is residency, and residency is a property of YOUR card,
not of this file. On the 24 GB RTX 3090 these were taken on, the whole render
came out roughly half the time of the 23.47 GB file and the video VAE decode
more than halved — but totals on that box drifted between 140 s and 210 s for
one unchanged configuration as the card warmed, so treat the ratio as a
direction and measure your own.
The point is the 20 GB card this variant exists for, where the difference is
not a ratio but a cliff: a reader of this repo measured 20 minutes for one
512x288 clip on an RTX 3080 20 GB, paging the 23.47 GB file through a ~19 GB
weight budget. At 13.16 GB, with a polled whole-run peak of 15.1 GB of VRAM,
there is nothing left to page.
The parity probe takes the device arm on both files at the same rel rms 4.65e-3, which is its own small proof that the DiT came through untouched.
What it costs in quality
Judge it on the clip at the top of this section, not on a frame: a still
catches the pancake at rest and reads as a loss that is not there.
Across the 39 frames the 4B does the whole job: corgi, chef hat, flat clean
style, and the pancake leaves the surface and comes back, the dish empty
underneath at the top of the toss. The verb survives — that is what q2tp
lost, along with the animal and the pan.
What drifts is set dressing. The 32B renders the cooking surface as a griddle
and puts patterned wallpaper behind; the 4B gives a rimmed white plate and a
plain ground. A 0.92 cosine is close, not equal, and where it is not equal is
the scene's furniture rather than its subject or its action.
So the ordering is not "smaller is worse". The two-bit file answers a different
question; this one answers the right question with a plainer set, at 56% of
the size. If you have the VRAM for the 32B encoder, use it —
its framing is richer. If you are paging, this is the better trade, and unlike
the two-bit build it is a trade rather than a loss.
Sound is untouched either way: the audio branch never sees the prompt encoder.
The clip with its audio.
--te-layers is not a size knob, it is the tap: layers above it never execute,
so packing them is pure file. --clip-proj carries the projection in exact and
replaces the encoder as a whole component — matching tensor names alone
would leave te.layers.24..49 of the 32B behind, six gigabytes that
num_hidden_layers then excludes from the forward while disk and VRAM budget
still pay for them.
The encoder is Comfy-Org/Qwen3-VLtext_encoders/qwen3vl_4b_bf16.safetensors, and the projection is
NicoLab28/ClipProj-MiniMax-H3mmh3-4b-ClipProj-celeb-mlp.safetensors — use the file the projection was
fitted on, since the map is only valid for those exact weights. An 8B
projection is published too and scores higher on its author's corpus (0.8037
against 0.7930); it costs about 4 GB more.
What it costs to run
The file is memory-mapped, so plan on RAM at least its size or every step
touches non-resident pages.
One RTX 5090, 4 steps, 39 frames. RUST_LOG=info prints this breakdown for
every run:
text
denoise
video VAE
audio VAE
total
512×288
2.5 s
34.8 s
16.6 s
4.3 s
60.2 s
512×256
2.3 s
31.4 s
8.4 s
4.3 s
48.5 s
256×160, 22 frames
15.9 s
512×288, host only (CMF_MMH3_GPU=0)
2.6 s
363.5 s
271.5 s
7.1 s
646.1 s
The card is 10.7× the host path on the same machine, same seed. At 8
steps a 512×288 clip is 91.6 s; at 2 it is 42.8. The same 4-step
render took 172 s when this card was written — the pipeline has since moved
onto the card end to end, both VAE decoders with it.
Nearly all of the decode is the video VAE — the vocoder is 4.3 s of it. The
packed sequence is [text | audio | video] and everything attends to
everything, so cost grows with the token count and then with its square: a
512×288 second is five times the tokens of a 256×160 one.
A free 2× on the decoder, if you want it. The video VAE decodes in
256-pixel tiles, always, and grows the OVERLAP rather than the tile count —
so a 288-pixel edge is covered by two 256-pixel tiles overlapping by 224, and
you pay for 512 rows to get 288. An edge of exactly 256 is one tile. 512×256
therefore decodes three tiles where 512×288 decodes six, for 89% of the
pixels. Measured on the current build: the video VAE goes 16.6 s → 8.4 s and
the whole render 60.2 s → 48.5 s. The schedule is the reference's and this
port reproduces it exactly; picking an edge that lands on it is free.
Host and device do not agree to the last bit, and neither is wrong. The
host arm quantizes activations to int8 (CMF_SDOT) where the device
dequantizes to f32, so the two renders differ by a few per cent in latent rms
and visibly in fine texture. Set CMF_SDOT=0 on both sides to compare
arithmetic instead of that approximation.
Why the device took a while to trust. It is no longer opt-in — the parity
probe decides per file — but getting there took three fixes, and one thing is
still held back.
The engine's blocked f32 GEMM cached its weight-side device buffer by
pointer address. Every batched attention allocates one k/v scratch pair per
call and refills it per head — same address, different matrix — so head 0's
keys came back for every head, on the GPU only, silently. It is keyed on a
content fingerprint now. The same GEMM also took every job over 4 M MACs on
sight with no CPU arm to lose to, which on this model's decoder was three
times slower than the host it displaced; it goes through the same
measure-don't-assume probe as every other op class now, and on this stack the
probe hands that work back (0.24 ms device against 0.13 host) while sending
the weight GEMMs to the card (25.8 ms against 92.0).
Still held: the cooperative-matrix kernel runs this model out of f16 range.
At 256×160 the render is correct; at 512×288 the audio stream goes NaN on the
second sampling step and the video follows. Bisected — CMF_BAKE_GPU=0 does
not help, CMF_COOP=0 does — so cortiq animate pins CMF_COOP=0.
That hold is specific to this model, not a verdict on the kernel: the image
model on the same card and the same kernel renders 20.5 s without it against
14.8 with, and the two agree to 42.6 dB — the price of f16 operands, which
the kernel documents, not a fault. MiniMax-H3's activations are simply larger.
Giving that kernel a scale is the next real speedup here.
What the conversion did
The adaLN collapse. Forty per cent of the released DiT is one matrix per
block: adaln_proj.linear is [96768, 2688], 520 MB at bf16, 13 B of the
model's 33 B parameters — for a map whose input is one number, the timestep.
Its output over the whole schedule is a one-dimensional curve in R^96768, and
Comfy-Org's pruned checkpoints already ship it as one: an adaln_t_table of
[1025, 8] shared by every block and per-block weights of [96768, 8].
Measured against the full matrix on block 0 (tools/mmh3_fetch.py check, which
range-reads 520 MB out of the 66 GB file rather than downloading it):
adaln max|Δ| 8.0e-4 rms 8.7e-5 against a signal of rms 0.464
time-curve singular values 1..12, relative:
1.00e0 2.96e-1 1.05e-1 6.63e-2 6.60e-3 2.11e-3 5.61e-4 2.92e-4
3.67e-5 2.73e-5 1.32e-5 1.34e-6
The ninth singular value is already 3.7e-5 of the first. Rank eight is not an
approximation anyone should feel nervous about; the 26 GB is redundant.
The Turbo LoRA is written against the FULL matrix (lora_A is [16, 2688]),
which is why the ComfyUI node re-injects the time conditioning at run time when
the base is pruned. cortiq animate-pack does it once, at conversion:
adaln(t) = W_p · u(t) + b + B · (A · silu(e(t)))
= [W_p | B] · [u(t) ; A · silu(e(t))]
— a rank-24 curve, driven by a [1025, 24] table per block. 4.6 MB a block
instead of 520, with the LoRA already inside it.
The rest.
Backbone — bf16 → q4tp, 4.16 bits a weight with a predicted per-row
scale ladder. The LoRA's rank-64 update is merged before quantizing.
Prompt encoder — Qwen3-VL-32B truncated to 50 layers, 51.5 GB → 12.2 GB.
It is the largest single component of the file and it runs once per
generation.
Video VAE — decoder only. It is a ViT3D, not a conv stack: 36 transformer
blocks over the latent grid and one linear that expands each cell into a
4×16×16 block of pixels. The 3-D causal CNN encoder is a third of the
checkpoint and text-to-video never runs it.
Audio VAE — decoder only, f16. Quantizing a vocoder buys 45 MB and costs
audible hiss. Its 254 kaiser-sinc resampling filters are read from the
checkpoint rather than re-derived — the design formula is in the code as a
fallback, but a filter you compute is a filter that can drift from the one
the weights were trained against.
Integrity: 47.83 B parameters over 2 361 tensors; cortiq verify checks
every one against the directory's hashes.
On parity
Established, not assumed, and separately for each of the four stacks. The
reference is ComfyUI's own module, run on a toy checkpoint carrying the
release's real tensor names and the release's real schedules — tools/
builds them, tools/mmh3_toy_gate.sh runs the diff. The packs are exact f32
on purpose: q4tp's noise floor sits an order of magnitude above the
arithmetic difference these are looking for, so quantizing here would pass a
broken port.
stack
worst
rms
signal rms
DiT — video velocity
8.8e-5
2.1e-5
0.515
DiT — audio velocity
5.2e-5
2.5e-5
0.409
DiT — token refiner
8.3e-7
2.6e-7
1.003
Qwen3-VL encoder
1.1e-6
3.3e-7
0.812
video VAE decoder
4.2e-7
4.0e-8
0.470
audio VAE decoder
1.7e-9
3.5e-10
8.9e-4
A dozen conventions in this model pass at one token and fail differently at a
hundred, which is why the toys are not one-vector unit tests: the packed
layout's cursor, the video time axis's 1,4,4,4,4 span pattern, which 96 of 128
head dimensions rotate, the adaLN row order (timestep-major, modality-minor),
the video VAE's 256-pixel tiling — global attention makes a tile a different
computation from a whole frame, so the tiling is part of the output, not a
memory strategy — and the audio stream's separate clock.
Two clocks
The video and audio latents ride different flow schedules (shift 12 and 3).
The sampler walks the video grid, which at four steps is
1, 0.973, 0.923, 0.8, 0, and integrates the audio on its own remap of it.
Stepping both on the video grid is what a stock sampler does; it is fine at
twenty steps and wrong at four, because over the last interval Δσ_a and Δσ_v
differ by a factor of three and no per-step slope correction survives a step
that large. --stock-sampler reproduces the broken behaviour if you want to
hear it.
What this port does not do yet
The release is tagged for six conditioning paths. This container packs one of
them — fl2va, text and/or keyframes → video + audio. What the others need is
listed here so nobody has to guess whether it is a missing feature or a
missing possibility. None of them needs training.
the release's path
what it takes
what is missing here
audio-to-audio-video
a reference soundtrack
the audio VAE's encoder half. The container carries the decoder only — pack_audio_vae skips the encoder, pre_block and the mean/logs heads as unused. The DiT side already exists: the packed layout has a reference-audio segment kind and its own condition timestep
video-to-audio-video
a reference clip
the sampler plumbing. The video VAE encoder is already packed in the fl2va files — it is what encodes --first-frame — so this is a layout and CLI change, not a weight change
ref2va (subject / character references)
1–N reference images
a different DiT checkpoint (minimax_h3_ref2va_*) with its own turbo LoRA. It would be a second container, not a flag on this one
Two more, on the runtime side: the latent upscaler published for H3
(a 345 M-parameter 3-D conv net) is not ported, and there is no fused
device kernel for adapter branches on this model — see below.
The eight-bit build, and what it costs today
mmh3-turbo-clipproj4b-fl2va-v2-q8_2f.cmf (26.90 GB, 26.21 B parameters,
cortiq verify clean) packs the DiT as q8_2f — the two-field int8,
w = q·row[o]·col[i]: eight bits with a second scale field along the input
axis, which is where an activation-outlier channel shows up from the weight
side. As a codec it is strictly more faithful than the four-bit ladder.
It renders on the card — after two gates were fixed, and the story is
worth reading if you pack your own containers.
The first run of this file left the RTX 5090 idle at 2 MiB and took
357.3 s. Neither cause was a missing kernel:
The startup probe matched on Q4TiledP by name and dtype. Finding no
four-bit qkv weight it declared the host path for the whole render — for a
codec that has a device GEMM of its own. The two-field int8 folds its
column field into the activation and what is left is the per-row int8
kernel both backends already ship. The probe now asks the tensor which
entry point its codec has (QTensor::device_matmat).
The weight-residency budget was the whole heap minus a gigabyte. That
survived only because every container published before this one had
weights far under it; 24 GB of eight-bit weights took the card and the
first scratch allocation died with wgpu error: Out of Memory. A quarter
of the heap is held back now — 32 GB card → 24 GB of weights.
With both: 171.5 s on the same card and clip (103.3 s denoise,
62.2 s video VAE), against 357.3 s on the host, and the probe agrees with
the CPU arm to 5.77e-3.
It is still slower than the four-bit file's 60.2 s, and that part is about
kernels: q4tp has the fused qkv → attention → output submission and the
packed FFN, q8_2f goes through the generic per-op GEMM. Fusing those for the
two-field codec is the next piece of work, and it is worth roughly what the
fusion is worth on q4tp.
The net is LBH-123-AI's
— 345 M parameters, twelve residual blocks and six temporal convolutions on
each side of a trilinear resize, with a scalar scale embedding modulating
every block. It is ported into the engine and gated against the node's own
torch module on the same weights: worst 6.68e-6, relative rms 3.92e-7.
lanczos
the net
the 512×288 render, resampled to 1024×576
the same latent through the net, decoded at 1024×576
Same seed, same prompt, same denoise. On the left a resampler has nothing to
add and softens hair, coat seams and the wet road; on the right the detail is
generated. Measured on an M4 (24 GB), 22 frames: denoise 76.9 s, upscale
13.8 s, VAE 98.2 s — 193 s for a 1024×576 clip whose denoise only ever paid
for 512×288. The upscale itself is the cheapest stage in that list.
Why it matters on a 24 GB machine: the alternative is decoding through a
5 B-parameter VAE, resizing pixels and encoding again — the expensive path —
or interpolating the latent, which is the cheap path that ghosts. This is
neither. You keep the frame budget of the small render and get the detail of
the large one, and the file is loaded beside the container like an adapter,
so it stays under its own licence.
Every eighth .ppm of the directory becomes a condition pinned to its own
moment of the render, and the source clip is mapped onto the render's length
by position — a 100-frame reference conditions a 39-frame clip at the same
moments, not the same indices. This is the fl2va keyframe path with more
than two frames. It is not a port of the release's own v2v node, which
conditions differently; what it gives you is composition and motion carried
across a chain of shots, which is what the keyframe hack in discussion #6 was
reaching for.
Adapters at runtime
Community LoRAs for H3 run against this container as they ship:
bash
1cortiq animate mmh3-turbo-clipproj4b-fl2va-v2-q4tp.cmf \2 --prompt "r34l1sm a woman in a red raincoat on a neon street, close-up"\3 --lora h3-realism-people.safetensors --lora-strength 0.8\4 --out take.avi
--lora reads a .safetensors in any of the three conventions in the wild
(diffusion_model.…, base_model.model.dit.…, or the bare module path), at
F32/F16/BF16, with either lora_A/lora_B or lora_down/lora_up naming.
It binds attn.qkv_proj, attn.out_proj, mlp.fc1, mlp.fc2 on all fifty
blocks and on the two token-refiner blocks, and it prints what it bound:
lora: rank 32, 104/104 branches bound
Branches it cannot bind are named rather than dropped in silence. The one real
gap is adaln_proj.linear: this container carries the modulation as a rank-24
curve over the timestep (that collapse is 40% of the released model's weights
and most of why the file is 14 GB), and folding an adaLN update into it needs
the time embedding, which only the packer has. Adapters that touch adaLN —
the spatial-physics and streaming ones — therefore land partially, and say so.
Bake them in instead with animate-pack --lora … --time-embedder …, which is
exactly how the Turbo LoRA got into these files.
What it costs. On an M4 (24 GB), 512×288, 22 frames, four steps, with
fal/MiniMax-H3-Realism-People-LoRA (rank 32, 104 branches, all of which
bind), the two runs taken back to back with memory free: denoise 133.6 s
with the adapter against 126.1 s without — 1.06×. The video VAE, which no
adapter touches, moved 53.4 → 56.9 s between those same runs, so on this
machine the adapter costs at or below the noise. The branch itself is half a
per cent of the projection's arithmetic and rides inside the base GEMM's Metal
submission; what it can cost is the attention fusion standing down, because
a branch on qkv_proj or out_proj needs the panels that fusion keeps on the
card — visible where the DiT is fully resident, hidden where weights stream.
--lora-strength 0 reproduces the base render byte for byte.
Which parts of an adapter matter.CMF_LORA_PROBE=1 prints every branch by
its measured contribution ‖s·ΔY‖/‖Y‖, and CMF_LORA_ROUTE=<r> switches off
the ones below r. For the Realism adapter the loudest branch is 150× the
quietest, blocks 0–2 contribute nothing it would miss, and 41 of its 104
branches carry the look:
lora branches by contribution ‖sΔY‖/‖Y‖ (41 of 104 live):
0.1633 on blocks.13.attn.qkv_proj
0.0946 on blocks.9.attn.qkv_proj
…
0.0011 off blocks.1.attn.qkv_proj
Rendered on those 41, the clip still looks like the adapter and not like the
base (13.34 dB PSNR against the base, where the full adapter is 13.76, and
16.75 dB against the full adapter). It did not make the step shorter on Metal
— the reason, and where routing does pay, is in
docs/LORA.md.
Provenance
Weights derive from MiniMax's H3 release as repackaged by Comfy-Org, and from
larryvrh's Turbo LoRA; both remain under their own licences. The Turbo LoRA is
a preview — its own card notes plastic-looking skin and over-sharp grain at
ckpt850, and nothing here changes that. The CMF container and the cortiq
runtime are Apache-2.0 (see the repository's LICENSE and PATENTS.md).
