The VIDEOONLY distill LoRAs now have their own repo:joeygambino/ltx23-videoonly-distill-loras.
They still ship here for continuity, but the dedicated repo is where they are
documented (measurements, strength guidance, the "small strip is the better
strip" result) and where updates will land.
JoyAI-Echo x LTX-2.3 — echoVid + ltxAud surgical merge
A word on expectations: the multishot workflow this feeds is powerful but
not one-click - first renders usually take some tuning to your VRAM, system
RAM and model build. The workflow repo's
INSTRUCTIONS.md covers setup and known failure modes, and I am happy to
troubleshoot: open a discussion and I will answer.
JoyAI-Echo's video/conditioning branch married to LTX-2.3-distilled-1.1's audio branch.
Speed: turn on fp8_scaled_mm
If your renders crawl in the JoyAI-Echo pack, this is almost always why. It is
off by default.
fp8_scaled_mm stores the DiT linears as float8_e4m3fnand computes the
matmuls natively via torch._scaled_mm on RTX 40/50-series — unlike
fp8_transformer there is no per-layer upcast tax.
source file in model_file
fp8_scaled_mm
result
bf16
ON
linears downcast at load, native fp8 compute — ~22 GB resident, sequential_offload can go OFF
INT8 ConvRot
ON
reconstructs to bf16, then downcasts — identical to the bf16 row, at ~60 % of the download
bf16 or INT8
off
runs in bf16 — ~40 GB staged, streams over PCIe on a 32 GB card
fp8
off
upcasts to bf16 at load — saves download size only, not memory or time
fp8
ON
hard error. The toggle downcasts from bf16 itself; an fp8 file would load norms and tables as fp8 too and crash the denoise pipeline
The short version: feed it bf16 or INT8 and turn the toggle on. The
fp8 file is for running without the toggles.
(Under stock ComfyUI rather than the pack, the INT8 build instead runs on the
native W8A8 loader — a different path with its own speed profile.)
Render at 24 fps — voices and accents (2026-07-30)
LTX-2.3's joint audio-video prior is 24 fps-native, and render fps turns out
to be a hidden accent dial: at 25 fps the same prompt and seed render
non-rhotic southern British, at 30 fps broad Australian — and any off-24 fps
setting overrides accent wording in the prompt entirely (a few conditioning
tokens cannot outvote a geometric signal present in every attention op at
every step). Verified by A/B on identical configs: 24 fps = rhotic General
American at 5/5 reviewer confidence, same text and seed.
Keep video_fps at 24 and state the accent in the positive prompt
("in a casual American accent"). Flip side: if you want an authentic
British or Australian voice, rendering that scene at 25/30 fps is more
consistent than any accent wording.
Demo
THE WITNESS - five shots, five locations, one woman, one voice; nothing carries her between shots except one repeated sentence and the cross-shot memory bank. Rendered with the bf16 checkpoint in this repo under the fp8 scaled-mm compute path. AI-generated video and audio.
Two strong models, each with one disqualifying weakness for multi-shot
character work:
identity across shots
voice quality
JoyAI-Echo
holds (paired memory bank)
B-grade, TTS-flat
LTX-2.3 distilled-1.1
drifts — recasts between shots
A-grade, natural
Neither is shippable alone for a talking-character piece: Echo keeps the same
person across ten shots but sounds synthetic; LTX sounds like a real performer
but casts "three different actresses" across three shots.
This merge takes the half that works from each.
What was merged
Embedded in the file's own __metadata__ as rift_built_from:
SURGICAL MERGE: video/conditioning+av_ca-modulation from JoyAI-Echo-release;
audio + both cross-attns from distilled-1.1
From JoyAI-Echo release — video branch, conditioning path, and the
audio-video cross-attention modulation. This is what carries the slot-paired
memory bank, so face/wardrobe identity survives across shots.
From LTX-2.3 distilled-1.1 — the audio branch and both cross-attention
stacks. This is what produces the natural voice and lip-sync.
Architecture is unchanged (AVTransformer3DModel, 48 layers); this is a
tensor-level graft, not a retrain or a weight-average.
Both are full checkpoints — DiT + video VAE + audio VAE + vocoder +
text connectors — so they drop into a JoyEcho/LTX-2.3 graph as a single
model_file with no companion checkpoint needed.
All 5,947 tensors are present, **including the four
text_embedding_projection.* tensors**, so each file loads as a single
self-contained artifact.
That last point is worth knowing if you also run GGUF quants of this family.
LTX-2.3 ships those four tensors as a separate ~2.2 GB component file
rather than inside the DiT, while JoyAI-Echo bundles them in — the two
projects package the same weights differently, and they are byte-identical
between the two. Some JoyAI-Echo GGUF conversions follow the LTX layout and
leave them out, which is a reasonable thing for a converter to do, but if
nothing in your graph then loads the companion file those tensors get built
random-initialised. The symptoms are subtle enough to be misdiagnosed as a bad
prompt — robotic voice, the model reading prompt text aloud, identity resetting
between shots — and load logs look clean either way, because they sit outside
the swept path. Verify any checkpoint with:
python
1234from safetensors import safe_open
5678with safe_open("model.safetensors", framework="pt")as f:9101112 proj =[k for k in f.keys()if"text_embedding_projection"in k]13141516print(len(proj),"expected 4")171819
Observed behaviour
Used in production for multi-shot narrated video (10 shots × ~13 s, one
recurring character). Against the same script and seed:
Voice: independent blind review scored it **"unquestionably a real recorded
human"**, with no drift in timbre or accent across all ten shots. The
JoyAI-Echo parent's best result on the same harness was 6–7/10 ("performed"
emotion, metallic sibilance).
Lip sync: 9.5–10/10 per shot, phoneme-accurate, no progressive drift.
Identity: a single character held across ten completely different locations
with a byte-identical identity sentence per shot and memory_max_size=7.
Picture: read as a real camera recording rather than AI, with correctly
formed hands and realistic eye reflection.
Two-character single-frame shots also hold, given stable ID_A/ID_B labels
and contrast on several axes at once (age band, hair, build, clothing colour).
Talking heads in plain core-node workflows (measured 2026-07-25)
Scope note for the numbers above: they were earned under the JoyAI-Echo pack's
production recipe — slow, deliberate lines (~1.1 words/sec), head trims, and
memory-bank conditioning.
At conversational speech rates (~2.5–3 words/sec) in a stock-sampler
I2V talking-head path (LTXVAddGuide + LTXVReferenceAudio + the TalkVid
ID-LoRA, 25 fps), this merge articulates the mouth measurably less than stock
ltx-2.3-22b-distilled-1.1: mouth-region motion 4.8–5.2 vs 9.3 on the
same seed, prompt, and settings. Rewiring experiments (reverting the
cross-modal modulation to stock; moving the full cross-modal apparatus
Echo-ward) did not close the gap — the ceiling lives in the echoVid
video-attention weights themselves, and this merge is already the best
arrangement of its parts (every alternative tested scored equal or worse).
Recommendation: for reference-voice talking heads in plain core-node
workflows (e.g. the Multishot Lite workflow), use stock
ltx-2.3-22b-distilled-1.1 — it articulates roughly twice as hard. Use this
merge where it wins: identity-held, multi-shot narrative under the pack's
recipe, where the pacing above keeps sync scoring 9.5+.
Update (later 2026-07-25): the TALKINGHEAD variant — articulation fix, baked in
For reference-voice talking heads at conversational speech rates, use the
TALKINGHEAD builds instead of the base merge — no extra nodes, no LoRA to
wire:
They are the base merge with the c72 video-only distilled LoRA (below)
baked in at 0.5 — weight-equivalence verified per-module against fp32 compute.
Measured across two independent seed draws at default settings: shot-start
articulation 4.8 → 6.9–7.5 (stock distilled-1.1 = 9.3) with envelope
timing preserved (0.23–0.27 vs the 0.25 no-LoRA baseline; the earlier
full-rank bake traded timing down to ~0.09 for a little more raw motion —
that is why this bake uses the truncated strip). Face untouched.
Which file do I want?
use
file
reference-voice talking heads at conversational pace
e50 builds (newest; see below)
same, via the earlier baked-LoRA route
talkinghead builds
chained multi-shot, the JoyAI-Echo pack recipe
base merge (unchanged)
maximum articulation, no Echo look needed
stock ltx-2.3-22b-distilled-1.1
The base merge stays the pack's model: at the pack's pacing (~1.1 words/sec,
head trims, memory bank) its sync already scores 9.5+, and the baked boost is
measured to overdrive chained shots that open on a moving guide frame
(second-shot envelope correlation ~0.40 → ~0.20). That tradeoff is why this is
a separate file and not an update to the base merge.
Update (2026-07-25, latest): the e50 two-dose rebuild
ltx23_echoVid-ltxAud_surgical_e50_* attacks the same articulation deficit as
the TALKINGHEAD builds, but from the weights rather than by baking a LoRA on
top.
Scope: this is a fix for the reference-audio path, not a general lip-sync
repair. The deficit appears when a character is driven from a supplied voice
clip (LTXVAddGuide + LTXVReferenceAudio + the TalkVid ID-LoRA) at
conversational speech rates (~2.5–3 words/sec). Under the pack's own production
recipe — slower, deliberate delivery plus memory-bank conditioning — the base
merge already scores 9.5+ on sync and is not outclassed by e50. If you are not
driving from reference audio, you do not need this build.
The video side is reconstructed from two independent doses:
video = dev + 1.0 x (distilled-1.1 - dev) + 0.5 x (JoyAI-Echo - dev)
tables/adalns = dev + 0.3 x (distilled-1.1 - dev) + 0.5 x (JoyAI-Echo - dev)
audio + cross-modal = stock distilled-1.1 (unchanged, so TalkVid still works)
Why this shape. JoyAI-Echo is a dev-lineage finetune — its training
fingerprint matches ltx-2.3-22b-dev, not distilled-1.1 — so on the 8-step
distilled sigma ladder its video stack is effectively under-distilled. That
surfaces as under-articulated mouths and smeared fine motion when the mouth is
being driven hard by reference audio at speed; at slower delivery it is not the
limiting factor. Adding the full distillation direction
back restores articulation; holding Echo's own finetune at 0.5 recovers
stock's crisper viseme formation, which is most visible on rounded vowels (on
the test line, "cloned" forms a real /oʊ/ with visible tongue movement instead
of a flattened closure). The modulation tables take a lighter distill dose
because at full strength they produce specular, over-shiny skin.
Both doses were chosen by rendering and eye, not by theory: a full-strength
distill correction reads rubbery/exaggerated, stock alone reads stiff, and 0.5
was picked over 0.6 and 0.7 on viseme shape.
Known trade. Echo's finetune is what suppresses the occasional robotic
voice and drives the pack's audio memory bank. At 0.5 you keep half of it. For
long multishot runs through the pack, the base merge remains the safe choice.
e50 vs TALKINGHEAD. Two different routes to the same problem; they have not
been benchmarked head-to-head at equal settings. e50 is the newer and the one
in current use here. If you already have a tuned TALKINGHEAD setup, there is no
measured reason to switch.
Shot length: this is a nodes issue, not a weights issue. Through the
JoyAI-Echo pack, lip sync decays and visibly breaks about 9.6 s into every
shot — the wrapper's video RoPE clock is hardcoded to 24 fps while audio RoPE
is in true seconds, so a 25 fps render drifts ~4 %/s. It reads as a model
limitation and is not one: no LTX-2.3 or Echo checkpoint caps shot length
(temporal RoPE range is 20 s, identical across Echo, dev and distilled-1.1),
and no merge can fix it because the bug is in coordinate math. The
multishot patch
stamps the real render fps onto the generator (Bug fix #0); with it, 60–105 s
multishot masters hold sync.
Quantised e50 builds (Q8_0 / Q5_0 / Q4_0) are in the
GGUF repo.
The underlying fix, as a LoRA (for strength-tuners)
Two strips are available; the small one is the better one:
Recommended: ltx-2.3-22b-distilled-lora-c72_VIDEOONLY.safetensors
(390MB, rank-72 truncation). At 0.5 under the TalkVid ID-LoRA it keeps the
articulation boost AND times better at default identity guidance —
shot-start envelope correlation 0.32 vs 0.10 for the full-rank strip,
replicated across independent seeds. Rank truncation appears to drop
motion-texture components that decorrelate from the audio envelope.
ltx-2.3-22b-distilled-lora-384-1.1_VIDEOONLY.safetensors (3.76GB,
full-rank). Slightly more raw motion; measurably worse envelope timing at
default guidance. Kept for completeness and strength-sweepers.
The deficit above decomposes into two independent causes, and the amplitude
half is fixed by an adapter shipped in this repo:
ltx-2.3-22b-distilled-lora-384-1.1_VIDEOONLY.safetensors.
Why it works: this merge's video side is under-distilled — dev-flavoured
attention driven by an 8-step distilled schedule undersamples, and mouth
articulation is the first casualty. The LoRA is the dev→distilled delta
(rank 384) with all audio and cross-modal keys stripped (1174 video keys
kept), so it restores the missing distillation direction without touching the
pathway the TalkVid ID-LoRA adapts.
Measured, same seed/prompt/settings (mouth-region motion / audio-envelope
correlation): shot-start articulation goes from 4.8 → 7.6–8.0 (stock
distilled-1.1 = 9.3) at strength 0.5, chained under the TalkVid ID-LoRA,
with no face restyling.
Known costs, measured: in chained multi-shot use the boost overdrives
shots that open on a moving guide frame (second-shot envelope correlation
drops from ~0.40 to ~0.20) — for multi-shot work, sweep strength down
(0.2–0.35) or apply it per-shot. Do NOT use the full (non-VIDEOONLY) rank-384
distilled LoRA under the TalkVid ID-LoRA: its audio/cross-modal keys fight the
ID-LoRA and envelope timing collapses to zero.
Separate timing note:identity_guidance_scale amplifies adherence to the
voice reference. When the reference clip's words differ from the spoken
line (always true for the first shot), that amplification fights the script's
timing — measured 0.25 → 0.43–0.54 envelope correlation by setting the FIRST
shot's scale to 0–1.5. But in chained workflows the second shot references the
first shot's own audio, and an unguided first shot produces a worse chain
reference (second-shot timing drops). Single-shot talking heads: lower the
scale. Chained shots: keep 3.0 on the chain, tune only shot 1, and judge the
whole clip.
Usage notes
Requires the memory-multishot path to get the identity benefit — a
plain single-shot graph will not exercise the memory bank.
sequential_offload is effectively mandatory: 43 GB of bf16 weights will
not fit a 32 GB card unquantised. The fp8 build at 23.4 GB is the practical
choice for 24 GB hardware.
Keep negative-steering gentle. negative_scale_video above ~1.0 rotates the
conditioning hard enough to scramble per-shot scene content — every shot
collapses onto shot 1's composition while the audio advances correctly.
0.5–0.8 is the working band.
Prompts should state the accent affirmatively ("in a casual American
