This is the first community audio-visual (AV) LoRA for LTX-Video 2.3, trained using the joint audio-video cross-attention architecture of the LTX-2.3 22B model. The LoRA enables talking head video generation with synchronized lip sync and internalized voice characteristics from a reference character.
This release is a character-specific implementation and reference pipeline. The weights demonstrate a working AV LoRA trained on a custom dataset. The methodology, dataset structure, caption format, and training config are fully documented and reusable for training your own character-specific AV LoRA.
What It Does
Generates talking head videos with synchronized lip sync from a reference image
Internalizes voice characteristics without requiring external audio input at inference time
Preserves character identity across unseen reference images and backgrounds
Demo Results (v1)
Lip sync: accurate and consistent
Identity preservation: locks in at step 1250, improves linearly to step 2000
Voice characteristics: internalized from training data
Load LTX-2.3-22b-AV-LoRA-talking-head-v1.safetensors via the Power Lora Loader node in ComfyUI.
Set LoRA strength to 1.0.
Recommended Inference Settings
Parameter
Value
Resolution
1280x736
FPS
24
Video length
Any (10+ seconds recommended)
LoRA strength
1.0
Trigger word
OHWXPERSON
CFG scale
1.0
Note: 1280x736 @ 24fps is recommended for image-to-video inference. For image + audio to video inference, use 1280x704 @ 25fps to match the training distribution.
Prompt Format
Include the trigger word OHWXPERSON and end the prompt with the speech transcript:
OHWXPERSON, [visual description]. The person is talking, and he says: "[transcript]"
Training Your Own AV LoRA
This section documents the full pipeline so you can train a character-specific AV LoRA for your own subject.
Pipeline Overview
Reference Images
|
v
Flux.1 Kontext / Flux.2 Klein -- Image generation
|
v
Fish Audio S2 Pro -- Voice cloning + TTS
|
v
LTX-Video 2.3 -- Talking head video generation
|
v
LTX-2 trainer -- AV LoRA training
|
v
Trained AV LoRA weights
Step 1 -- Generate Reference Images
Use Flux Kontext in ComfyUI to generate consistent reference images of your character across varied poses, angles, lighting conditions, and expressions.
Use Fish Audio S2 Pro (model) with a 10-15 second reference audio clip of your target voice. Supports [pause], [short pause], and [emphasis] tags for pacing control.
Generate TTS audio for each clip's script using the cloned voice.
Step 3 -- Generate Training Clips
Use LTX-2.3 in ComfyUI to generate talking head clips from your reference images.
Variety: front facing, 3/4 angles, side profile, different backgrounds, multiple emotions
Prompt format for each clip:
[scene description]. Mouth partially open during speech with only the front teeth partially visible, lips moving naturally without fully exposing all teeth. Smooth continuous motion, cinematic, realistic, sharp focus on subject. The person is talking, and he says: "[transcript]"
Background complexity directly impacts lip sync quality. Simple and dark backgrounds produce the best results. Complex backgrounds with many competing elements reduce lip sync accuracy.
Step 4 -- Prepare the Dataset
Structure your dataset folder as follows:
ohwxperson_dataset_v1/
clip_001.mp4 # video with embedded audio from LTX-2.3
clip_002.mp4
...
CAPTIONS.json
Caption format in CAPTIONS.json:
json
1{2"captions":[3{4"file":"clip_001.mp4",5"caption":"[VISUAL] OHWXPERSON, [visual description of scene, pose, clothing, background]. [SPEECH] OHWXPERSON speaks in a [voice description]: \"[exact transcript]\""6}7]8}
A reference CAPTIONS.json from this project is included in this repository.
