You can deploy this model and start earning money today!

aufklarer/
Audio2Face-3D-v2.3.1-James-MLX

Name: Audio2Face-3D-v2.3.1-James-MLX
Availability: InStock

mlx

audio2face_3d_v2_3_1_james

nvidia/Audio2Face-3D-v2.3.1-James

finetune

other

Views

No views yet

Audio2Face-3D v2.3.1 James — MLX

NVIDIA Audio2Face-3D v2.3.1 (James) exported for the MLX Swift runtime in speech-swift. Speech audio in, timestamped facial-animation coefficients out — fully on-device on Apple Silicon.

The bundle contains the learned tensors from the official ONNX checkpoint in safetensors form plus the NVIDIA runtime metadata. The hand-written MLX graph in speech-swift runs the real model forward pass; outputs are parity-checked against the official ONNX model.

Output

Each frame carries 169 coefficients: 140 skin blendshapes, 10 tongue blendshapes, 15 jaw controls, and 4 eye controls, with a 10-value explicit + 16-value implicit emotion input (16 kHz mono audio, 8320-sample windows).

Files

File	Purpose
`audio2face3d.safetensors`	Learned tensors for `MLX.loadArrays`
`network_info.json`	NVIDIA runtime geometry/audio metadata
`model_config.json`	Runtime configuration (input strength)
`default_emotion.f32`	Default emotion vector
`graph_metadata.json`	Source ONNX graph provenance

Usage (Swift)

swift

import Audio2Face3D

let model = try await Audio2Face3DModel.fromPretrained(
    modelId: "aufklarer/Audio2Face-3D-v2.3.1-James-MLX")
let frames = try model.frames(for: samples, sampleRate: 16_000)

Or from the speech CLI:

speech avatar-motion input.wav --output frames.jsonl

Identities

Audio2Face-3D-v2.3-Mark-MLX — 301 coefficients (272 skin)
Audio2Face-3D-v2.3.1-Claire-MLX — 169 coefficients (140 skin)
Audio2Face-3D-v2.3.1-James-MLX (this repo, runtime default) — 169 coefficients (140 skin)

Coefficients are identity-specific: renderers need the matching rig or a retarget projection for the chosen identity.

License and attribution

The weights are NVIDIA's Audio2Face-3D v2.3.1 James model, redistributed in MLX-ready form under the NVIDIA Open Model License. All model credit goes to NVIDIA.

Ecosystem

soniqo.audio — documentation and use cases
speech-swift — MLX Swift runtime (this model's target)
speech-core — C++ engine
speech-android — Android engine

Other MLX models in this collection

See the MLX Speech Models collection.