Evaluated on Paulescu/OHF-Voice-audio-20260504 test split, 397 stratified samples (10 per function across 41 classes), system prompt "Perform ASR.", temperature 0.0.
Metric
v1 (950 samples, 10K steps)
v2 (50K samples, 4K steps)
Cookbook (full FT, A100)
Format compliance
99.0%
100.0%
99.7%
Function-name accuracy
82.0%
92.2%
98.8%
Argument accuracy
61.0%
70.0%
~97%
Key takeaway: going from 950 → 49,909 training samples gained +10% function-name accuracy and +9% argument accuracy, using fewer training steps. Data volume matters more than training duration for LoRA fine-tuning.
Per-function highlights
Function
Name acc
Arg acc
HassStartTimer
10/10
10/10
HassSetPosition
10/10
10/10
HassGetCurrentDate
6/6
6/6
HassGetCurrentTime
7/7
7/7
HassCancelAllTimers
10/10
9/10
HassSetVolume
10/10
9/10
HassRespond
0/5
0/5 ← known weak spot
HassSetVolumeRelative
10/10
2/10 ← relative values hard
Key Finding: Modality Positioning
LFM 2.5 Audio's __call__ interface appends audio embeddings after all text tokens. In a causal model the decoder cannot attend to audio when predicting the assistant response — the model will produce plain transcription regardless of fine-tuning.
The fix: use model._prefill with an explicit modalities array that places AUDIO_IN tokens inside the user turn, before the assistant tokens. This matches the inference-time ChatState.add_audio() layout.
