Model Card
A multimodal text + audio model currently in training. This card documents the in-progress run; metrics and details will be updated when training completes.
Status: 🟢 Training in progress — ~38% complete (Epoch 2 of 3, step ~12,510 of ~32,500). No instability observed.
Model Details
- Modalities: Joint text and audio
- Objective: Combined text + audio loss
- Status: Mid-training checkpoint (not final)
Training Procedure
Configuration
| Setting | Value |
|---|
| Epochs | 3 |
| Examples per epoch | 86,671 |
| Micro-batch size | 1 |
| Gradient accumulation | 8 |
| Effective batch size | 8 |
| Steps per epoch | ~10,834 |
| Total planned steps | ~32,500 |
| Learning rate | 3e-5 |
| LR schedule | WarmupLR (linear warmup to 3e-5 by ~step 110, held flat, no decay) |
| Checkpoint interval | Every 500 steps |
| Throughput | ~10 sec/step (+ ~5–6 min checkpoint stall per 500 steps) |
Run Timeline
- Started: 2026-05-23 20:37
- Last logged: 2026-06-01 04:03 (~8.5 days elapsed, still running)
- Estimated remaining: ~20,000 steps, on the order of a couple more days
- Checkpoints retained:
step_12000, step_12500 (older checkpoints rotated out)
Loss Curve
Loss averaged per 1,000 steps. The large initial drop occurs during warmup; thereafter both losses grind down steadily, with text falling faster than audio. Audio loss is the harder signal and is plateauing around ~1.4–1.5.
| Step range | Total | Text | Audio |
|---|
| 0–999 | 4.65 | 2.12 | 2.53 |
| 2k–3k | 2.77 | 1.16 | 1.61 |
| 5k–6k | 2.65 | 1.13 | 1.51 |
| 8k–9k | 2.52 | 1.02 | 1.50 |
| 9k–10k | 2.34 | 0.93 | 1.41 |
| 11k–12k | 2.47 | 1.04 | 1.42 |
| 12k+ | 2.14 | 0.84 | 1.30 |
Notes on loss: Per-step loss is noisy (individual steps swing from ~0.17 to ~4.4), which is expected with micro-batch=1 and grad-accum to an effective batch of 8. The binned per-1,000-step averages are the meaningful view of the trend.
Stability
No NaN, no OOM, no exceptions, and no tracebacks across the full run. Training is progressing normally with loss still trending down.
Inference Test (step_12000)
A live, oracle-enabled full-duplex inference test was run on the
step_12000 checkpoint. Full setup, server patches, and reproduction steps:
INFERENCE.md.
Stack: kame.server_oracle (browser UI, port 8001) · ASR = Deepgram nova-3 multi (Hinglish code-switching) · oracle LLM = self-hosted Qwen3 (OpenAI-compatible endpoint) with a Devanagari-aligned prompt.
What works
- Real-time full-duplex speech at ~0.5 s latency.
- Deepgram transcribes spoken Hinglish, code-mixed (Devanagari + Roman).
- The oracle fires and produces proper Devanagari Hinglish hints from the live LLM.
What does not (yet)
- At ~38% training the model speaks fluent English call-centre dialogue and does not code-switch to Hinglish — even when the user speaks Hindi and explicitly asks it to.
- It largely ignores the oracle hints.
Root causes identified
- The oracle training data was ~87% corrupted — LLM oracle-generation failures produced token-soup predictions, so the model learned to distrust/ignore the oracle. A cleaned dataset was produced (
train_text_oracle_a0b1_events_cleaned-001-of-001.parquet): genuine Devanagari predictions kept, the rest replaced with each event's real next-utterance hint, hint-less events zeroed — 0 garbage remaining.
- The model's text stream is Roman/Latin by data design (0% Devanagari), so its spoken text is Roman Hinglish / English, never Devanagari.
- Early checkpoint (~38%); Hinglish code-switching is expected to strengthen with continued training on the cleaned oracle.
See
INFERENCE.md for the full server patches, commands, and analysis.
Intended Use
This is an intermediate training artifact. The final model and evaluation results are not yet available. Use mid-training checkpoints only for monitoring or experimentation, not for production.
Limitations
- Training is not complete; performance will continue to change.
- No formal evaluation has been run yet.
- Audio loss is plateauing higher than text loss, reflecting the greater difficulty of the audio signal.
