Whispbook FunctionGemma Speaker Attribution MLX LoRA HPO QVO Step 500
This repository contains an MLX-LM LoRA adapter for FunctionGemma speaker attribution.
It was selected from the local HPO batch because it had the best balanced score on the 100-row known-speaker / not_in_candidates probe while avoiding false fallback on known-speaker examples.
Base Model
mlx-community/functiongemma-270m-it-4bit
Selected Trial
- Trial:
r0075-qvo-r8-d010-lr5e6-b6
- Checkpoint:
0000500_adapters.safetensors
- Target modules:
self_attn.q_proj, self_attn.v_proj, self_attn.o_proj
- LoRA rank:
8
- LoRA dropout:
0.10
- Learning rate:
5e-6
- Batch size:
6
not_in_candidates training ratio: 0.075
HPO Probe Result
On the 100-row HPO probe:
| metric | value |
|---|
| total accuracy | 12 / 100 |
| known-speaker accuracy | 6 / 58 |
not_in_candidates accuracy | 6 / 42 |
| false fallback on known speakers | 0 |
| forced known-speaker pick on fallback examples | 36 |
This checkpoint is conservative: it prioritizes not incorrectly rejecting known-speaker examples. It should be treated as an HPO milestone, not a final production model.
Follow-up HPO Note
The next local trial, gen2-r008-qvo-r8-d010-lr3e6-b6-s700-save25, lowered the learning rate to 3e-6, used a 0.08 not_in_candidates ratio, and saved dense checkpoints from 400 to 700 steps. It did not beat this adapter.
Best Gen2 probe checkpoint:
| metric | value |
|---|
| checkpoint | 0000625_adapters.safetensors |
| HPO score | -0.1507 |
| total accuracy | 11 / 100 |
| known-speaker accuracy | 2 / 58 |
not_in_candidates accuracy | 9 / 42 |
| false fallback on known speakers | 2 |
| forced known-speaker pick on fallback examples | 33 |
The 700-step checkpoint learned more fallback behavior, but collapsed known-speaker accuracy to 0 / 58 and produced 10 false fallback predictions. The current selected adapter remains the safer checkpoint.
Larger Local Check
On a 500-row local generation check using the same test split shape:
| metric | value |
|---|
| total accuracy | 57 / 500 |
| known-speaker accuracy | 51 / 458 |
not_in_candidates accuracy | 6 / 42 |
| false fallback on known speakers | 4 |
| forced known-speaker pick on fallback examples | 36 |
The larger sample confirms the model is conservative around not_in_candidates: it rarely rejects known speakers incorrectly, but still misses most true out-of-candidate cases.
Files
adapters.safetensors: selected MLX LoRA adapter weights.adapter_config.json: MLX-LM adapter metadata.trial_config.json: HPO trial configuration.eval_0000500_summary.json: generation probe summary.eval_full500_0000500_summary.json: larger local generation-check summary.hpo_leaderboard.json: leaderboard snapshot for the completed local seed, gen1, and gen2 trials.functiongemma_lora_config.yaml: MLX-LM training configuration used for this trial.hpo/gen2-r008-qvo-r8-d010-lr3e6-b6-s700-save25/: selected Gen2 trial config and checkpoint probe summaries.
