A LoRA adapter for openai/whisper-small fine-tuned for Swahili–English code-switched speech recognition (including Sheng slang), part of a bilingual voice-agent project for East Africa. The adapter keeps Whisper's multilingual weights frozen and adds low-rank adapters to the attention q_proj/v_proj layers, then is merged back into the base weights for deployment.

⚠️ Status: experimental quick-test checkpoint. This was trained with the notebook's QUICK_TEST setting (50 steps), not the full 5,000-step production run. It is intended to validate the training pipeline and is not a converged, production-quality model.

Model Details

Model Description

Developed by: Karsamanja (project: Swahili–English Code-Switching Voice Agent)
Funded by [optional]: [More Information Needed]
Shared by [optional]: Karsamanja
Model type: Speech recognition (automatic speech recognition) — PEFT/LoRA adapter for Whisper encoder-decoder
Language(s) (NLP): Swahili (sw), English (en), and code-switched Swahili–English (incl. Sheng)
License: CC-BY-NC-4.0 — applied because the fine-tuning dataset (liva-ai/swahili-english-asr) is CC-BY-NC. The base model (openai/whisper-small, MIT) does not impose this; retraining on CC0/own data would lift the non-commercial restriction.
Finetuned from model: openai/whisper-small (244M params)

Model Sources

Repository: https://huggingface.co/Karsamanja/swahili-english-whisper-cs-lora
Paper [optional]: Whisper (Radford et al. 2022); LoRA (Hu et al. 2021); PEFT
Demo [optional]: [More Information Needed]

Uses

Direct Use

Transcription of audio containing Swahili, English, or code-switched Swahili–English speech (e.g., conversational recordings, phone calls). Because the adapter was fine-tuned on code-switched data, it is intended to handle language mixing within a single utterance better than forcing a single language.

python

from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel
import torch

base_model_id = "openai/whisper-small"
adapter_id = "Karsamanja/swahili-english-whisper-cs-lora"

processor = WhisperProcessor.from_pretrained(adapter_id)
model = WhisperForConditionalGeneration.from_pretrained(base_model_id)
model = PeftModel.from_pretrained(model, adapter_id)

# Optional: merge for deployment (no peft dependency at inference)
# model = model.merge_and_unload()

audio_array = ...  # 16 kHz mono float32 numpy array
inputs = processor.feature_extractor(audio_array, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
    generated = model.generate(
        inputs.input_features,
        language=None,      # auto-detect: do not force a language (code-switch mode)
        task="transcribe",
        temperature=0.0,
    )
print(processor.batch_decode(generated, skip_special_tokens=True)[0])

Downstream Use

Designed to be dropped into a voice pipeline (e.g., faster-whisper/CTranslate2 after conversion, or the merged model) as the ASR component of a bilingual voice agent, where the language mix is detected per utterance and the LLM/TTS mirror the detected mix.

Out-of-Scope Use

Forcing a single language on code-switched audio (degrades accuracy on mixed speech).
Languages other than Swahili/English.
This checkpoint should not be used in production — it is a 50-step pipeline-validation run.

Bias, Risks, and Limitations

Not converged: 50-step quick test; predictions are often degenerate/short. Retrain with QUICK_TEST = False (5,000 steps) before any real use.
Training data bias: fine-tuned on ~50h of conversational Swahili–English from liva-ai/swahili-english-asr; dialectal/Sheng coverage is limited and other East-African language varieties (Kiswahili sanifu vs. colloquial) may be underrepresented.
Non-commercial license: training data is CC-BY-NC; commercial deployment requires retraining on CC0/own data.
Evaluation caveat: reported WER numbers below are not comparable to standard ASR benchmarks — see Evaluation section.

Recommendations

Users should treat the reported metrics as pipeline artifacts, not quality claims, and should re-evaluate with a clean single-utterance benchmark before drawing conclusions about model quality.

How to Get Started with the Model

Use the code below to get started with the model.

python

# Adapter-style loading (see Direct Use for the full snippet)
from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
model = PeftModel.from_pretrained(model, "Karsamanja/swahili-english-whisper-cs-lora")
processor = WhisperProcessor.from_pretrained("Karsamanja/swahili-english-whisper-cs-lora")

Training Details

Training Data

Primary: liva-ai/swahili-english-asr (~50h, conversational Swahili–English, CC-BY-NC, gated).
Note: Shelton1013/SwitchLingua_audio was loaded as an intended supplementary corpus in the training notebook but was not actually consumed by the quick-test training run.

Training Procedure

PEFT LoRA fine-tuning on frozen whisper-small. Encoder–decoder compatibility required a small PEFT wrapper that strips the leaked input_ids kwarg. Audio resampled to 16 kHz mono; text tokenized with max_length=448.

Preprocessing

Audio cast to 16 kHz mono float32 via datasets Audio(sampling_rate=16000).
Labels tokenized to input_ids (max 448); collator pads features and masks labels with -100.

Training Hyperparameters

LoRA: r=32, α=64, dropout=0.05, bias=none, target_modules=[q_proj, v_proj]
Learning rate: 1e-4 · Warmup: 500 steps · Weight decay: 0.01
Max steps: 50 (quick test) · Save/Eval steps: 25
Batch size (per device): 8 TPU / 4 GPU · Gradient accumulation: 4 TPU / 2 GPU
Gradient checkpointing: on (non-TPU) · Precision: fp16 (GPU) / bf16 (TPU)
Generation max length: 225 · Training regime: fp16 mixed precision (GPU)

Speeds, Sizes, Times

Merged model: 241,734,912 params (~0.97 GB fp32); whisper-small LoRA adds only ~1.2M trainable params (r=32 on q/v proj).
Quick test: ~minutes on Colab T4/TPU. Full 5,000-step run: 2–6 hrs (planned).

Evaluation

Testing Data, Factors & Metrics

Testing Data

15 samples from liva-ai/swahili-english-asr (streaming, first 15), evaluated on CPU at batch size 2, language auto-detect. Indicative only — tiny, non-representative sample.

Metrics

WER, CER, approximate per-language token WER (heuristic language detection), and PIER@3 (switch-window error rate).

Results

Metric	Fine-tuned (merged LoRA)	whisper-small baseline	whisper-tiny baseline (older)
Overall WER	0.996	0.999	0.9996
Overall CER	0.962	0.968	0.995
Swahili WER (approx.)	0.998	1.000	1.000
English WER (approx.)	1.000	0.993	1.000
PIER@3	0.998	0.996	1.000

Summary

These numbers are methodology artifacts, NOT quality measurements. The dataset references are full multi-turn speaker transcripts containing markers the model can never produce — e.g. Speaker 2 [00:00:00.000 - 00:00:04.930]: …, [laughing], [unintelligible] — while the model transcribes a single audio segment (≤30s window). Consequently every model saturates at WER ≈ 1.0 (whisper-tiny also scored 0.9996; one smoke run even produced WER > 1.0, which is structurally impossible with sane reference/prediction pairs). The fine-tuned model slightly edges the same-base baseline (0.996 vs 0.999 WER), but no conclusion about quality should be drawn. A fair evaluation requires a clean single-utterance benchmark (e.g., Common Voice sw, FLEURS sw, or turn-segmented transcripts).

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Hardware Type: Google Colab (GPU T4 runtime; notebook device detection may fall back to CPU when torch_xla is unavailable)
Hours used: < 1 (quick test)
Cloud Provider: Google Colab
Compute Region: [More Information Needed]
Carbon Emitted: [More Information Needed]

Technical Specifications

Model Architecture and Objective

Standard Whisper encoder–decoder (whisper-small) with LoRA adapters on attention projections, fine-tuned with the standard cross-entropy ASR objective.

Compute Infrastructure

Hardware

Google Colab (T4 GPU / TPU v2-8 per notebook).

Software

PyTorch, Transformers 5.x, PEFT (peft ≥ 0.8), datasets, evaluate/jiwer.

Citation

BibTeX:

bibtex

@misc{karsamanja2026swahilienglishwhispercs,
  title={Swahili--English Code-Switching Whisper (LoRA)},
  author={Karsamanja},
  year={2026},
  howpublished={\url{https://huggingface.co/Karsamanja/swahili-english-whisper-cs-lora}},
  note={Quick-test LoRA adapter fine-tuned from openai/whisper-small}
}

APA: Karsamanja. (2026). Swahili–English Code-Switching Whisper (LoRA) [Model]. Hugging Face. https://huggingface.co/Karsamanja/swahili-english-whisper-cs-lora

Model Card Contact

Karsamanja via Hugging Face