The final model uses a frozen HuBERT encoder (facebook/hubert-base-ls960) to extract frame-level speech representations, followed by an LSTM classifier for temporal command detection.

Commands

Class	Spoken command
`command_1`	get ready
`command_2`	load
`command_3`	left
`command_4`	right
`command_5`	shoot
`no_command`	background / non-command audio

Dataset

The published dataset includes:

original command recordings
real non-command recordings
robot movement noise
DEMAND home-environment background noise used in the experiments
generated 3-second synthetic clips
frame-level labels used for training

The final synthetic dataset contains 10,400 clips:

8,000 training
1,200 validation
1,200 test

Audio is processed at 16 kHz with a fixed duration of 3 seconds.

Model

The model pipeline is:

text

3-second audio
      ↓
HuBERT Base
(frozen encoder)
      ↓
149 × 768 frame representations
      ↓
Layer Normalization
      ↓
LSTM (128 units)
      ↓
Dense layer
      ↓
6-class frame logits

HuBERT is not fine-tuned in this project. It is used as a frozen feature encoder while the downstream LSTM classifier is trained.

Final Trigger Configuration

The final deployment configuration was selected after temperature and class-specific threshold tuning.

Temperature: 2.2
False-trigger clips on the test set: 0
Event recall: 85.33%
Wrong-action clip rate: 1.42%
Exact clip accuracy: 82.38%

Thresholds and minimum consecutive-frame requirements are different for each command and are documented in the training and inference notebooks.

Try the Model

Install the dependencies:

pip install -r requirements.txt

Open:

notebooks/04_single_audio_inference.ipynb

Then change:

AUDIO_PATH = Path("your_3_second_audio.wav")

The notebook will:

load the WAV file
convert it to mono
resample it to 16 kHz if needed
pad or trim it to 3 seconds
run HuBERT + LSTM inference
apply temperature scaling and command-specific trigger thresholds
print the detected command

The pretrained HuBERT files are downloaded automatically from Hugging Face the first time the model is loaded.

Notebooks

The notebooks follow the original experiment pipeline:

text

01_data_synthesis_and_split.ipynb
        ↓
02_hubert_frame_label_preprocessing.ipynb
        ↓
03_hubert_lstm_training_evaluation_and_tuning.ipynb
        ↓
04_single_audio_inference.ipynb

01 — Dataset Synthesis and Split

Creates reproducible source splits and generates the synthetic 3-second dataset.

02 — HuBERT Frame-Label Preprocessing

Creates frame-level labels and weights aligned with HuBERT's 149 output frames.

03 — HuBERT + LSTM Training, Evaluation, and Tuning

Contains the final V4 training run, test evaluation, temperature scaling, threshold tuning, and saved results.

04 — Single Audio Inference

Loads the trained model and runs inference on one WAV recording.

Repository Structure

text

robot-arm-voice-command-classifier/
├── README.md
├── requirements.txt
├── LICENSE
├── data/
│   ├── source_commands/
│   ├── source_backgrounds/
│   ├── source_splits/
│   ├── synthetic_trigger_data/
│   └── processed/
├── notebooks/
│   ├── 01_data_synthesis_and_split.ipynb
│   ├── 02_hubert_frame_label_preprocessing.ipynb
│   ├── 03_hubert_lstm_training_evaluation_and_tuning.ipynb
│   └── 04_single_audio_inference.ipynb
├── model/
│   └── best_model.weights.h5
└── results/
    ├── training/
    ├── test_forward_pass/
    ├── temperature_tuning/
    ├── threshold_tuning/
    └── evaluation/

Reproducibility Note

The training and evaluation notebooks preserve the outputs from the original experiment. The final training run was computationally expensive, so the notebooks were cleaned for publication without rerunning the training process.

Because the saved outputs come from the original run, some printed log paths may still show the original local project directories. The executable notebook code uses the public repository structure shown above.

Intended Use

This project is intended as a portfolio and research demo for small-vocabulary speech-command classification. It is not designed as a general-purpose speech-recognition system.

abdel-lall-31/robot-arm-voice-command-classifier

Robot Arm Voice Command Classifier

Commands

Dataset

Model

Final Trigger Configuration

Try the Model

Notebooks

01 — Dataset Synthesis and Split

02 — HuBERT Frame-Label Preprocessing

03 — HuBERT + LSTM Training, Evaluation, and Tuning

04 — Single Audio Inference

Repository Structure

Reproducibility Note

Intended Use

abdel-lall-31/
robot-arm-voice-command-classifier