You can deploy this model and start earning money today!

opria123/
strum

Views

No views yet

STRUM — Spectral Transcription & Rhythm Understanding Model

End-to-end pipeline that turns a song (.wav / .mp3 / .ogg) into a fully playable Clone Hero / YARG chart package: drums, guitar, bass, vocals (with lyrics), and keys.

Source: https://github.com/opria123/strum

What's in this repo

Folder	What it is	Used by
`drums/drums_v14/`	TwoStageDrumsCRNN onset detector (mel input, 22050 Hz)	`batch_infer_hybrid.py` Stage 1
`drums/drums_mc_onset/`	Multi-class onset head fine-tuned on V14 backbone	Stage-1 alt head
`drums/drums_phase3/`	Phase-3 multi-class rescue model	Late-stage rescue / reclassify
`drums/drums_cymbal_onset/`	Cymbal-specialist onset head	Cymbal-specific rescue
`drums/tom_refinement_demucs/`	Tom vs. cymbal CNN running on Demucs drum stem	Tom/cymbal disambiguation
`drums_classifier_ensemble/`	6-model OnsetClassifier ensemble (V2, V4, V6, V12c, V15, V16) + V17	Per-onset 8-lane classification
`guitar/guitar_v2_onset/`	Guitar onset CRNN (Event F1 0.81)	Hybrid guitar pipeline
`guitar/fret_mapper_v4.pt`	Pitch → 5-fret mapper (replaces librosa rule mapper)	Hybrid guitar pipeline
`section_classifier/`	Verse/chorus/bridge section labeler	Chart sections

Performance

Held-out test set (from 3,299 human-authored Pro Drum charts):

Component	Metric	Score
Drums onset detection (V14)	Frame F1	93.9%
Drums lane classification (6-ensemble)	Per-onset F1	85.2%

End-to-end vs ground-truth Clone Hero / YARG charts on an in-envelope benchmark of 29 songs sampled from a 3,299-song held-out pool. Songs were pre-screened with a single audio-feature gate (median Demucs htdemucs_6s drum-stem RMS ≥ 0.018, 1 s windows at 22050 Hz mono). Eval is Expert difficulty, ±100 ms tolerance, with a per-song global offset search (±200 ms / 10 ms steps).

Instrument	F1	Precision	Recall
Drums	83.8%	82.4%	85.4%
Guitar	65.1%	74.5%	57.8%
Bass	69.4%	65.8%	73.4%
Vocals	53.9%	63.2%	47.0%

See the source repo's benchmark_results.json for per-song breakdown and scripts/eval_benchmark.py for the harness.

Usage

The checkpoints are loaded by the STRUM pipeline scripts. Clone the repo and download the checkpoints into checkpoints/ preserving the layout:

bash

git clone https://github.com/opria123/strum
cd strum
python -m venv .venv && source .venv/bin/activate
pip install -e .

# Pull weights from the Hub
huggingface-cli download opria123/strum --local-dir checkpoints/ \
    --local-dir-use-symlinks False

# Run the full pipeline on a folder of audio files
python scripts/batch_pipeline.py /path/to/songs /path/to/charts

The pipeline expects this layout (mirrors the drums/ and guitar/ subfolders here, just under checkpoints/):

checkpoints/
├── drums_v14/best.pt
├── drums_mc_onset/best.pt
├── drums_phase3/best.pt
├── drums_cymbal_onset/best_union_f1.pt
├── tom_refinement_demucs/best.pt
├── onset_classifier/best_f1.pt
├── onset_classifier_v4/best_f1.pt
├── onset_classifier_v6/best_f1.pt
├── onset_classifier_v12_clean/best_f1.pt
├── onset_classifier_v12c_community/best_f1.pt
├── onset_classifier_v15/best_f1.pt
├── onset_classifier_v16/best_f1.pt
├── onset_classifier_v17/best_f1.pt
├── guitar_v2/guitar_v2_onset/best.pt
├── fret_mapper_v4.pt
└── section_classifier/best.pt

A small reorganisation script scripts/sync_from_hf.sh in the source repo handles the drums/ → flat-checkpoints/ mapping.

License

MIT. See the source repository for full attribution of the underlying training data (Clone Hero / YARG community charters) and dependencies (Demucs v4, librosa, OpenAI Whisper, Spotify Basic Pitch).