k9cli/
video-vec2wav2-tokenizer

Name: video-vec2wav2-tokenizer
Creator: k9cli

Are you the author?

Views

video-vec2wav2-tokenizer

Production-ready pipeline (Python package video_vec2wav2_tokenizer, CLI command video2dataset) that turns a folder of videos into clean AI training datasets for speech recognition (ASR) and text-to-speech (TTS).

videos  ──►  audio (16 kHz mono PCM)  ──►  whisper transcript  ──►  clips  ──►  metadata.csv / dataset.jsonl / tts_metadata.csv / report.json

Video processing — recursive scan of mp4 / mkv / avi / mov / webm, FFmpeg audio extraction to mono · 16 kHz · 16-bit PCM WAV.
Speech recognition — faster-whisper, CPU & CUDA, automatic language detection, word-level timestamps.
Segmentation — cut audio by transcript timestamps into dataset/audio/000001.wav ….
Dataset generation — metadata.csv, dataset.jsonl, tts_metadata.csv.
Feature extraction (optional) — streaming features/train.bin + train.dat with float32 samples, mel spectrograms, duration and sample rate.
Statistics — report.json with totals, durations and language distribution.
Training — train_wav2vec2.py: HuggingFace Wav2Vec2 CTC with resume, multi-GPU, mixed precision and checkpointing.
Performance — multiprocessing, batch processing, tqdm progress bars and memory-efficient streaming that scales past 1 TB of source media.

Installation

Requires Python 3.11+ and the FFmpeg binary on your PATH.

bash

# 1. FFmpeg (one of):
#    macOS:   brew install ffmpeg
#    Ubuntu:  sudo apt install ffmpeg
#    Windows: winget install Gyan.FFmpeg   (or choco install ffmpeg)

# 2. The package
python -m venv .venv && source .venv/bin/activate     # Windows: .venv\Scripts\activate
pip install -r requirements.txt
pip install -e .                # exposes the `video2dataset` command

# Optional: training extras (torch + transformers)
pip install -e ".[train]"

Verify:

bash

video2dataset --version
ffmpeg -version

Quick start

bash

# Drop videos into input/videos/ (any nesting), then run the full pipeline:
video2dataset process ./input/videos --extract-features

Outputs land under dataset/:

dataset/
├── audio/
│   ├── 000001.wav
│   ├── 000002.wav
│   └── ...
├── metadata.csv          # 000001.wav|Hello world
├── dataset.jsonl         # {"audio":"audio/000001.wav","text":"Hello world","duration":2.5}
├── tts_metadata.csv      # 000001.wav|speaker_001|Hello world
└── report.json
features/
├── train.bin
└── train.dat

CLI

bash

video2dataset process    ./videos        # video -> full dataset
video2dataset transcribe ./audio         # wav -> transcripts/<name>.jsonl
video2dataset segment    ./audio         # wav + sidecar transcript -> clips
video2dataset features   ./dataset       # clips -> features/train.{bin,dat}
video2dataset train      ./dataset       # fine-tune Wav2Vec2 CTC

Global flags (override config.yaml): --config, --output-dir, --device {auto,cpu,cuda}, --whisper-model, --language, -v/--verbose.

bash

# Examples
video2dataset process ./input/videos --whisper-model large-v3 --device cuda
video2dataset process ./input/videos --language en --output-dir build/en
video2dataset transcribe ./dataset/_work --device cpu
video2dataset features ./dataset
video2dataset train ./dataset --epochs 30 --batch-size 8 --resume

Multi-GPU training

bash

torchrun --nproc_per_node=4 -m video_vec2wav2_tokenizer.training.train_wav2vec2 \
    --dataset-dir dataset --output-dir dataset/checkpoints

Configuration (`config.yaml`)

yaml

model_name: facebook/wav2vec2-base-960h   # training model
whisper_model: large-v3                    # ASR model
language: null                             # null = auto-detect
device: auto                               # auto | cpu | cuda
sample_rate: 16000
channels: 1
min_segment_length: 1.0
max_segment_length: 20.0
default_speaker: speaker_001
speaker_mode: fixed                        # fixed | per_video | per_file
output_dir: dataset
features_dir: features
num_workers: 4
batch_size: 16

speaker_mode controls TTS speaker identification: a single fixed speaker, one per source video, or one per clip.

Architecture

video-vec2wav2-tokenizer/          # repo root
├── config.yaml
├── requirements.txt
├── pyproject.toml
├── README.md
├── input/videos/                  # drop source videos here
└── video_vec2wav2_tokenizer/      # importable package
    ├── cli/            # argparse command surface
    ├── audio/          # FFmpeg extraction + WAV loading/resampling
    ├── transcription/  # faster-whisper wrapper (lazy import)
    ├── segmentation/   # timestamp normalisation + clip cutting
    ├── features/       # streaming bin/dat feature store + mel spectrograms
    ├── training/       # Wav2Vec2 CTC trainer
    ├── utils/          # config, logging, data models, IO
    ├── configs/        # packaged default.yaml
    ├── tests/          # pytest suite (no heavy deps required)
    ├── pipeline.py     # end-to-end orchestration
    ├── dataset_builder.py / statistics.py
    └── main.py         # `python main.py` entry point

Design notes:

Lazy heavy imports. faster-whisper, torch and transformers are imported only when their command runs, so the core library and tests stay light and fast.
Streaming everywhere. Transcription consumes whisper's generator; clips and manifest rows are produced per source video; the feature store appends one clip at a time. Memory use is independent of corpus size.
Graceful degradation. WAV IO and mel spectrograms fall back to stdlib + NumPy when soundfile/librosa aren't installed.
Fault tolerant. A failing video is logged and skipped; the run continues.

Output formats

File	Format
`metadata.csv`	`000001.wav
`dataset.jsonl`	`{"audio":"audio/000001.wav","text":"Hello world","duration":2.5}`
`tts_metadata.csv`	`000001.wav
`report.json`	totals, total/average duration, language distribution
`features/train.bin`	flat little-endian float32: audio samples + mel per clip
`features/train.dat`	JSONL index (offsets, shapes, duration, sample rate)

Testing

bash

pip install -e ".[dev]"
pytest                                 # runs video_vec2wav2_tokenizer/tests
pytest --cov=video_vec2wav2_tokenizer  # with coverage

The default test suite needs only numpy, pyyaml and pytest — no FFmpeg, whisper or torch required.

License

MIT.

k9cli/video-vec2wav2-tokenizer

video-vec2wav2-tokenizer

Installation

Quick start

CLI

Multi-GPU training

Configuration (config.yaml)

Architecture

Output formats

Testing

License

k9cli/
video-vec2wav2-tokenizer

Configuration (`config.yaml`)