You can deploy this model and start earning money today!

Bbkblo/
vdub-hindi-dubbing

Name: vdub-hindi-dubbing
Availability: InStock

Views

No views yet

🎬 vdub — Complete Video Dubbing Pipeline (Hindi)

INPUT = video LINK ya FILE → OUTPUT = Hindi dubbed video.

Chinese (ya kisi bhi) video lo, aur complete dubbing pao: transcript → speaker detection → emotion → translation → voice-cloning TTS (Hindi) → timing/pattern fit → BGM separation → final mux. Saare models isi repo me hain — kahin aur jaane ki zaroorat nahi.

📦 Is repo me SAB KUCH hai

Cheez	Kahan	Size
Pipeline code	`vdub_full.py` (12 stages)	39 KB
Colab/Kaggle runner	`notebooks/vdub_full_runner.ipynb`	—
Chatterbox Hindi TTS (voice clone + emotion)	`models/chatterbox_hi/`	3.2 GB
Chatterbox 23-lang pack (optional A/B)	`models/mtl23/`	2.1 GB
Kim_Vocal_2 separation (light, default)	`models/separation/`	67 MB
MelBandRoformer separation (best quality)	`models/separation/`	913 MB
NLLB-200-600M translation (zh→hi)	`models/nllb-200-distilled-600M/`	2.5 GB
Pattern transfer (DSP)	`scripts/pattern_transfer.py`	—
System docs	`docs/SYSTEM_TREE.md`	—
Model refill script	`fetch_models.sh`	—

Total repo: ~8.8 GB — git clone karte hi sab kuch aa jata hai (LFS).

⚡ Quickstart

A) Notebook (sabse aasaan — Colab/Kaggle)

notebooks/vdub_full_runner.ipynb kholo (Colab: File→Upload notebook; Kaggle: upload/import)
GPU (T4) + Internet ON karo
Cell 2 me INPUT_URL ya INPUT_FILE do
Run All → vdub_out/dubbed_video.mp4

B) CLI (local GPU / cloud)

bash

git clone https://huggingface.co/Bbkblo/vdub-hindi-dubbing   # code + saare models
cd vdub-hindi-dubbing
pip install -e "git+https://github.com/resemble-ai/chatterbox.git" --no-deps
pip install librosa safetensors s3tokenizer transformers huggingface_hub soundfile scipy conformer omegaconf pyloudnorm diffusers ml-collections
pip install "git+https://github.com/resemble-ai/Perth.git@master"
# vdub_full.py ke upar CONFIG me INPUT_URL do, phir:
python3 vdub_full.py

(models agar clone se na mile to: bash fetch_models.sh)

🔄 Pipeline (12 stages, resume-safe)

VIDEO (link/file)
  → 1. fetch (yt-dlp ya file)
  → 2. transcript (SRT ya SenseVoice ASR)
  → 3. clip cutting
  → 4. speaker diarization (campplus clustering)
  → 5. emotion (SenseVoice tags)
  → 6. translation → Hindi (NLLB-200 offline / manual json)
  → 7. TTS (Chatterbox: speaker clone + emotion exaggeration)
  → 8. pattern transfer (original ke pauses/speed/loudness pe fit)
  → 9. separation (Kim_Vocal_2 / MelBandRoformer)
  → 10. mux (SRT timing + instrumental + keep-original ranges + 2 audio tracks)
  → 11. QC (voice similarity report)
  → 12. dubbed_video.mp4 + vdub_output.zip (+ Drive upload optional)

Har stage ka marker banta hai — VM/process mare to wahi se resume.

🎛️ Key config (vdub_full.py ke upar)

Param	Default	Kya karta hai
`INPUT_URL` / `INPUT_FILE`	—	dubbing ka source
`SRT_FILE`	""	apna subtitle (warna ASR)
`TRANSLATE_MODE`	nllb	nllb (offline) / manual (json)
`SPEAKER_MAP`	""	{"Speaker 1": "Hero"} names
`SONG_RANGES`	""	[[s,e],...] — yahan original audio rahega
`SEP_MODE`	kim	kim (67MB) / melband (913MB)
`DRIVE_UPLOAD`	False	output Drive me bhi bhejo

TTS knobs: exaggeration (emotion: ANGRY 1.4 / HAPPY 1.1 / SAD 0.4 / NEUTRAL 0.5), cfg_weight 0.3 (cross-lingual accent kam), temperature 0.8.

🤖 Models — license table

Model	License	Use
Chatterbox t3_hi + S3Gen + ve (ResembleAI)	MIT	TTS core
Chatterbox t3_mtl23ls (23-lang)	MIT	optional A/B
Kim_Vocal_2 (UVR5)	MIT	separation default
MelBandRoformer (FunAudioLLM)	Apache-2.0	separation best
NLLB-200-distilled-600M (Meta)	CC-BY-NC-4.0	translation (non-commercial)
SenseVoice/campplus (funasr)	auto-download	ASR/emotion/speaker QC

⚖️ Honest notes

Chatterbox output me PerTh watermark embed hota hai
Cloned voice use karne se pehle awaaz wale insaan ki permission lein
NLLB translation CC-BY-NC hai — commercial use ke liye apna translation API lagao
funasr ke chhote models (SenseVoice/campplus) pehli baar auto-download hote hain

👥 Credits

Voice cloning + Hindi TTS: Resemble AI (Chatterbox, MIT)
Separation (best): FunAudioLLM (Fun-CineForge, Apache-2.0)
Translation: Meta (NLLB-200)
Pipeline + pattern transfer: vdub project (Azim Khorajiya)