This repository hosts
DiCoW v3.3, a Target-Speaker ASR (TS-ASR) model developed by
BUT Speech@FIT. It is designed to transcribe the speech of a specific speaker within a multi-talker mixture by conditioning on speaker diarization outputs.
This model version incorporates the refinements and training strategies described in the paper SE-DiCoW: Self-Enrolled Diarization-Conditioned Whisper.
This version represents a significant stabilization and enhancement over the original DiCoW (v1):
The easiest way to use this model is via the
DiCoW inference repository. We provide a Gradio app that handles diarization and STNO mask generation automatically:
1from transformers import AutoModelForSpeechSeq2Seq
2
3# Load the model (requires remote code for custom FDDT layers)
4model = AutoModelForSpeechSeq2Seq.from_pretrained(
5 "BUT-FIT/DiCoW_v3_3",
6 trust_remote_code=True
7)
8
9# Note: The model expects specific STNO conditioning inputs.
10# See inference.py in the GitHub repo for the full pipeline.
It's all yours with just two commands! This model is fully open-source and reproducible using our toolkit.
1. Data Preparation
Clone the
mt-asr-data-prep repository and run the setup script to generate the required manifests:
2. Training
Clone the training repository
TS-ASR-Whisper and launch the experiment using the pre-configured
dicow_v3 recipe:
1@article{polok2026sedicow,
2 title={SE-DiCoW: Self-Enrolled Diarization-Conditioned Whisper},
3 author={Alexander Polok and Dominik Klement and Samuele Cornell and Matthew Wiesner and Jan Černocký and Sanjeev Khudanpur and Lukáš Burget},
4 journal={arXiv preprint arXiv:2601.19194},
5 year={2026}
6}
7
8@article{POLOK2026101841,
9 title = {DiCoW: Diarization-conditioned Whisper for target speaker automatic speech recognition},
10 journal = {Computer Speech & Language},
11 volume = {95},
12 year = {2026},
13 doi = {10.1016/j.csl.2025.101841},
14 author = {Alexander Polok et al.}
15}
16
17@INPROCEEDINGS{10887683,
18 title={Target Speaker ASR with Whisper},
19 author={Polok, Alexander et al.},
20 booktitle={ICASSP 2025},
21 year={2025},
22 doi={10.1109/ICASSP49660.2025.10887683}
23}
