NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction

Authors: Qichao Wang*, Ziqiao Meng*, Wenqian Cui, Yifei Zhang, Pengcheng Wu, Bingzhe Wu, Irwin King, Liang Chen, Peilin Zhao†

• Pre-training: Transform single-channel audio into discrete tokens for next-token prediction
• SFT: Novel "next-token-pair prediction" objective for natural conversation comprehension
• Result: More natural and fluid spoken interactions compared to baseline approaches

Installation

Usage

1. Prepare audio data for pre-training and fine-tuning
2. Pre-train: python pretrain.py --input_data path/to/single_channel_data
3. Fine-tune: python finetune.py --input_data path/to/double_channel_data
4. Inference: python inference.py --input_audio path/to/input.wav