Skip to content
AaronZ345Public

About

PyTorch Implementation of TCSinger 2(ACL 2025): Customizable Multilingual Zero-shot Singing Voice Synthesis

Topics

Resources

Stars

194 stars

Watchers

16 watching

Forks

Latest commit

 

History

41 Commits

Folders and files

Repository files navigation

TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesis
ACL 2025 Findings

Yu Zhang, Wenxiang Guo, Changhao Pan, Dongyu Yao, Zhiyuan Zhu, Ziyue Jiang, Yuhan Wang, Tao Jin, Zhou Zhao Zhejiang University

Paper Demo Page 语音之家 PaperWeekly 知乎

Overview

TCSinger 2 models multilingual singing with style transfer and control from audio and text prompts. This repository provides training and inference code; prepare your trained checkpoints and the auxiliary models before generation.

Start with the model assets and input format, then follow Quick start or the training guide. To cite this work, use the BibTeX entry.

News

  • 2025.07 · Released TCSinger 2 code.
  • 2025.07 · Released STARS code.
  • 2025.05 · Accepted to ACL 2025 Findings.

Quick start

Implementation Requirements
Included code Training, preprocessing, and singing inference
Model checkpoint Your trained TCSinger 2 model, VAE, and matching encoders
Output 48 kHz singing waveform
Runtime Python 3.10, PyTorch 2.1.0 + CUDA 12.1

1. Install the environment

git clone https://github.com/AaronZ345/TCSinger2.git
cd TCSinger2
conda create -n tcsinger2 python=3.10 -y
conda activate tcsinger2
python -m pip install -r requirements.txt \
  --find-links https://download.pytorch.org/whl/torch_stable.html

2. Prepare training or inference assets

The training guide covers GTSinger metadata, phoneme sets, F0 extraction, mel preprocessing, VAE training, the Custom Audio Encoder, and main-model training. Update the paths in your configuration and preprocessing scripts for your data.

For inference, prepare the trained checkpoint, matching model configuration, and processed manifest. The example script reads per-item mel, reference-prompt mel, F0, phoneme, score, and style fields; the manifest must point to existing files. For speech inputs, use the corresponding VAE setup.

3. Run singing inference

PYTHONPATH=. CUDA_VISIBLE_DEVICES=0 python scripts/test_sing.py \
  --config configs/tcsinger2.yaml --ckpt /path/to/your/model.ckpt \
  --manifest_path data/sing/new.tsv --save_dir infer_out

Replace the checkpoint path with your trained model. Use --scales to configure classifier-free guidance. Outputs and their clap.csv index are written under --save_dir.

Model assets

Asset Preparation
VAE and duration predictor Train with configs/ae_singing.yaml
Main singing model Train with configs/tcsinger2.yaml
HIFI-GAN Download to useful_ckpts/hifigan
FLAN-T5 Hugging Face to useful_ckpts/flan-t5-large
Custom Audio Encoder Follow the contrastive-data and VAE instructions in the training guide

Input format

The training metadata includes ph, word, item_name, ph_durs, wav_fn, singer, ep_pitches, ep_notedurs, ep_types, emotion, singing_method, and technique. Keep the phoneme vocabulary, alignment, F0 files, and checkpoint configuration consistent.

Describe singer by gender and vocal range. Represent technique as a list of skills or text describing their sequence. The training guide explains conversion to the model's manifest format.

Inference handles every eligible row (duration up to 20 seconds), including single-row manifests and repeated names. Use --sample-limit 10 --sample-seed 0 for a reproducible subset; this seed controls row selection. Multiple GPUs receive disjoint rows without duplication. The combined clap.csv is ordered by manifest_index after all ranks complete.

Method and training

TCSinger 2 uses a Blurred Boundary Content Encoder, a contrastive Custom Audio Encoder, and a Flow-based Custom Transformer with Cus-MOE and F0 conditioning. See the paper for comparisons across style transfer, style control, cross-lingual transfer, and speech-to-singing transfer.

TCSinger 2 style transfer architecture, Blurred Boundary Content Encoder, and Flow-based Custom Transformer.
Figure 1. TCSinger 2 architecture, from score and prompt encoding to audio generation.

Acknowledgements

This implementation uses parts of the code from the following Github repos: Make-An-Audio-3, TCSinger Lumina-T2X as described in our code.

Citations

If you find this code useful in your research, please cite our work:

@article{zhang2025tcsinger,
  title={TCSinger 2: Customizable Multilingual Zero-shot Singing Voice Synthesis},
  author={Zhang, Yu and Guo, Wenxiang and Pan, Changhao and Yao, Dongyu and Zhu, Zhiyuan and Jiang, Ziyue and Wang, Yuhan and Jin, Tao and Zhao, Zhou},
  journal={arXiv preprint arXiv:2505.14910},
  year={2025}
}

Disclaimer

Any organization or individual is prohibited from using any technology mentioned in this paper to generate someone's singing without his/her consent, including but not limited to government leaders, political figures, and celebrities. If you do not comply with this item, you could be in violation of copyright laws.

About

PyTorch Implementation of TCSinger 2(ACL 2025): Customizable Multilingual Zero-shot Singing Voice Synthesis

Topics

Resources

Stars

194 stars

Watchers

16 watching

Forks

Releases

Packages

Contributors

Languages