sbv2-v2/docs/MATCHA_FLOW.md

Matcha flow integration

Style-Bert-VITS2 can optionally generate its aligned prior latent with a Matcha-TTS-inspired conditional flow matching (CFM) decoder. The existing BERT/style encoder, duration predictors, VITS normalizing flow, and waveform generator are retained.

Set model.use_matcha to true for a newly trained or fine-tuned model. The template configurations enable it. Models whose configuration does not contain this key continue to use the original Gaussian prior sampling path.

Relevant settings:

• train.c_matcha: weight of the CFM velocity loss.
• model.matcha_channels: U-Net hidden width.
• model.matcha_num_heads: attention head count.
• model.matcha_dropout: transformer dropout.
• model.matcha_sigma_min: minimum flow path noise.
• model.matcha_n_timesteps: Euler steps at inference; more steps trade speed for refinement quality.
• model.matcha_use_diff_attention: enables Differential Attention V2 in the U-Net transformer blocks. It uses explicit PyTorch matrix multiplication and softmax, with no FlashAttention, CUDA-only kernel, or extra dependency.

The Matcha branch has new parameters, so enabling it on an existing checkpoint requires fine-tuning before inference. It is trained on the same random latent segments used by the waveform generator to keep memory use bounded.

Matcha-only fine-tuning

Set train.matcha_only to true to train only net_g.matcha. All legacy generator parameters are frozen, and adversarial, duration-discriminator, WavLM-discriminator, mel, duration, and KL optimization steps are skipped. The existing VITS normalizing-flow latent z_p remains the detached CFM target.

This mode requires model.use_matcha: true. Optimizer state from a full-model checkpoint is intentionally not restored because its parameter groups differ; model weights are still loaded. After the Matcha loss has converged, set train.matcha_only back to false and use a lower learning rate for joint fine-tuning.

For an already preprocessed dataset (including Google Colab), the mode can be configured without rerunning preprocessing:

python configure_matcha_ft.py --config Data/MyModel/config.json \
  --learning-rate 0.0001 --epochs 20 --save-every-steps 100

To switch to the second-stage joint fine-tuning:

python configure_matcha_ft.py --config Data/MyModel/config.json \
  --joint --learning-rate 0.00002