56 lines
2.4 KiB
Markdown
56 lines
2.4 KiB
Markdown
# Matcha flow integration
|
|
|
|
Style-Bert-VITS2 can optionally generate its aligned prior latent with a
|
|
Matcha-TTS-inspired conditional flow matching (CFM) decoder. The existing
|
|
BERT/style encoder, duration predictors, VITS normalizing flow, and waveform
|
|
generator are retained.
|
|
|
|
Set `model.use_matcha` to `true` for a newly trained or fine-tuned model. The
|
|
template configurations enable it. Models whose configuration does not contain
|
|
this key continue to use the original Gaussian prior sampling path.
|
|
|
|
Relevant settings:
|
|
|
|
- `train.c_matcha`: weight of the CFM velocity loss.
|
|
- `model.matcha_channels`: U-Net hidden width.
|
|
- `model.matcha_num_heads`: attention head count.
|
|
- `model.matcha_dropout`: transformer dropout.
|
|
- `model.matcha_sigma_min`: minimum flow path noise.
|
|
- `model.matcha_n_timesteps`: Euler steps at inference; more steps trade speed
|
|
for refinement quality.
|
|
- `model.matcha_use_diff_attention`: enables Differential Attention V2 in the
|
|
U-Net transformer blocks. It uses explicit PyTorch matrix multiplication and
|
|
softmax, with no FlashAttention, CUDA-only kernel, or extra dependency.
|
|
|
|
The Matcha branch has new parameters, so enabling it on an existing checkpoint
|
|
requires fine-tuning before inference. It is trained on the same random latent
|
|
segments used by the waveform generator to keep memory use bounded.
|
|
|
|
## Matcha-only fine-tuning
|
|
|
|
Set `train.matcha_only` to `true` to train only `net_g.matcha`. All legacy
|
|
generator parameters are frozen, and adversarial, duration-discriminator,
|
|
WavLM-discriminator, mel, duration, and KL optimization steps are skipped. The
|
|
existing VITS normalizing-flow latent `z_p` remains the detached CFM target.
|
|
|
|
This mode requires `model.use_matcha: true`. Optimizer state from a full-model
|
|
checkpoint is intentionally not restored because its parameter groups differ;
|
|
model weights are still loaded. After the Matcha loss has converged, set
|
|
`train.matcha_only` back to `false` and use a lower learning rate for joint
|
|
fine-tuning.
|
|
|
|
For an already preprocessed dataset (including Google Colab), the mode can be
|
|
configured without rerunning preprocessing:
|
|
|
|
```bash
|
|
python configure_matcha_ft.py --config Data/MyModel/config.json \
|
|
--learning-rate 0.0001 --epochs 20 --save-every-steps 100
|
|
```
|
|
|
|
To switch to the second-stage joint fine-tuning:
|
|
|
|
```bash
|
|
python configure_matcha_ft.py --config Data/MyModel/config.json \
|
|
--joint --learning-rate 0.00002
|
|
```
|