Dev emo (#171)
* SYNC CHANGE TO EMO BRANCH (#162) * Update README.md * 更新 bert_models.json * fix * Update data_utils.py * Update infer.py * performance improve * Feat: support auto split in webui (#158) * Feat: support auto split in webui * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix: change /voice api to post (#160) * Fix: change /voice api to post * Fix: support /voice api get * Fix: Add missing torch.cuda.empty_cache() (#161) --------- Co-authored-by: Sora <atri@suzakuintsubaki.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Artrajz <969242373@qq.com> * sync (#163) * Update README.md * 更新 bert_models.json * fix * Update data_utils.py * Update infer.py * performance improve * Feat: support auto split in webui (#158) * Feat: support auto split in webui * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix: change /voice api to post (#160) * Fix: change /voice api to post * Fix: support /voice api get * Fix: Add missing torch.cuda.empty_cache() (#161) * del emo * del emo * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Sora <atri@suzakuintsubaki.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Artrajz <969242373@qq.com> * Add files via upload * Update infer.py * add emo * add emo * Update default_config.yml * Fix slice segments GPU perf (#165) * Fix slice segments GPU perf * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update commons.py --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update infer.py * Update models.py * Update infer.py * remove spec cache * Update data_utils.py * Update data_utils.py * Update train_ms.py * Revert "Fix slice segments GPU perf (#165)" (#169) This reverts commit 28430fc76bc628297bb59d8f8d25100dbe46ab59. * Update train_ms.py * Update train_ms.py * Update data_utils.py * Update data_utils.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update default_config.yml * Switch to Japanese wwm DeBERTa (#172) * Switch to Japanese wwm DeBERTa * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix wrong ellipsis g2p (#173) * Switch to Japanese wwm DeBERTa * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix ellipsis g2p * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add files via upload * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix English phones not aligned with BERT features (#174) * Fix English phones not aligned with BERT features * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix english bert gen (#175) * Update webui.py * Update webui.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add NCCL timeout * Update train_ms.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update train_ms.py * Update default_config.yml * Update infer.py * Update models.py * Update train_ms.py * Update infer.py * Update emo_gen.py * Feat: Support load and infer 2.0 models (#178) * Feat: Support load and infer 2.0 models * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * 复用相同逻辑，修正静音添加错误 (#181) * Refactor: reuse the same part of voice api. * Fix: server_fastapi.py * Update train_ms.py * Update data_utils.py * Update data_utils.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update data_utils.py * Update data_utils.py * Add files via upload * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update default_config.yml * Update utils.py * Update train_ms.py * Update utils.py * Update default_config.yml * Update data_utils.py * Update default_config.yml * Update train_ms.py * Update train_ms.py * Update config.py * Update utils.py * Update train_ms.py * Update train_ms.py * feat: add voice mix and tone mix (#187) * feat: add voice mix and tone mix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Stardust·减 <star_dust_chen@foxmail.com> * Add files via upload * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Sora <atri@suzakuintsubaki.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Artrajz <969242373@qq.com> Co-authored-by: Leng Yue <lengyue@lengyue.me> Co-authored-by: OedoSoldier <31711261+OedoSoldier@users.noreply.github.com> Co-authored-by: 潮幻Mark <sunyunfei201@gmail.com>
This commit is contained in:
Stardust·减
52
models.py
52
models.py
@@ -10,6 +10,8 @@ import monotonic_align
|
||||
|
||||
from torch.nn import Conv1d, ConvTranspose1d, Conv2d
|
||||
from torch.nn.utils import weight_norm, remove_weight_norm, spectral_norm
|
||||
from vector_quantize_pytorch import VectorQuantize
|
||||
|
||||
from commons import init_weights, get_padding
|
||||
from text import symbols, num_tones, num_languages
|
||||
|
||||
@@ -320,6 +322,7 @@ class TextEncoder(nn.Module):
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
n_speakers,
|
||||
gin_channels=0,
|
||||
):
|
||||
super().__init__()
|
||||
@@ -341,6 +344,18 @@ class TextEncoder(nn.Module):
|
||||
self.bert_proj = nn.Conv1d(1024, hidden_channels, 1)
|
||||
self.ja_bert_proj = nn.Conv1d(1024, hidden_channels, 1)
|
||||
self.en_bert_proj = nn.Conv1d(1024, hidden_channels, 1)
|
||||
self.emo_proj = nn.Linear(1024, 1024)
|
||||
self.emo_quantizer = [
|
||||
VectorQuantize(
|
||||
dim=1024,
|
||||
codebook_size=10,
|
||||
decay=0.8,
|
||||
commitment_weight=1.0,
|
||||
learnable_codebook=True,
|
||||
ema_update=False,
|
||||
)
|
||||
] * n_speakers
|
||||
self.emo_q_proj = nn.Linear(1024, hidden_channels)
|
||||
|
||||
self.encoder = attentions.Encoder(
|
||||
hidden_channels,
|
||||
@@ -354,11 +369,32 @@ class TextEncoder(nn.Module):
|
||||
self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
|
||||
|
||||
def forward(
|
||||
self, x, x_lengths, tone, language, bert, ja_bert, en_bert, sid, g=None
|
||||
self, x, x_lengths, tone, language, bert, ja_bert, en_bert, emo, sid, g=None
|
||||
):
|
||||
sid = sid.cpu()
|
||||
bert_emb = self.bert_proj(bert).transpose(1, 2)
|
||||
ja_bert_emb = self.ja_bert_proj(ja_bert).transpose(1, 2)
|
||||
en_bert_emb = self.en_bert_proj(en_bert).transpose(1, 2)
|
||||
if emo.size(-1) == 1024:
|
||||
emo_emb = self.emo_proj(emo.unsqueeze(1))
|
||||
emo_commit_loss = torch.zeros(1)
|
||||
emo_emb_ = []
|
||||
for i in range(emo_emb.size(0)):
|
||||
temp_emo_emb, _, temp_emo_commit_loss = self.emo_quantizer[sid[i]](
|
||||
emo_emb[i].unsqueeze(0).cpu()
|
||||
)
|
||||
emo_commit_loss += temp_emo_commit_loss
|
||||
emo_emb_.append(temp_emo_emb)
|
||||
emo_emb = torch.cat(emo_emb_, dim=0).to(emo_emb.device)
|
||||
emo_commit_loss = emo_commit_loss.to(emo_emb.device)
|
||||
else:
|
||||
emo_emb = (
|
||||
self.emo_quantizer[sid[0]]
|
||||
.get_output_from_indices(emo.to(torch.int).cpu())
|
||||
.unsqueeze(0)
|
||||
.to(emo.device)
|
||||
)
|
||||
emo_commit_loss = torch.zeros(1)
|
||||
x = (
|
||||
self.emb(x)
|
||||
+ self.tone_emb(tone)
|
||||
@@ -366,6 +402,7 @@ class TextEncoder(nn.Module):
|
||||
+ bert_emb
|
||||
+ ja_bert_emb
|
||||
+ en_bert_emb
|
||||
+ self.emo_q_proj(emo_emb)
|
||||
) * math.sqrt(
|
||||
self.hidden_channels
|
||||
) # [b, t, h]
|
||||
@@ -378,7 +415,7 @@ class TextEncoder(nn.Module):
|
||||
stats = self.proj(x) * x_mask
|
||||
|
||||
m, logs = torch.split(stats, self.out_channels, dim=1)
|
||||
return x, m, logs, x_mask
|
||||
return x, m, logs, x_mask, emo_commit_loss
|
||||
|
||||
|
||||
class ResidualCouplingBlock(nn.Module):
|
||||
@@ -811,6 +848,7 @@ class SynthesizerTrn(nn.Module):
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
self.n_speakers,
|
||||
gin_channels=self.enc_gin_channels,
|
||||
)
|
||||
self.dec = Generator(
|
||||
@@ -884,8 +922,8 @@ class SynthesizerTrn(nn.Module):
|
||||
g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
|
||||
else:
|
||||
g = self.ref_enc(y.transpose(1, 2)).unsqueeze(-1)
|
||||
x, m_p, logs_p, x_mask = self.enc_p(
|
||||
x, x_lengths, tone, language, bert, ja_bert, en_bert, sid, g=g
|
||||
x, m_p, logs_p, x_mask, loss_commit = self.enc_p(
|
||||
x, x_lengths, tone, language, bert, ja_bert, en_bert, emo, sid, g=g
|
||||
)
|
||||
z, m_q, logs_q, y_mask = self.enc_q(y, y_lengths, g=g)
|
||||
z_p = self.flow(z, y_mask, g=g)
|
||||
@@ -951,6 +989,7 @@ class SynthesizerTrn(nn.Module):
|
||||
y_mask,
|
||||
(z, z_p, m_p, logs_p, m_q, logs_q),
|
||||
(x, logw, logw_),
|
||||
loss_commit,
|
||||
)
|
||||
|
||||
def infer(
|
||||
@@ -963,6 +1002,7 @@ class SynthesizerTrn(nn.Module):
|
||||
bert,
|
||||
ja_bert,
|
||||
en_bert,
|
||||
emo=None,
|
||||
noise_scale=0.667,
|
||||
length_scale=1,
|
||||
noise_scale_w=0.8,
|
||||
@@ -976,8 +1016,8 @@ class SynthesizerTrn(nn.Module):
|
||||
g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
|
||||
else:
|
||||
g = self.ref_enc(y.transpose(1, 2)).unsqueeze(-1)
|
||||
x, m_p, logs_p, x_mask = self.enc_p(
|
||||
x, x_lengths, tone, language, bert, ja_bert, en_bert, sid, g=g
|
||||
x, m_p, logs_p, x_mask, _ = self.enc_p(
|
||||
x, x_lengths, tone, language, bert, ja_bert, en_bert, emo, sid, g=g
|
||||
)
|
||||
logw = self.sdp(x, x_mask, g=g, reverse=True, noise_scale=noise_scale_w) * (
|
||||
sdp_ratio
|
||||
|
||||
Reference in New Issue
Block a user