tuna2134/sbv2-v2
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Use clap to achieve prompt controlled generation (#223)

Browse Source 
* 快速分类音频并把yml格式结果存在训练根目录里 (#190)

* Add files via upload

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update models.py

* Update webui.py

* Update infer.py

* Create compress_model.py

* 重新提交，更新Gradio推理UI (#193)

* Update webui.py

* Update webui.py

* 更新 train_ms.py

* 更新 models.py

* 更新 models.py

* 更新 models.py

* 更新 train_ms.py

* 更新 train_ms.py

* 更新 models.py

* Update preprocess_text.py

* Update config.json

* Update train_ms.py

* Update webui.py (#206)

* Add files via upload (#209)

* Update train_ms.py

* Update train_ms.py

* Update preprocess_text.py

* Update train_ms.py

* fix (#211)

* Update emotion_clustering.py

* Add files via upload

* Update emotion_clustering.py

* add cluster center save

* Add files via upload

* Update config.py

* Update default_config.yml

* Update config.py

* Update config.py

* Update emotion_clustering.py

* Update emotion_clustering.py

* Update config.py

* Update emotion_clustering.py

* Update emotion_clustering.py

* Update webui.py

* Update emotion_clustering.py

* Update commons.py

* Update emotion_clustering.py

* Update webui.py

* Update webui.py

* Add files via upload

* Update train_ms.py

* Update train_ms.py

* Update train_ms.py

* Update train_ms.py

* Update train_ms.py

* Update webui.py

* Update emotion_clustering.py

* Update emotion_clustering.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix default_config.yml.

* Update infer.py

* feat: support infer 2.1 models

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: support infer 2.1 models 兼容bug修复

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update train_ms.py

* Add CLAP

* Fix data loader

* Fix infer.py

* Fix webui.py

* Add prompt template

* Update clap_gen.py

* Fix wrong environ value

* Add g for dur disc

* Update clap_gen.py

* Fix multilang generation

* Update config.json

* Prompt mode

* Improve slice segments performance

* Add preprocess webui

* Update webui_preprocess.py

* Update webui_preprocess.py

* Update config.py

* Update default_config.yml

* Update config.py

* Update clap_gen.py

* Delete emo_gen.py

* Delete get_emo.py

* Delete emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim directory

* Update README.md

* Update README

* Split val per lang

* Delete emotion_clustering.py

* Update default_config.yml

* Update default_config.yml

* Update config.py

* Update preprocess_text.py

* Update webui_preprocess.py

* Update defalut_config.yml

* Update webui_preprocess.py

* Update preprocess_text.py

* Random augmentation for CLAP

* Update data_utils.py

* Update preprocess_text.py

* Add vq for CLAP features to avoid overfitting

* Random dummy inputs

* Update webui.py

* Update models.py

* Update infer.py

* Apply Code Formatter Change

* Update config.json

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: YYuX-1145 <138500330+YYuX-1145@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sora <654163754@qq.com>
Co-authored-by: Sihan Wang <wangsihan1995@gmail.com>
Co-authored-by: Stardust-minus <Stardust-minus@users.noreply.github.com>
This commit is contained in:
Stardust·减
2023-12-12 19:19:50 +08:00
committed by GitHub
parent 9cc786d781
commit eaefc57d71
49 changed files with 287404 additions and 3426 deletions
Download Patch File Download Diff File Expand all files Collapse all files

722
infer.py
View File

@@ -1,341 +1,381 @@
"""
版本管理、兼容推理及模型加载实现。
版本说明：
1. 版本号与github的release版本号对应，使用哪个release版本训练的模型即对应其版本号
2. 请在模型的config.json中显示声明版本号，添加一个字段"version" : "你的版本号"
特殊版本说明：
1.1.1-fix： 1.1.1版本训练的模型，但是在推理时使用dev的日语修复
1.1.1-dev： dev开发
2.1：当前版本
"""
import torch
import commons
from text import cleaned_text_to_sequence, get_bert
from get_emo import get_emo
from text.cleaner import clean_text
import utils
from models import SynthesizerTrn
from text.symbols import symbols
from oldVersion.V200.models import SynthesizerTrn as V200SynthesizerTrn
from oldVersion.V200.text import symbols as V200symbols
from oldVersion.V111.models import SynthesizerTrn as V111SynthesizerTrn
from oldVersion.V111.text import symbols as V111symbols
from oldVersion.V110.models import SynthesizerTrn as V110SynthesizerTrn
from oldVersion.V110.text import symbols as V110symbols
from oldVersion.V101.models import SynthesizerTrn as V101SynthesizerTrn
from oldVersion.V101.text import symbols as V101symbols
from oldVersion import V111, V110, V101, V200
# 当前版本信息
latest_version = "2.1"
# 版本兼容
SynthesizerTrnMap = {
"2.0.2-fix": V200SynthesizerTrn,
"2.0.1": V200SynthesizerTrn,
"2.0": V200SynthesizerTrn,
"1.1.1-fix": V111SynthesizerTrn,
"1.1.1": V111SynthesizerTrn,
"1.1": V110SynthesizerTrn,
"1.1.0": V110SynthesizerTrn,
"1.0.1": V101SynthesizerTrn,
"1.0": V101SynthesizerTrn,
"1.0.0": V101SynthesizerTrn,
}
symbolsMap = {
"2.0.2-fix": V200symbols,
"2.0.1": V200symbols,
"2.0": V200symbols,
"1.1.1-fix": V111symbols,
"1.1.1": V111symbols,
"1.1": V110symbols,
"1.1.0": V110symbols,
"1.0.1": V101symbols,
"1.0": V101symbols,
"1.0.0": V101symbols,
}
def get_net_g(model_path: str, version: str, device: str, hps):
if version != latest_version:
net_g = SynthesizerTrnMap[version](
len(symbolsMap[version]),
hps.data.filter_length // 2 + 1,
hps.train.segment_size // hps.data.hop_length,
n_speakers=hps.data.n_speakers,
**hps.model,
).to(device)
else:
# 当前版本模型 net_g
net_g = SynthesizerTrn(
len(symbols),
hps.data.filter_length // 2 + 1,
hps.train.segment_size // hps.data.hop_length,
n_speakers=hps.data.n_speakers,
**hps.model,
).to(device)
_ = net_g.eval()
_ = utils.load_checkpoint(model_path, net_g, None, skip_optimizer=True)
return net_g
def get_text(text, language_str, hps, device):
# 在此处实现当前版本的get_text
norm_text, phone, tone, word2ph = clean_text(text, language_str)
phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
if hps.data.add_blank:
phone = commons.intersperse(phone, 0)
tone = commons.intersperse(tone, 0)
language = commons.intersperse(language, 0)
for i in range(len(word2ph)):
word2ph[i] = word2ph[i] * 2
word2ph[0] += 1
bert_ori = get_bert(norm_text, word2ph, language_str, device)
del word2ph
assert bert_ori.shape[-1] == len(phone), phone
if language_str == "ZH":
bert = bert_ori
ja_bert = torch.zeros(1024, len(phone))
en_bert = torch.zeros(1024, len(phone))
elif language_str == "JP":
bert = torch.zeros(1024, len(phone))
ja_bert = bert_ori
en_bert = torch.zeros(1024, len(phone))
elif language_str == "EN":
bert = torch.zeros(1024, len(phone))
ja_bert = torch.zeros(1024, len(phone))
en_bert = bert_ori
else:
raise ValueError("language_str should be ZH, JP or EN")
assert bert.shape[-1] == len(
phone
), f"Bert seq len {bert.shape[-1]} != {len(phone)}"
phone = torch.LongTensor(phone)
tone = torch.LongTensor(tone)
language = torch.LongTensor(language)
return bert, ja_bert, en_bert, phone, tone, language
def get_emo_(reference_audio, emotion):
emo = (
torch.from_numpy(get_emo(reference_audio))
if reference_audio
else torch.Tensor([emotion])
)
return emo
def infer(
text,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
language,
hps,
net_g,
device,
reference_audio=None,
emotion=None,
skip_start=False,
skip_end=False,
):
# 支持中日英三语版本
inferMap_V2 = {
"2.0.2-fix": V200.infer,
"2.0.1": V200.infer,
"2.0": V200.infer,
"1.1.1-fix": V111.infer_fix,
"1.1.1": V111.infer,
"1.1": V110.infer,
"1.1.0": V110.infer,
}
# 仅支持中文版本
# 在测试中，并未发现两个版本的模型不能互相通用
inferMap_V1 = {
"1.0.1": V101.infer,
"1.0": V101.infer,
"1.0.0": V101.infer,
}
version = hps.version if hasattr(hps, "version") else latest_version
# 非当前版本，根据版本号选择合适的infer
if version != latest_version:
if version in inferMap_V2.keys():
return inferMap_V2[version](
text,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
language,
hps,
net_g,
device,
)
if version in inferMap_V1.keys():
return inferMap_V1[version](
text,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
hps,
net_g,
device,
)
# 在此处实现当前版本的推理
bert, ja_bert, en_bert, phones, tones, lang_ids = get_text(
text, language, hps, device
)
emo = get_emo_(reference_audio, emotion)
if skip_start:
phones = phones[3:]
tones = tones[3:]
lang_ids = lang_ids[3:]
bert = bert[:, 3:]
ja_bert = ja_bert[:, 3:]
en_bert = en_bert[:, 3:]
if skip_end:
phones = phones[:-2]
tones = tones[:-2]
lang_ids = lang_ids[:-2]
bert = bert[:, :-2]
ja_bert = ja_bert[:, :-2]
en_bert = en_bert[:, :-2]
with torch.no_grad():
x_tst = phones.to(device).unsqueeze(0)
tones = tones.to(device).unsqueeze(0)
lang_ids = lang_ids.to(device).unsqueeze(0)
bert = bert.to(device).unsqueeze(0)
ja_bert = ja_bert.to(device).unsqueeze(0)
en_bert = en_bert.to(device).unsqueeze(0)
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
emo = emo.to(device).unsqueeze(0)
del phones
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
audio = (
net_g.infer(
x_tst,
x_tst_lengths,
speakers,
tones,
lang_ids,
bert,
ja_bert,
en_bert,
emo,
sdp_ratio=sdp_ratio,
noise_scale=noise_scale,
noise_scale_w=noise_scale_w,
length_scale=length_scale,
)[0][0, 0]
.data.cpu()
.float()
.numpy()
)
del x_tst, tones, lang_ids, bert, x_tst_lengths, speakers, ja_bert, en_bert, emo
if torch.cuda.is_available():
torch.cuda.empty_cache()
return audio
def infer_multilang(
text,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
language,
hps,
net_g,
device,
reference_audio=None,
emotion=None,
skip_start=False,
skip_end=False,
):
bert, ja_bert, en_bert, phones, tones, lang_ids = [], [], [], [], [], []
emo = get_emo_(reference_audio, emotion)
for idx, (txt, lang) in enumerate(zip(text, language)):
skip_start = (idx != 0) or (skip_start and idx == 0)
skip_end = (idx != len(text) - 1) or (skip_end and idx == len(text) - 1)
(
temp_bert,
temp_ja_bert,
temp_en_bert,
temp_phones,
temp_tones,
temp_lang_ids,
) = get_text(txt, lang, hps, device)
if skip_start:
temp_bert = temp_bert[:, 3:]
temp_ja_bert = temp_ja_bert[:, 3:]
temp_en_bert = temp_en_bert[:, 3:]
temp_phones = temp_phones[3:]
temp_tones = temp_tones[3:]
temp_lang_ids = temp_lang_ids[3:]
if skip_end:
temp_bert = temp_bert[:, :-2]
temp_ja_bert = temp_ja_bert[:, :-2]
temp_en_bert = temp_en_bert[:, :-2]
temp_phones = temp_phones[:-2]
temp_tones = temp_tones[:-2]
temp_lang_ids = temp_lang_ids[:-2]
bert.append(temp_bert)
ja_bert.append(temp_ja_bert)
en_bert.append(temp_en_bert)
phones.append(temp_phones)
tones.append(temp_tones)
lang_ids.append(temp_lang_ids)
bert = torch.concatenate(bert, dim=1)
ja_bert = torch.concatenate(ja_bert, dim=1)
en_bert = torch.concatenate(en_bert, dim=1)
phones = torch.concatenate(phones, dim=0)
tones = torch.concatenate(tones, dim=0)
lang_ids = torch.concatenate(lang_ids, dim=0)
with torch.no_grad():
x_tst = phones.to(device).unsqueeze(0)
tones = tones.to(device).unsqueeze(0)
lang_ids = lang_ids.to(device).unsqueeze(0)
bert = bert.to(device).unsqueeze(0)
ja_bert = ja_bert.to(device).unsqueeze(0)
en_bert = en_bert.to(device).unsqueeze(0)
emo = emo.to(device).unsqueeze(0)
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
del phones
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
audio = (
net_g.infer(
x_tst,
x_tst_lengths,
speakers,
tones,
lang_ids,
bert,
ja_bert,
en_bert,
emo,
sdp_ratio=sdp_ratio,
noise_scale=noise_scale,
noise_scale_w=noise_scale_w,
length_scale=length_scale,
)[0][0, 0]
.data.cpu()
.float()
.numpy()
)
del x_tst, tones, lang_ids, bert, x_tst_lengths, speakers, ja_bert, en_bert, emo
if torch.cuda.is_available():
torch.cuda.empty_cache()
return audio
"""
版本管理、兼容推理及模型加载实现。
版本说明：
1. 版本号与github的release版本号对应，使用哪个release版本训练的模型即对应其版本号
2. 请在模型的config.json中显示声明版本号，添加一个字段"version" : "你的版本号"
特殊版本说明：
1.1.1-fix： 1.1.1版本训练的模型，但是在推理时使用dev的日语修复
2.2：当前版本
"""
import torch
import commons
from text import cleaned_text_to_sequence, get_bert
from clap_wrapper import get_clap_audio_feature, get_clap_text_feature
from text.cleaner import clean_text
import utils
import numpy as np
from models import SynthesizerTrn
from text.symbols import symbols
from oldVersion.V210.models import SynthesizerTrn as V210SynthesizerTrn
from oldVersion.V210.text import symbols as V210symbols
from oldVersion.V200.models import SynthesizerTrn as V200SynthesizerTrn
from oldVersion.V200.text import symbols as V200symbols
from oldVersion.V111.models import SynthesizerTrn as V111SynthesizerTrn
from oldVersion.V111.text import symbols as V111symbols
from oldVersion.V110.models import SynthesizerTrn as V110SynthesizerTrn
from oldVersion.V110.text import symbols as V110symbols
from oldVersion.V101.models import SynthesizerTrn as V101SynthesizerTrn
from oldVersion.V101.text import symbols as V101symbols
from oldVersion import V111, V110, V101, V200
# 当前版本信息
latest_version = "2.2"
# 版本兼容
SynthesizerTrnMap = {
"2.1": V210SynthesizerTrn,
"2.0.2-fix": V200SynthesizerTrn,
"2.0.1": V200SynthesizerTrn,
"2.0": V200SynthesizerTrn,
"1.1.1-fix": V111SynthesizerTrn,
"1.1.1": V111SynthesizerTrn,
"1.1": V110SynthesizerTrn,
"1.1.0": V110SynthesizerTrn,
"1.0.1": V101SynthesizerTrn,
"1.0": V101SynthesizerTrn,
"1.0.0": V101SynthesizerTrn,
}
symbolsMap = {
"2.1": V210symbols,
"2.0.2-fix": V200symbols,
"2.0.1": V200symbols,
"2.0": V200symbols,
"1.1.1-fix": V111symbols,
"1.1.1": V111symbols,
"1.1": V110symbols,
"1.1.0": V110symbols,
"1.0.1": V101symbols,
"1.0": V101symbols,
"1.0.0": V101symbols,
}
# def get_emo_(reference_audio, emotion, sid):
# emo = (
# torch.from_numpy(get_emo(reference_audio))
# if reference_audio and emotion == -1
# else torch.FloatTensor(
# np.load(f"emo_clustering/{sid}/cluster_center_{emotion}.npy")
# )
# )
# return emo
def get_net_g(model_path: str, version: str, device: str, hps):
if version != latest_version:
net_g = SynthesizerTrnMap[version](
len(symbolsMap[version]),
hps.data.filter_length // 2 + 1,
hps.train.segment_size // hps.data.hop_length,
n_speakers=hps.data.n_speakers,
**hps.model,
).to(device)
else:
# 当前版本模型 net_g
net_g = SynthesizerTrn(
len(symbols),
hps.data.filter_length // 2 + 1,
hps.train.segment_size // hps.data.hop_length,
n_speakers=hps.data.n_speakers,
**hps.model,
).to(device)
_ = net_g.eval()
_ = utils.load_checkpoint(model_path, net_g, None, skip_optimizer=True)
return net_g
def get_text(text, language_str, hps, device):
# 在此处实现当前版本的get_text
norm_text, phone, tone, word2ph = clean_text(text, language_str)
phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
if hps.data.add_blank:
phone = commons.intersperse(phone, 0)
tone = commons.intersperse(tone, 0)
language = commons.intersperse(language, 0)
for i in range(len(word2ph)):
word2ph[i] = word2ph[i] * 2
word2ph[0] += 1
bert_ori = get_bert(norm_text, word2ph, language_str, device)
del word2ph
assert bert_ori.shape[-1] == len(phone), phone
if language_str == "ZH":
bert = bert_ori
ja_bert = torch.rand(1024, len(phone))
en_bert = torch.rand(1024, len(phone))
elif language_str == "JP":
bert = torch.rand(1024, len(phone))
ja_bert = bert_ori
en_bert = torch.rand(1024, len(phone))
elif language_str == "EN":
bert = torch.rand(1024, len(phone))
ja_bert = torch.rand(1024, len(phone))
en_bert = bert_ori
else:
raise ValueError("language_str should be ZH, JP or EN")
assert bert.shape[-1] == len(
phone
), f"Bert seq len {bert.shape[-1]} != {len(phone)}"
phone = torch.LongTensor(phone)
tone = torch.LongTensor(tone)
language = torch.LongTensor(language)
return bert, ja_bert, en_bert, phone, tone, language
def infer(
text,
emotion,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
language,
hps,
net_g,
device,
reference_audio=None,
skip_start=False,
skip_end=False,
):
# 2.2版本参数位置变了
# 2.1 参数新增 emotion reference_audio skip_start skip_end
# inferMap_V3 = {
# "2.1": V210.infer,
# }
# 支持中日英三语版本
inferMap_V2 = {
"2.0.2-fix": V200.infer,
"2.0.1": V200.infer,
"2.0": V200.infer,
"1.1.1-fix": V111.infer_fix,
"1.1.1": V111.infer,
"1.1": V110.infer,
"1.1.0": V110.infer,
}
# 仅支持中文版本
# 在测试中，并未发现两个版本的模型不能互相通用
inferMap_V1 = {
"1.0.1": V101.infer,
"1.0": V101.infer,
"1.0.0": V101.infer,
}
version = hps.version if hasattr(hps, "version") else latest_version
# 非当前版本，根据版本号选择合适的infer
if version != latest_version:
# if version in inferMap_V3.keys():
# return inferMap_V3[version](
# text,
# sdp_ratio,
# noise_scale,
# noise_scale_w,
# length_scale,
# sid,
# language,
# hps,
# net_g,
# device,
# reference_audio,
# emotion,
# skip_start,
# skip_end,
# )
if version in inferMap_V2.keys():
return inferMap_V2[version](
text,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
language,
hps,
net_g,
device,
)
if version in inferMap_V1.keys():
return inferMap_V1[version](
text,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
hps,
net_g,
device,
)
# 在此处实现当前版本的推理
# emo = get_emo_(reference_audio, emotion, sid)
if isinstance(reference_audio, np.ndarray):
emo = get_clap_audio_feature(reference_audio, device)
else:
emo = get_clap_text_feature(emotion, device)
emo = torch.squeeze(emo, dim=1)
bert, ja_bert, en_bert, phones, tones, lang_ids = get_text(
text, language, hps, device
)
if skip_start:
phones = phones[3:]
tones = tones[3:]
lang_ids = lang_ids[3:]
bert = bert[:, 3:]
ja_bert = ja_bert[:, 3:]
en_bert = en_bert[:, 3:]
if skip_end:
phones = phones[:-2]
tones = tones[:-2]
lang_ids = lang_ids[:-2]
bert = bert[:, :-2]
ja_bert = ja_bert[:, :-2]
en_bert = en_bert[:, :-2]
with torch.no_grad():
x_tst = phones.to(device).unsqueeze(0)
tones = tones.to(device).unsqueeze(0)
lang_ids = lang_ids.to(device).unsqueeze(0)
bert = bert.to(device).unsqueeze(0)
ja_bert = ja_bert.to(device).unsqueeze(0)
en_bert = en_bert.to(device).unsqueeze(0)
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
emo = emo.to(device).unsqueeze(0)
del phones
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
audio = (
net_g.infer(
x_tst,
x_tst_lengths,
speakers,
tones,
lang_ids,
bert,
ja_bert,
en_bert,
emo,
sdp_ratio=sdp_ratio,
noise_scale=noise_scale,
noise_scale_w=noise_scale_w,
length_scale=length_scale,
)[0][0, 0]
.data.cpu()
.float()
.numpy()
)
del x_tst, tones, lang_ids, bert, x_tst_lengths, speakers, ja_bert, en_bert, emo
if torch.cuda.is_available():
torch.cuda.empty_cache()
return audio
def infer_multilang(
text,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
language,
hps,
net_g,
device,
reference_audio=None,
emotion=None,
skip_start=False,
skip_end=False,
):
bert, ja_bert, en_bert, phones, tones, lang_ids = [], [], [], [], [], []
# emo = get_emo_(reference_audio, emotion, sid)
if isinstance(reference_audio, np.ndarray):
emo = get_clap_audio_feature(reference_audio, device)
else:
emo = get_clap_text_feature(emotion, device)
emo = torch.squeeze(emo, dim=1)
for idx, (txt, lang) in enumerate(zip(text, language)):
skip_start = (idx != 0) or (skip_start and idx == 0)
skip_end = (idx != len(text) - 1) or (skip_end and idx == len(text) - 1)
(
temp_bert,
temp_ja_bert,
temp_en_bert,
temp_phones,
temp_tones,
temp_lang_ids,
) = get_text(txt, lang, hps, device)
if skip_start:
temp_bert = temp_bert[:, 3:]
temp_ja_bert = temp_ja_bert[:, 3:]
temp_en_bert = temp_en_bert[:, 3:]
temp_phones = temp_phones[3:]
temp_tones = temp_tones[3:]
temp_lang_ids = temp_lang_ids[3:]
if skip_end:
temp_bert = temp_bert[:, :-2]
temp_ja_bert = temp_ja_bert[:, :-2]
temp_en_bert = temp_en_bert[:, :-2]
temp_phones = temp_phones[:-2]
temp_tones = temp_tones[:-2]
temp_lang_ids = temp_lang_ids[:-2]
bert.append(temp_bert)
ja_bert.append(temp_ja_bert)
en_bert.append(temp_en_bert)
phones.append(temp_phones)
tones.append(temp_tones)
lang_ids.append(temp_lang_ids)
bert = torch.concatenate(bert, dim=1)
ja_bert = torch.concatenate(ja_bert, dim=1)
en_bert = torch.concatenate(en_bert, dim=1)
phones = torch.concatenate(phones, dim=0)
tones = torch.concatenate(tones, dim=0)
lang_ids = torch.concatenate(lang_ids, dim=0)
with torch.no_grad():
x_tst = phones.to(device).unsqueeze(0)
tones = tones.to(device).unsqueeze(0)
lang_ids = lang_ids.to(device).unsqueeze(0)
bert = bert.to(device).unsqueeze(0)
ja_bert = ja_bert.to(device).unsqueeze(0)
en_bert = en_bert.to(device).unsqueeze(0)
emo = emo.to(device).unsqueeze(0)
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
del phones
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
audio = (
net_g.infer(
x_tst,
x_tst_lengths,
speakers,
tones,
lang_ids,
bert,
ja_bert,
en_bert,
emo,
sdp_ratio=sdp_ratio,
noise_scale=noise_scale,
noise_scale_w=noise_scale_w,
length_scale=length_scale,
)[0][0, 0]
.data.cpu()
.float()
.numpy()
)
del x_tst, tones, lang_ids, bert, x_tst_lengths, speakers, ja_bert, en_bert, emo
if torch.cuda.is_available():
torch.cuda.empty_cache()
return audio