Dev emo (#171)
* SYNC CHANGE TO EMO BRANCH (#162) * Update README.md * 更新 bert_models.json * fix * Update data_utils.py * Update infer.py * performance improve * Feat: support auto split in webui (#158) * Feat: support auto split in webui * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix: change /voice api to post (#160) * Fix: change /voice api to post * Fix: support /voice api get * Fix: Add missing torch.cuda.empty_cache() (#161) --------- Co-authored-by: Sora <atri@suzakuintsubaki.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Artrajz <969242373@qq.com> * sync (#163) * Update README.md * 更新 bert_models.json * fix * Update data_utils.py * Update infer.py * performance improve * Feat: support auto split in webui (#158) * Feat: support auto split in webui * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix: change /voice api to post (#160) * Fix: change /voice api to post * Fix: support /voice api get * Fix: Add missing torch.cuda.empty_cache() (#161) * del emo * del emo * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Sora <atri@suzakuintsubaki.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Artrajz <969242373@qq.com> * Add files via upload * Update infer.py * add emo * add emo * Update default_config.yml * Fix slice segments GPU perf (#165) * Fix slice segments GPU perf * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update commons.py --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update infer.py * Update models.py * Update infer.py * remove spec cache * Update data_utils.py * Update data_utils.py * Update train_ms.py * Revert "Fix slice segments GPU perf (#165)" (#169) This reverts commit 28430fc76bc628297bb59d8f8d25100dbe46ab59. * Update train_ms.py * Update train_ms.py * Update data_utils.py * Update data_utils.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update default_config.yml * Switch to Japanese wwm DeBERTa (#172) * Switch to Japanese wwm DeBERTa * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix wrong ellipsis g2p (#173) * Switch to Japanese wwm DeBERTa * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix ellipsis g2p * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add files via upload * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix English phones not aligned with BERT features (#174) * Fix English phones not aligned with BERT features * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix english bert gen (#175) * Update webui.py * Update webui.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add NCCL timeout * Update train_ms.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update train_ms.py * Update default_config.yml * Update infer.py * Update models.py * Update train_ms.py * Update infer.py * Update emo_gen.py * Feat: Support load and infer 2.0 models (#178) * Feat: Support load and infer 2.0 models * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * 复用相同逻辑，修正静音添加错误 (#181) * Refactor: reuse the same part of voice api. * Fix: server_fastapi.py * Update train_ms.py * Update data_utils.py * Update data_utils.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update data_utils.py * Update data_utils.py * Add files via upload * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update default_config.yml * Update utils.py * Update train_ms.py * Update utils.py * Update default_config.yml * Update data_utils.py * Update default_config.yml * Update train_ms.py * Update train_ms.py * Update config.py * Update utils.py * Update train_ms.py * Update train_ms.py * feat: add voice mix and tone mix (#187) * feat: add voice mix and tone mix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Stardust·减 <star_dust_chen@foxmail.com> * Add files via upload * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Sora <atri@suzakuintsubaki.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Artrajz <969242373@qq.com> Co-authored-by: Leng Yue <lengyue@lengyue.me> Co-authored-by: OedoSoldier <31711261+OedoSoldier@users.noreply.github.com> Co-authored-by: 潮幻Mark <sunyunfei201@gmail.com>
This commit is contained in:
Stardust·减
@@ -30,6 +30,7 @@ rep_map = {
|
||||
"$": ".",
|
||||
"“": "'",
|
||||
"”": "'",
|
||||
'"': "'",
|
||||
"‘": "'",
|
||||
"’": "'",
|
||||
"（": "'",
|
||||
|
||||
121
text/english.py
121
text/english.py
@@ -2,6 +2,7 @@ import pickle
|
||||
import os
|
||||
import re
|
||||
from g2p_en import G2p
|
||||
from transformers import DebertaV2Tokenizer
|
||||
|
||||
from text import symbols
|
||||
|
||||
@@ -9,6 +10,8 @@ current_file_path = os.path.dirname(__file__)
|
||||
CMU_DICT_PATH = os.path.join(current_file_path, "cmudict.rep")
|
||||
CACHE_PATH = os.path.join(current_file_path, "cmudict_cache.pickle")
|
||||
_g2p = G2p()
|
||||
LOCAL_PATH = "./bert/deberta-v3-large"
|
||||
tokenizer = DebertaV2Tokenizer.from_pretrained(LOCAL_PATH)
|
||||
|
||||
arpa = {
|
||||
"AH0",
|
||||
@@ -96,7 +99,9 @@ def post_replace_ph(ph):
|
||||
"\n": ".",
|
||||
"·": ",",
|
||||
"、": ",",
|
||||
"...": "…",
|
||||
"…": "...",
|
||||
"···": "...",
|
||||
"・・・": "...",
|
||||
"v": "V",
|
||||
}
|
||||
if ph in rep_map.keys():
|
||||
@@ -108,6 +113,62 @@ def post_replace_ph(ph):
|
||||
return ph
|
||||
|
||||
|
||||
rep_map = {
|
||||
"：": ",",
|
||||
"；": ",",
|
||||
"，": ",",
|
||||
"。": ".",
|
||||
"！": "!",
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"．": ".",
|
||||
"…": "...",
|
||||
"···": "...",
|
||||
"・・・": "...",
|
||||
"·": ",",
|
||||
"・": ",",
|
||||
"、": ",",
|
||||
"$": ".",
|
||||
"“": "'",
|
||||
"”": "'",
|
||||
'"': "'",
|
||||
"‘": "'",
|
||||
"’": "'",
|
||||
"（": "'",
|
||||
"）": "'",
|
||||
"(": "'",
|
||||
")": "'",
|
||||
"《": "'",
|
||||
"》": "'",
|
||||
"【": "'",
|
||||
"】": "'",
|
||||
"[": "'",
|
||||
"]": "'",
|
||||
"—": "-",
|
||||
"−": "-",
|
||||
"～": "-",
|
||||
"~": "-",
|
||||
"「": "'",
|
||||
"」": "'",
|
||||
}
|
||||
|
||||
|
||||
def replace_punctuation(text):
|
||||
pattern = re.compile("|".join(re.escape(p) for p in rep_map.keys()))
|
||||
|
||||
replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
|
||||
|
||||
# replaced_text = re.sub(
|
||||
# r"[^\u3040-\u309F\u30A0-\u30FF\u4E00-\u9FFF\u3400-\u4DBF\u3005"
|
||||
# + "".join(punctuation)
|
||||
# + r"]+",
|
||||
# "",
|
||||
# replaced_text,
|
||||
# )
|
||||
|
||||
return replaced_text
|
||||
|
||||
|
||||
def read_dict():
|
||||
g2p_dict = {}
|
||||
start_line = 49
|
||||
@@ -308,38 +369,68 @@ def normalize_numbers(text):
|
||||
|
||||
def text_normalize(text):
|
||||
text = normalize_numbers(text)
|
||||
text = replace_punctuation(text)
|
||||
text = re.sub(r"([,;.\?\!])([\w])", r"\1 \2", text)
|
||||
return text
|
||||
|
||||
|
||||
def distribute_phone(n_phone, n_word):
|
||||
phones_per_word = [0] * n_word
|
||||
for task in range(n_phone):
|
||||
min_tasks = min(phones_per_word)
|
||||
min_index = phones_per_word.index(min_tasks)
|
||||
phones_per_word[min_index] += 1
|
||||
return phones_per_word
|
||||
|
||||
|
||||
def sep_text(text):
|
||||
words = re.split(r"([,;.\?\!\s+])", text)
|
||||
words = [word for word in words if word.strip() != ""]
|
||||
return words
|
||||
|
||||
|
||||
def g2p(text):
|
||||
phones = []
|
||||
tones = []
|
||||
word2ph = []
|
||||
words = re.split(r"([,;.\-\?\!\s+])", text)
|
||||
words = [word for word in words if word.strip() != ""]
|
||||
# word2ph = []
|
||||
words = sep_text(text)
|
||||
tokens = [tokenizer.tokenize(i) for i in words]
|
||||
for word in words:
|
||||
if word.upper() in eng_dict:
|
||||
phns, tns = refine_syllables(eng_dict[word.upper()])
|
||||
phones += phns
|
||||
tones += tns
|
||||
word2ph.append(len(phns))
|
||||
phones.append([post_replace_ph(i) for i in phns])
|
||||
tones.append(tns)
|
||||
# word2ph.append(len(phns))
|
||||
else:
|
||||
phone_list = list(filter(lambda p: p != " ", _g2p(word)))
|
||||
phns = []
|
||||
tns = []
|
||||
for ph in phone_list:
|
||||
if ph in arpa:
|
||||
ph, tn = refine_ph(ph)
|
||||
phones.append(ph)
|
||||
tones.append(tn)
|
||||
phns.append(ph)
|
||||
tns.append(tn)
|
||||
else:
|
||||
phones.append(ph)
|
||||
tones.append(0)
|
||||
word2ph.append(len(phone_list))
|
||||
phns.append(ph)
|
||||
tns.append(0)
|
||||
phones.append([post_replace_ph(i) for i in phns])
|
||||
tones.append(tns)
|
||||
# word2ph.append(len(phns))
|
||||
# phones = [post_replace_ph(i) for i in phones]
|
||||
|
||||
phones = [post_replace_ph(i) for i in phones]
|
||||
word2ph = []
|
||||
for token, phoneme in zip(tokens, phones):
|
||||
phone_len = len(phoneme)
|
||||
word_len = len(token)
|
||||
|
||||
phones = ["_"] + phones + ["_"]
|
||||
tones = [0] + tones + [0]
|
||||
aaa = distribute_phone(phone_len, word_len)
|
||||
word2ph += aaa
|
||||
|
||||
phones = ["_"] + [j for i in phones for j in i] + ["_"]
|
||||
tones = [0] + [j for i in tones for j in i] + [0]
|
||||
word2ph = [1] + word2ph + [1]
|
||||
assert len(phones) == len(tones), text
|
||||
assert len(phones) == sum(word2ph), text
|
||||
|
||||
return phones, tones, word2ph
|
||||
|
||||
|
||||
@@ -30,7 +30,7 @@ def get_bert_feature(text, word2ph, device=config.bert_gen_config.device):
|
||||
inputs[i] = inputs[i].to(device)
|
||||
res = models[device](**inputs, output_hidden_states=True)
|
||||
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
|
||||
# assert len(word2ph) == len(text)+2
|
||||
assert len(word2ph) == res.shape[0], (text, res.shape[0], len(word2ph))
|
||||
word2phone = word2ph
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
|
||||
@@ -266,15 +266,16 @@ rep_map = {
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"．": ".",
|
||||
"...": "…",
|
||||
"···": "…",
|
||||
"・・・": "…",
|
||||
"…": "...",
|
||||
"···": "...",
|
||||
"・・・": "...",
|
||||
"·": ",",
|
||||
"・": ",",
|
||||
"、": ",",
|
||||
"$": ".",
|
||||
"“": "'",
|
||||
"”": "'",
|
||||
'"': "'",
|
||||
"‘": "'",
|
||||
"’": "'",
|
||||
"（": "'",
|
||||
@@ -317,6 +318,7 @@ def text_normalize(text):
|
||||
res = japanese_convert_numbers_to_words(res)
|
||||
# res = "".join([i for i in res if is_japanese_character(i)])
|
||||
res = replace_punctuation(res)
|
||||
res = res.replace("゙", "")
|
||||
return res
|
||||
|
||||
|
||||
@@ -340,7 +342,7 @@ def handle_long(sep_phonemes):
|
||||
return sep_phonemes
|
||||
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/deberta-v2-large-japanese")
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/deberta-v2-large-japanese-char-wwm")
|
||||
|
||||
|
||||
def align_tones(phones, tones):
|
||||
@@ -365,9 +367,35 @@ def align_tones(phones, tones):
|
||||
return res
|
||||
|
||||
|
||||
def rearrange_tones(tones, phones):
|
||||
res = [0] * len(tones)
|
||||
for i in range(len(tones)):
|
||||
if i == 0:
|
||||
if tones[i] not in punctuation:
|
||||
res[i] = 1
|
||||
elif tones[i] == prev:
|
||||
if phones[i] in punctuation:
|
||||
res[i] = 0
|
||||
else:
|
||||
res[i] = 1
|
||||
elif tones[i] > prev:
|
||||
res[i] = 2
|
||||
elif tones[i] < prev:
|
||||
res[i - 1] = 3
|
||||
res[i] = 1
|
||||
prev = tones[i]
|
||||
return res
|
||||
|
||||
|
||||
def g2p(norm_text):
|
||||
sep_text, sep_kata, acc = text2sep_kata(norm_text)
|
||||
sep_tokenized = [tokenizer.tokenize(i) for i in sep_text]
|
||||
sep_tokenized = []
|
||||
for i in sep_text:
|
||||
if i not in punctuation:
|
||||
sep_tokenized.append(tokenizer.tokenize(i))
|
||||
else:
|
||||
sep_tokenized.append([i])
|
||||
|
||||
sep_phonemes = handle_long([kata2phoneme(i) for i in sep_kata])
|
||||
# 异常处理，MeCab不认识的词的话会一路传到这里来，然后炸掉。目前来看只有那些超级稀有的生僻词会出现这种情况
|
||||
for i in sep_phonemes:
|
||||
@@ -383,6 +411,7 @@ def g2p(norm_text):
|
||||
aaa = distribute_phone(phone_len, word_len)
|
||||
word2ph += aaa
|
||||
phones = ["_"] + [j for i in sep_phonemes for j in i] + ["_"]
|
||||
# tones = [0] + rearrange_tones(tones, phones[1:-1]) + [0]
|
||||
tones = [0] + tones + [0]
|
||||
word2ph = [1] + word2ph + [1]
|
||||
assert len(phones) == len(tones)
|
||||
|
||||
@@ -6,7 +6,7 @@ from transformers import AutoModelForMaskedLM, AutoTokenizer
|
||||
from config import config
|
||||
from text.japanese import text2sep_kata
|
||||
|
||||
LOCAL_PATH = "./bert/deberta-v2-large-japanese"
|
||||
LOCAL_PATH = "./bert/deberta-v2-large-japanese-char-wwm"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(LOCAL_PATH)
|
||||
|
||||
@@ -14,14 +14,7 @@ models = dict()
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph, device=config.bert_gen_config.device):
|
||||
sep_text, _, _ = text2sep_kata(text)
|
||||
sep_tokens = [tokenizer.tokenize(t) for t in sep_text]
|
||||
sep_ids = [tokenizer.convert_tokens_to_ids(t) for t in sep_tokens]
|
||||
sep_ids = [2] + [item for sublist in sep_ids for item in sublist] + [3]
|
||||
return get_bert_feature_with_token(sep_ids, word2ph, device)
|
||||
|
||||
|
||||
def get_bert_feature_with_token(tokens, word2ph, device=config.bert_gen_config.device):
|
||||
text = "".join(text2sep_kata(text)[0])
|
||||
if (
|
||||
sys.platform == "darwin"
|
||||
and torch.backends.mps.is_available()
|
||||
@@ -33,20 +26,13 @@ def get_bert_feature_with_token(tokens, word2ph, device=config.bert_gen_config.d
|
||||
if device not in models.keys():
|
||||
models[device] = AutoModelForMaskedLM.from_pretrained(LOCAL_PATH).to(device)
|
||||
with torch.no_grad():
|
||||
inputs = torch.tensor(tokens).to(device).unsqueeze(0)
|
||||
token_type_ids = torch.zeros_like(inputs).to(device)
|
||||
attention_mask = torch.ones_like(inputs).to(device)
|
||||
inputs = {
|
||||
"input_ids": inputs,
|
||||
"token_type_ids": token_type_ids,
|
||||
"attention_mask": attention_mask,
|
||||
}
|
||||
|
||||
# for i in inputs:
|
||||
# inputs[i] = inputs[i].to(device)
|
||||
inputs = tokenizer(text, return_tensors="pt")
|
||||
for i in inputs:
|
||||
inputs[i] = inputs[i].to(device)
|
||||
res = models[device](**inputs, output_hidden_states=True)
|
||||
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
|
||||
assert inputs["input_ids"].shape[-1] == len(word2ph)
|
||||
|
||||
assert len(word2ph) == len(text) + 2
|
||||
word2phone = word2ph
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
|
||||
Reference in New Issue
Block a user