Dev emo (#171)
* SYNC CHANGE TO EMO BRANCH (#162) * Update README.md * 更新 bert_models.json * fix * Update data_utils.py * Update infer.py * performance improve * Feat: support auto split in webui (#158) * Feat: support auto split in webui * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix: change /voice api to post (#160) * Fix: change /voice api to post * Fix: support /voice api get * Fix: Add missing torch.cuda.empty_cache() (#161) --------- Co-authored-by: Sora <atri@suzakuintsubaki.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Artrajz <969242373@qq.com> * sync (#163) * Update README.md * 更新 bert_models.json * fix * Update data_utils.py * Update infer.py * performance improve * Feat: support auto split in webui (#158) * Feat: support auto split in webui * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix: change /voice api to post (#160) * Fix: change /voice api to post * Fix: support /voice api get * Fix: Add missing torch.cuda.empty_cache() (#161) * del emo * del emo * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Sora <atri@suzakuintsubaki.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Artrajz <969242373@qq.com> * Add files via upload * Update infer.py * add emo * add emo * Update default_config.yml * Fix slice segments GPU perf (#165) * Fix slice segments GPU perf * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update commons.py --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update infer.py * Update models.py * Update infer.py * remove spec cache * Update data_utils.py * Update data_utils.py * Update train_ms.py * Revert "Fix slice segments GPU perf (#165)" (#169) This reverts commit 28430fc76bc628297bb59d8f8d25100dbe46ab59. * Update train_ms.py * Update train_ms.py * Update data_utils.py * Update data_utils.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update default_config.yml * Switch to Japanese wwm DeBERTa (#172) * Switch to Japanese wwm DeBERTa * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix wrong ellipsis g2p (#173) * Switch to Japanese wwm DeBERTa * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix ellipsis g2p * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add files via upload * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix English phones not aligned with BERT features (#174) * Fix English phones not aligned with BERT features * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix english bert gen (#175) * Update webui.py * Update webui.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add NCCL timeout * Update train_ms.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update train_ms.py * Update default_config.yml * Update infer.py * Update models.py * Update train_ms.py * Update infer.py * Update emo_gen.py * Feat: Support load and infer 2.0 models (#178) * Feat: Support load and infer 2.0 models * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * 复用相同逻辑，修正静音添加错误 (#181) * Refactor: reuse the same part of voice api. * Fix: server_fastapi.py * Update train_ms.py * Update data_utils.py * Update data_utils.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update data_utils.py * Update data_utils.py * Add files via upload * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update default_config.yml * Update utils.py * Update train_ms.py * Update utils.py * Update default_config.yml * Update data_utils.py * Update default_config.yml * Update train_ms.py * Update train_ms.py * Update config.py * Update utils.py * Update train_ms.py * Update train_ms.py * feat: add voice mix and tone mix (#187) * feat: add voice mix and tone mix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Stardust·减 <star_dust_chen@foxmail.com> * Add files via upload * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Sora <atri@suzakuintsubaki.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Artrajz <969242373@qq.com> Co-authored-by: Leng Yue <lengyue@lengyue.me> Co-authored-by: OedoSoldier <31711261+OedoSoldier@users.noreply.github.com> Co-authored-by: 潮幻Mark <sunyunfei201@gmail.com>
This commit is contained in:
Stardust·减
34
bert/deberta-v2-large-japanese-char-wwm/.gitattributes
vendored
Normal file
34
bert/deberta-v2-large-japanese-char-wwm/.gitattributes
vendored
Normal file
@@ -0,0 +1,34 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
89
bert/deberta-v2-large-japanese-char-wwm/README.md
Normal file
89
bert/deberta-v2-large-japanese-char-wwm/README.md
Normal file
@@ -0,0 +1,89 @@
|
||||
---
|
||||
language: ja
|
||||
license: cc-by-sa-4.0
|
||||
library_name: transformers
|
||||
tags:
|
||||
- deberta
|
||||
- deberta-v2
|
||||
- fill-mask
|
||||
- character
|
||||
- wwm
|
||||
datasets:
|
||||
- wikipedia
|
||||
- cc100
|
||||
- oscar
|
||||
metrics:
|
||||
- accuracy
|
||||
mask_token: "[MASK]"
|
||||
widget:
|
||||
- text: "京都大学で自然言語処理を[MASK][MASK]する。"
|
||||
---
|
||||
|
||||
# Model Card for Japanese character-level DeBERTa V2 large
|
||||
|
||||
## Model description
|
||||
|
||||
This is a Japanese DeBERTa V2 large model pre-trained on Japanese Wikipedia, the Japanese portion of CC-100, and the Japanese portion of OSCAR.
|
||||
This model is trained with character-level tokenization and whole word masking.
|
||||
|
||||
## How to use
|
||||
|
||||
You can use this model for masked language modeling as follows:
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer, AutoModelForMaskedLM
|
||||
tokenizer = AutoTokenizer.from_pretrained('ku-nlp/deberta-v2-large-japanese-char-wwm')
|
||||
model = AutoModelForMaskedLM.from_pretrained('ku-nlp/deberta-v2-large-japanese-char-wwm')
|
||||
|
||||
sentence = '京都大学で自然言語処理を[MASK][MASK]する。'
|
||||
encoding = tokenizer(sentence, return_tensors='pt')
|
||||
...
|
||||
```
|
||||
|
||||
You can also fine-tune this model on downstream tasks.
|
||||
|
||||
## Tokenization
|
||||
|
||||
There is no need to tokenize texts in advance, and you can give raw texts to the tokenizer.
|
||||
The texts are tokenized into character-level tokens by [sentencepiece](https://github.com/google/sentencepiece).
|
||||
|
||||
## Training data
|
||||
|
||||
We used the following corpora for pre-training:
|
||||
|
||||
- Japanese Wikipedia (as of 20221020, 3.2GB, 27M sentences, 1.3M documents)
|
||||
- Japanese portion of CC-100 (85GB, 619M sentences, 66M documents)
|
||||
- Japanese portion of OSCAR (54GB, 326M sentences, 25M documents)
|
||||
|
||||
Note that we filtered out documents annotated with "header", "footer", or "noisy" tags in OSCAR.
|
||||
Also note that Japanese Wikipedia was duplicated 10 times to make the total size of the corpus comparable to that of CC-100 and OSCAR. As a result, the total size of the training data is 171GB.
|
||||
|
||||
## Training procedure
|
||||
|
||||
We first segmented texts in the corpora into words using [Juman++ 2.0.0-rc3](https://github.com/ku-nlp/jumanpp/releases/tag/v2.0.0-rc3) for whole word masking.
|
||||
Then, we built a sentencepiece model with 22,012 tokens including all characters that appear in the training corpus.
|
||||
|
||||
We tokenized raw corpora into character-level subwords using the sentencepiece model and trained the Japanese DeBERTa model using [transformers](https://github.com/huggingface/transformers) library.
|
||||
The training took 26 days using 16 NVIDIA A100-SXM4-40GB GPUs.
|
||||
|
||||
The following hyperparameters were used during pre-training:
|
||||
|
||||
- learning_rate: 1e-4
|
||||
- per_device_train_batch_size: 26
|
||||
- distributed_type: multi-GPU
|
||||
- num_devices: 16
|
||||
- gradient_accumulation_steps: 8
|
||||
- total_train_batch_size: 3,328
|
||||
- max_seq_length: 512
|
||||
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-06
|
||||
- lr_scheduler_type: linear schedule with warmup (lr = 0 at 300k steps)
|
||||
- training_steps: 260,000
|
||||
- warmup_steps: 10,000
|
||||
|
||||
The accuracy of the trained model on the masked language modeling task was 0.795.
|
||||
The evaluation set consists of 5,000 randomly sampled documents from each of the training corpora.
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
This work was supported by Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures (JHPCN) through General Collaboration Project no. jh221004, "Developing a Platform for Constructing and Sharing of Large-Scale Japanese Language Models".
|
||||
For training models, we used the mdx: a platform for the data-driven future.
|
||||
37
bert/deberta-v2-large-japanese-char-wwm/config.json
Normal file
37
bert/deberta-v2-large-japanese-char-wwm/config.json
Normal file
@@ -0,0 +1,37 @@
|
||||
{
|
||||
"architectures": [
|
||||
"DebertaV2ForMaskedLM"
|
||||
],
|
||||
"attention_head_size": 64,
|
||||
"attention_probs_dropout_prob": 0.1,
|
||||
"conv_act": "gelu",
|
||||
"conv_kernel_size": 3,
|
||||
"hidden_act": "gelu",
|
||||
"hidden_dropout_prob": 0.1,
|
||||
"hidden_size": 1024,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 4096,
|
||||
"layer_norm_eps": 1e-07,
|
||||
"max_position_embeddings": 512,
|
||||
"max_relative_positions": -1,
|
||||
"model_type": "deberta-v2",
|
||||
"norm_rel_ebd": "layer_norm",
|
||||
"num_attention_heads": 16,
|
||||
"num_hidden_layers": 24,
|
||||
"pad_token_id": 0,
|
||||
"pooler_dropout": 0,
|
||||
"pooler_hidden_act": "gelu",
|
||||
"pooler_hidden_size": 1024,
|
||||
"pos_att_type": [
|
||||
"p2c",
|
||||
"c2p"
|
||||
],
|
||||
"position_biased_input": false,
|
||||
"position_buckets": 256,
|
||||
"relative_attention": true,
|
||||
"share_att_key": true,
|
||||
"torch_dtype": "float16",
|
||||
"transformers_version": "4.25.1",
|
||||
"type_vocab_size": 0,
|
||||
"vocab_size": 22012
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"cls_token": "[CLS]",
|
||||
"mask_token": "[MASK]",
|
||||
"pad_token": "[PAD]",
|
||||
"sep_token": "[SEP]",
|
||||
"unk_token": "[UNK]"
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"cls_token": "[CLS]",
|
||||
"do_lower_case": false,
|
||||
"do_subword_tokenize": true,
|
||||
"do_word_tokenize": true,
|
||||
"jumanpp_kwargs": null,
|
||||
"mask_token": "[MASK]",
|
||||
"mecab_kwargs": null,
|
||||
"model_max_length": 1000000000000000019884624838656,
|
||||
"never_split": null,
|
||||
"pad_token": "[PAD]",
|
||||
"sep_token": "[SEP]",
|
||||
"special_tokens_map_file": null,
|
||||
"subword_tokenizer_type": "character",
|
||||
"sudachi_kwargs": null,
|
||||
"tokenizer_class": "BertJapaneseTokenizer",
|
||||
"unk_token": "[UNK]",
|
||||
"word_tokenizer_type": "basic"
|
||||
}
|
||||
22012
bert/deberta-v2-large-japanese-char-wwm/vocab.txt
Normal file
22012
bert/deberta-v2-large-japanese-char-wwm/vocab.txt
Normal file
File diff suppressed because it is too large
Load Diff
@@ -120,11 +120,15 @@ class Train_ms_config:
|
||||
env: Dict[str, any],
|
||||
base: Dict[str, any],
|
||||
model: str,
|
||||
num_workers: int,
|
||||
spec_cache: bool,
|
||||
):
|
||||
self.env = env # 需要加载的环境变量
|
||||
self.base = base # 底模配置
|
||||
self.model = model # 训练模型存储目录，该路径为相对于dataset_path的路径，而非项目根目录
|
||||
self.config_path = config_path # 配置文件路径
|
||||
self.num_workers = num_workers # worker数量
|
||||
self.spec_cache = spec_cache # 是否启用spec缓存
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
|
||||
|
||||
@@ -3,11 +3,13 @@ import random
|
||||
import torch
|
||||
import torch.utils.data
|
||||
from tqdm import tqdm
|
||||
import numpy as np
|
||||
from tools.log import logger
|
||||
import commons
|
||||
from mel_processing import spectrogram_torch, mel_spectrogram_torch
|
||||
from utils import load_wav_to_torch, load_filepaths_and_text
|
||||
from text import cleaned_text_to_sequence
|
||||
import config as config
|
||||
|
||||
"""Multi speaker version"""
|
||||
|
||||
@@ -40,7 +42,7 @@ class TextAudioSpeakerLoader(torch.utils.data.Dataset):
|
||||
|
||||
self.add_blank = hparams.add_blank
|
||||
self.min_text_len = getattr(hparams, "min_text_len", 1)
|
||||
self.max_text_len = getattr(hparams, "max_text_len", 300)
|
||||
self.max_text_len = getattr(hparams, "max_text_len", 384)
|
||||
|
||||
random.seed(1234)
|
||||
random.shuffle(self.audiopaths_sid_text)
|
||||
@@ -91,7 +93,8 @@ class TextAudioSpeakerLoader(torch.utils.data.Dataset):
|
||||
|
||||
spec, wav = self.get_audio(audiopath)
|
||||
sid = torch.LongTensor([int(self.spk_map[sid])])
|
||||
return (phones, spec, wav, sid, tone, language, bert, ja_bert, en_bert)
|
||||
emo = torch.FloatTensor(np.load(audiopath.replace(".wav", ".emo.npy")))
|
||||
return (phones, spec, wav, sid, tone, language, bert, ja_bert, en_bert, emo)
|
||||
|
||||
def get_audio(self, filename):
|
||||
audio, sampling_rate = load_wav_to_torch(filename)
|
||||
@@ -131,7 +134,8 @@ class TextAudioSpeakerLoader(torch.utils.data.Dataset):
|
||||
center=False,
|
||||
)
|
||||
spec = torch.squeeze(spec, 0)
|
||||
torch.save(spec, spec_filename)
|
||||
if config.train_ms.spec_cache:
|
||||
torch.save(spec, spec_filename)
|
||||
return spec, audio_norm
|
||||
|
||||
def get_text(self, text, word2ph, phone, tone, language_str, wav_path):
|
||||
@@ -211,6 +215,7 @@ class TextAudioSpeakerCollate:
|
||||
bert_padded = torch.FloatTensor(len(batch), 1024, max_text_len)
|
||||
ja_bert_padded = torch.FloatTensor(len(batch), 1024, max_text_len)
|
||||
en_bert_padded = torch.FloatTensor(len(batch), 1024, max_text_len)
|
||||
emo = torch.FloatTensor(len(batch), 1024)
|
||||
|
||||
spec_padded = torch.FloatTensor(len(batch), batch[0][1].size(0), max_spec_len)
|
||||
wav_padded = torch.FloatTensor(len(batch), 1, max_wav_len)
|
||||
@@ -222,6 +227,7 @@ class TextAudioSpeakerCollate:
|
||||
bert_padded.zero_()
|
||||
ja_bert_padded.zero_()
|
||||
en_bert_padded.zero_()
|
||||
emo.zero_()
|
||||
|
||||
for i in range(len(ids_sorted_decreasing)):
|
||||
row = batch[ids_sorted_decreasing[i]]
|
||||
@@ -255,6 +261,8 @@ class TextAudioSpeakerCollate:
|
||||
en_bert = row[8]
|
||||
en_bert_padded[i, :, : en_bert.size(1)] = en_bert
|
||||
|
||||
emo[i, :] = row[9]
|
||||
|
||||
return (
|
||||
text_padded,
|
||||
text_lengths,
|
||||
@@ -268,6 +276,7 @@ class TextAudioSpeakerCollate:
|
||||
bert_padded,
|
||||
ja_bert_padded,
|
||||
en_bert_padded,
|
||||
emo,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -56,12 +56,19 @@ bert_gen:
|
||||
# 使用多卡推理
|
||||
use_multi_device: false
|
||||
|
||||
# emo_gen 相关配置
|
||||
# 注意， “:” 后需要加空格
|
||||
emo_gen:
|
||||
# 训练数据集配置文件路径
|
||||
config_path: "config.json"
|
||||
# 并行数
|
||||
num_processes: 2
|
||||
# 使用设备：可选项 "cuda" 显卡推理，"cpu" cpu推理
|
||||
device: "cuda"
|
||||
|
||||
# train 训练配置
|
||||
# 注意， “:” 后需要加空格
|
||||
train_ms:
|
||||
# 需要加载的环境变量，多显卡训练时RANK请手动在环境变量填写
|
||||
# 环境变量对应名称环境变量不存在时加载，也就是说手动添加的环境变量优先级更高，会覆盖本配置文件
|
||||
env:
|
||||
MASTER_ADDR: "localhost"
|
||||
MASTER_PORT: 10086
|
||||
@@ -79,6 +86,10 @@ train_ms:
|
||||
model: "models"
|
||||
# 配置文件路径
|
||||
config_path: "configs/config.json"
|
||||
# 训练使用的worker，不建议超过CPU核心数
|
||||
num_workers: 16
|
||||
# 关闭此项可以节约接近50%的磁盘空间，但是可能导致实际训练速度变慢和更高的CPU使用率。
|
||||
spec_cache: True
|
||||
|
||||
|
||||
# webui webui配置
|
||||
|
||||
@@ -111,7 +111,7 @@ def get_emo(path):
|
||||
wav, sr = librosa.load(path, 16000)
|
||||
device = config.bert_gen_config.device
|
||||
return process_func(
|
||||
np.expand_dims(wav, 0).astype(np.float),
|
||||
np.expand_dims(wav, 0).astype(np.float64),
|
||||
sr,
|
||||
model,
|
||||
processor,
|
||||
|
||||
38
infer.py
38
infer.py
@@ -11,11 +11,14 @@
|
||||
import torch
|
||||
import commons
|
||||
from text import cleaned_text_to_sequence, get_bert
|
||||
from emo_gen import get_emo
|
||||
from text.cleaner import clean_text
|
||||
import utils
|
||||
|
||||
from models import SynthesizerTrn
|
||||
from text.symbols import symbols
|
||||
from oldVersion.V200.models import SynthesizerTrn as V200SynthesizerTrn
|
||||
from oldVersion.V200.text import symbols as V200symbols
|
||||
from oldVersion.V111.models import SynthesizerTrn as V111SynthesizerTrn
|
||||
from oldVersion.V111.text import symbols as V111symbols
|
||||
from oldVersion.V110.models import SynthesizerTrn as V110SynthesizerTrn
|
||||
@@ -23,13 +26,16 @@ from oldVersion.V110.text import symbols as V110symbols
|
||||
from oldVersion.V101.models import SynthesizerTrn as V101SynthesizerTrn
|
||||
from oldVersion.V101.text import symbols as V101symbols
|
||||
|
||||
from oldVersion import V111, V110, V101
|
||||
from oldVersion import V111, V110, V101, V200
|
||||
|
||||
# 当前版本信息
|
||||
latest_version = "2.0"
|
||||
latest_version = "2.1"
|
||||
|
||||
# 版本兼容
|
||||
SynthesizerTrnMap = {
|
||||
"2.0.2-fix": V200SynthesizerTrn,
|
||||
"2.0.1": V200SynthesizerTrn,
|
||||
"2.0": V200SynthesizerTrn,
|
||||
"1.1.1-fix": V111SynthesizerTrn,
|
||||
"1.1.1": V111SynthesizerTrn,
|
||||
"1.1": V110SynthesizerTrn,
|
||||
@@ -40,6 +46,9 @@ SynthesizerTrnMap = {
|
||||
}
|
||||
|
||||
symbolsMap = {
|
||||
"2.0.2-fix": V200symbols,
|
||||
"2.0.1": V200symbols,
|
||||
"2.0": V200symbols,
|
||||
"1.1.1-fix": V111symbols,
|
||||
"1.1.1": V111symbols,
|
||||
"1.1": V110symbols,
|
||||
@@ -73,7 +82,7 @@ def get_net_g(model_path: str, version: str, device: str, hps):
|
||||
return net_g
|
||||
|
||||
|
||||
def get_text(text, language_str, hps, device):
|
||||
def get_text(text, reference_audio, emotion, language_str, hps, device):
|
||||
# 在此处实现当前版本的get_text
|
||||
norm_text, phone, tone, word2ph = clean_text(text, language_str)
|
||||
phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
|
||||
@@ -104,6 +113,12 @@ def get_text(text, language_str, hps, device):
|
||||
else:
|
||||
raise ValueError("language_str should be ZH, JP or EN")
|
||||
|
||||
emo = (
|
||||
torch.from_numpy(get_emo(reference_audio))
|
||||
if reference_audio
|
||||
else torch.Tensor([emotion])
|
||||
)
|
||||
|
||||
assert bert.shape[-1] == len(
|
||||
phone
|
||||
), f"Bert seq len {bert.shape[-1]} != {len(phone)}"
|
||||
@@ -111,7 +126,7 @@ def get_text(text, language_str, hps, device):
|
||||
phone = torch.LongTensor(phone)
|
||||
tone = torch.LongTensor(tone)
|
||||
language = torch.LongTensor(language)
|
||||
return bert, ja_bert, en_bert, phone, tone, language
|
||||
return bert, ja_bert, en_bert, emo, phone, tone, language
|
||||
|
||||
|
||||
def infer(
|
||||
@@ -125,11 +140,16 @@ def infer(
|
||||
hps,
|
||||
net_g,
|
||||
device,
|
||||
reference_audio=None,
|
||||
emotion=None,
|
||||
skip_start=False,
|
||||
skip_end=False,
|
||||
):
|
||||
# 支持中日双语版本
|
||||
# 支持中日英三语版本
|
||||
inferMap_V2 = {
|
||||
"2.0.2-fix": V200.infer,
|
||||
"2.0.1": V200.infer,
|
||||
"2.0": V200.infer,
|
||||
"1.1.1-fix": V111.infer_fix,
|
||||
"1.1.1": V111.infer,
|
||||
"1.1": V110.infer,
|
||||
@@ -171,8 +191,8 @@ def infer(
|
||||
device,
|
||||
)
|
||||
# 在此处实现当前版本的推理
|
||||
bert, ja_bert, en_bert, phones, tones, lang_ids = get_text(
|
||||
text, language, hps, device
|
||||
bert, ja_bert, en_bert, emo, phones, tones, lang_ids = get_text(
|
||||
text, reference_audio, emotion, language, hps, device
|
||||
)
|
||||
if skip_start:
|
||||
phones = phones[1:]
|
||||
@@ -285,6 +305,7 @@ def infer_multilang(
|
||||
bert = bert.to(device).unsqueeze(0)
|
||||
ja_bert = ja_bert.to(device).unsqueeze(0)
|
||||
en_bert = en_bert.to(device).unsqueeze(0)
|
||||
emo = emo.to(device).unsqueeze(0)
|
||||
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
|
||||
del phones
|
||||
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
|
||||
@@ -298,6 +319,7 @@ def infer_multilang(
|
||||
bert,
|
||||
ja_bert,
|
||||
en_bert,
|
||||
emo,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise_scale,
|
||||
noise_scale_w=noise_scale_w,
|
||||
@@ -307,7 +329,7 @@ def infer_multilang(
|
||||
.float()
|
||||
.numpy()
|
||||
)
|
||||
del x_tst, tones, lang_ids, bert, x_tst_lengths, speakers, ja_bert, en_bert
|
||||
del x_tst, tones, lang_ids, bert, x_tst_lengths, speakers, ja_bert, en_bert, emo
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
return audio
|
||||
|
||||
52
models.py
52
models.py
@@ -10,6 +10,8 @@ import monotonic_align
|
||||
|
||||
from torch.nn import Conv1d, ConvTranspose1d, Conv2d
|
||||
from torch.nn.utils import weight_norm, remove_weight_norm, spectral_norm
|
||||
from vector_quantize_pytorch import VectorQuantize
|
||||
|
||||
from commons import init_weights, get_padding
|
||||
from text import symbols, num_tones, num_languages
|
||||
|
||||
@@ -320,6 +322,7 @@ class TextEncoder(nn.Module):
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
n_speakers,
|
||||
gin_channels=0,
|
||||
):
|
||||
super().__init__()
|
||||
@@ -341,6 +344,18 @@ class TextEncoder(nn.Module):
|
||||
self.bert_proj = nn.Conv1d(1024, hidden_channels, 1)
|
||||
self.ja_bert_proj = nn.Conv1d(1024, hidden_channels, 1)
|
||||
self.en_bert_proj = nn.Conv1d(1024, hidden_channels, 1)
|
||||
self.emo_proj = nn.Linear(1024, 1024)
|
||||
self.emo_quantizer = [
|
||||
VectorQuantize(
|
||||
dim=1024,
|
||||
codebook_size=10,
|
||||
decay=0.8,
|
||||
commitment_weight=1.0,
|
||||
learnable_codebook=True,
|
||||
ema_update=False,
|
||||
)
|
||||
] * n_speakers
|
||||
self.emo_q_proj = nn.Linear(1024, hidden_channels)
|
||||
|
||||
self.encoder = attentions.Encoder(
|
||||
hidden_channels,
|
||||
@@ -354,11 +369,32 @@ class TextEncoder(nn.Module):
|
||||
self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
|
||||
|
||||
def forward(
|
||||
self, x, x_lengths, tone, language, bert, ja_bert, en_bert, sid, g=None
|
||||
self, x, x_lengths, tone, language, bert, ja_bert, en_bert, emo, sid, g=None
|
||||
):
|
||||
sid = sid.cpu()
|
||||
bert_emb = self.bert_proj(bert).transpose(1, 2)
|
||||
ja_bert_emb = self.ja_bert_proj(ja_bert).transpose(1, 2)
|
||||
en_bert_emb = self.en_bert_proj(en_bert).transpose(1, 2)
|
||||
if emo.size(-1) == 1024:
|
||||
emo_emb = self.emo_proj(emo.unsqueeze(1))
|
||||
emo_commit_loss = torch.zeros(1)
|
||||
emo_emb_ = []
|
||||
for i in range(emo_emb.size(0)):
|
||||
temp_emo_emb, _, temp_emo_commit_loss = self.emo_quantizer[sid[i]](
|
||||
emo_emb[i].unsqueeze(0).cpu()
|
||||
)
|
||||
emo_commit_loss += temp_emo_commit_loss
|
||||
emo_emb_.append(temp_emo_emb)
|
||||
emo_emb = torch.cat(emo_emb_, dim=0).to(emo_emb.device)
|
||||
emo_commit_loss = emo_commit_loss.to(emo_emb.device)
|
||||
else:
|
||||
emo_emb = (
|
||||
self.emo_quantizer[sid[0]]
|
||||
.get_output_from_indices(emo.to(torch.int).cpu())
|
||||
.unsqueeze(0)
|
||||
.to(emo.device)
|
||||
)
|
||||
emo_commit_loss = torch.zeros(1)
|
||||
x = (
|
||||
self.emb(x)
|
||||
+ self.tone_emb(tone)
|
||||
@@ -366,6 +402,7 @@ class TextEncoder(nn.Module):
|
||||
+ bert_emb
|
||||
+ ja_bert_emb
|
||||
+ en_bert_emb
|
||||
+ self.emo_q_proj(emo_emb)
|
||||
) * math.sqrt(
|
||||
self.hidden_channels
|
||||
) # [b, t, h]
|
||||
@@ -378,7 +415,7 @@ class TextEncoder(nn.Module):
|
||||
stats = self.proj(x) * x_mask
|
||||
|
||||
m, logs = torch.split(stats, self.out_channels, dim=1)
|
||||
return x, m, logs, x_mask
|
||||
return x, m, logs, x_mask, emo_commit_loss
|
||||
|
||||
|
||||
class ResidualCouplingBlock(nn.Module):
|
||||
@@ -811,6 +848,7 @@ class SynthesizerTrn(nn.Module):
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
self.n_speakers,
|
||||
gin_channels=self.enc_gin_channels,
|
||||
)
|
||||
self.dec = Generator(
|
||||
@@ -884,8 +922,8 @@ class SynthesizerTrn(nn.Module):
|
||||
g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
|
||||
else:
|
||||
g = self.ref_enc(y.transpose(1, 2)).unsqueeze(-1)
|
||||
x, m_p, logs_p, x_mask = self.enc_p(
|
||||
x, x_lengths, tone, language, bert, ja_bert, en_bert, sid, g=g
|
||||
x, m_p, logs_p, x_mask, loss_commit = self.enc_p(
|
||||
x, x_lengths, tone, language, bert, ja_bert, en_bert, emo, sid, g=g
|
||||
)
|
||||
z, m_q, logs_q, y_mask = self.enc_q(y, y_lengths, g=g)
|
||||
z_p = self.flow(z, y_mask, g=g)
|
||||
@@ -951,6 +989,7 @@ class SynthesizerTrn(nn.Module):
|
||||
y_mask,
|
||||
(z, z_p, m_p, logs_p, m_q, logs_q),
|
||||
(x, logw, logw_),
|
||||
loss_commit,
|
||||
)
|
||||
|
||||
def infer(
|
||||
@@ -963,6 +1002,7 @@ class SynthesizerTrn(nn.Module):
|
||||
bert,
|
||||
ja_bert,
|
||||
en_bert,
|
||||
emo=None,
|
||||
noise_scale=0.667,
|
||||
length_scale=1,
|
||||
noise_scale_w=0.8,
|
||||
@@ -976,8 +1016,8 @@ class SynthesizerTrn(nn.Module):
|
||||
g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
|
||||
else:
|
||||
g = self.ref_enc(y.transpose(1, 2)).unsqueeze(-1)
|
||||
x, m_p, logs_p, x_mask = self.enc_p(
|
||||
x, x_lengths, tone, language, bert, ja_bert, en_bert, sid, g=g
|
||||
x, m_p, logs_p, x_mask, _ = self.enc_p(
|
||||
x, x_lengths, tone, language, bert, ja_bert, en_bert, emo, sid, g=g
|
||||
)
|
||||
logw = self.sdp(x, x_mask, g=g, reverse=True, noise_scale=noise_scale_w) * (
|
||||
sdp_ratio
|
||||
|
||||
98
oldVersion/V200/__init__.py
Normal file
98
oldVersion/V200/__init__.py
Normal file
@@ -0,0 +1,98 @@
|
||||
"""
|
||||
@Desc: 2.0版本兼容 对应2.0.1 2.0.2-fix
|
||||
"""
|
||||
import torch
|
||||
import commons
|
||||
from .text import cleaned_text_to_sequence, get_bert
|
||||
from .text.cleaner import clean_text
|
||||
|
||||
|
||||
def get_text(text, language_str, hps, device):
|
||||
# 在此处实现当前版本的get_text
|
||||
norm_text, phone, tone, word2ph = clean_text(text, language_str)
|
||||
phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
|
||||
|
||||
if hps.data.add_blank:
|
||||
phone = commons.intersperse(phone, 0)
|
||||
tone = commons.intersperse(tone, 0)
|
||||
language = commons.intersperse(language, 0)
|
||||
for i in range(len(word2ph)):
|
||||
word2ph[i] = word2ph[i] * 2
|
||||
word2ph[0] += 1
|
||||
bert_ori = get_bert(norm_text, word2ph, language_str, device)
|
||||
del word2ph
|
||||
assert bert_ori.shape[-1] == len(phone), phone
|
||||
|
||||
if language_str == "ZH":
|
||||
bert = bert_ori
|
||||
ja_bert = torch.zeros(1024, len(phone))
|
||||
en_bert = torch.zeros(1024, len(phone))
|
||||
elif language_str == "JP":
|
||||
bert = torch.zeros(1024, len(phone))
|
||||
ja_bert = bert_ori
|
||||
en_bert = torch.zeros(1024, len(phone))
|
||||
elif language_str == "EN":
|
||||
bert = torch.zeros(1024, len(phone))
|
||||
ja_bert = torch.zeros(1024, len(phone))
|
||||
en_bert = bert_ori
|
||||
else:
|
||||
raise ValueError("language_str should be ZH, JP or EN")
|
||||
|
||||
assert bert.shape[-1] == len(
|
||||
phone
|
||||
), f"Bert seq len {bert.shape[-1]} != {len(phone)}"
|
||||
|
||||
phone = torch.LongTensor(phone)
|
||||
tone = torch.LongTensor(tone)
|
||||
language = torch.LongTensor(language)
|
||||
return bert, ja_bert, en_bert, phone, tone, language
|
||||
|
||||
|
||||
def infer(
|
||||
text,
|
||||
sdp_ratio,
|
||||
noise_scale,
|
||||
noise_scale_w,
|
||||
length_scale,
|
||||
sid,
|
||||
language,
|
||||
hps,
|
||||
net_g,
|
||||
device,
|
||||
):
|
||||
bert, ja_bert, en_bert, phones, tones, lang_ids = get_text(
|
||||
text, language, hps, device
|
||||
)
|
||||
with torch.no_grad():
|
||||
x_tst = phones.to(device).unsqueeze(0)
|
||||
tones = tones.to(device).unsqueeze(0)
|
||||
lang_ids = lang_ids.to(device).unsqueeze(0)
|
||||
bert = bert.to(device).unsqueeze(0)
|
||||
ja_bert = ja_bert.to(device).unsqueeze(0)
|
||||
en_bert = en_bert.to(device).unsqueeze(0)
|
||||
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
|
||||
del phones
|
||||
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
|
||||
audio = (
|
||||
net_g.infer(
|
||||
x_tst,
|
||||
x_tst_lengths,
|
||||
speakers,
|
||||
tones,
|
||||
lang_ids,
|
||||
bert,
|
||||
ja_bert,
|
||||
en_bert,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise_scale,
|
||||
noise_scale_w=noise_scale_w,
|
||||
length_scale=length_scale,
|
||||
)[0][0, 0]
|
||||
.data.cpu()
|
||||
.float()
|
||||
.numpy()
|
||||
)
|
||||
del x_tst, tones, lang_ids, bert, x_tst_lengths, speakers, ja_bert, en_bert
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
return audio
|
||||
1004
oldVersion/V200/models.py
Normal file
1004
oldVersion/V200/models.py
Normal file
File diff suppressed because it is too large
Load Diff
48
oldVersion/V200/text/__init__.py
Normal file
48
oldVersion/V200/text/__init__.py
Normal file
@@ -0,0 +1,48 @@
|
||||
from .symbols import *
|
||||
|
||||
_symbol_to_id = {s: i for i, s in enumerate(symbols)}
|
||||
|
||||
|
||||
def cleaned_text_to_sequence(cleaned_text, tones, language):
|
||||
"""Converts a string of text to a sequence of IDs corresponding to the symbols in the text.
|
||||
Args:
|
||||
text: string to convert to a sequence
|
||||
Returns:
|
||||
List of integers corresponding to the symbols in the text
|
||||
"""
|
||||
phones = [_symbol_to_id[symbol] for symbol in cleaned_text]
|
||||
tone_start = language_tone_start_map[language]
|
||||
tones = [i + tone_start for i in tones]
|
||||
lang_id = language_id_map[language]
|
||||
lang_ids = [lang_id for i in phones]
|
||||
return phones, tones, lang_ids
|
||||
|
||||
|
||||
def get_bert(norm_text, word2ph, language, device):
|
||||
from .chinese_bert import get_bert_feature as zh_bert
|
||||
from .english_bert_mock import get_bert_feature as en_bert
|
||||
from .japanese_bert import get_bert_feature as jp_bert
|
||||
|
||||
lang_bert_func_map = {"ZH": zh_bert, "EN": en_bert, "JP": jp_bert}
|
||||
bert = lang_bert_func_map[language](norm_text, word2ph, device)
|
||||
return bert
|
||||
|
||||
|
||||
def check_bert_models():
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from config import config
|
||||
from .bert_utils import _check_bert
|
||||
|
||||
if config.mirror.lower() == "openi":
|
||||
import openi
|
||||
|
||||
kwargs = {"token": config.openi_token} if config.openi_token else {}
|
||||
openi.login(**kwargs)
|
||||
|
||||
with open("./bert/bert_models.json", "r") as fp:
|
||||
models = json.load(fp)
|
||||
for k, v in models.items():
|
||||
local_path = Path("./bert").joinpath(k)
|
||||
_check_bert(v["repo_id"], v["files"], local_path)
|
||||
23
oldVersion/V200/text/bert_utils.py
Normal file
23
oldVersion/V200/text/bert_utils.py
Normal file
@@ -0,0 +1,23 @@
|
||||
from pathlib import Path
|
||||
|
||||
from huggingface_hub import hf_hub_download
|
||||
|
||||
from config import config
|
||||
|
||||
|
||||
MIRROR: str = config.mirror
|
||||
|
||||
|
||||
def _check_bert(repo_id, files, local_path):
|
||||
for file in files:
|
||||
if not Path(local_path).joinpath(file).exists():
|
||||
if MIRROR.lower() == "openi":
|
||||
import openi
|
||||
|
||||
openi.model.download_model(
|
||||
"Stardust_minus/Bert-VITS2", repo_id.split("/")[-1], "./bert"
|
||||
)
|
||||
else:
|
||||
hf_hub_download(
|
||||
repo_id, file, local_dir=local_path, local_dir_use_symlinks=False
|
||||
)
|
||||
198
oldVersion/V200/text/chinese.py
Normal file
198
oldVersion/V200/text/chinese.py
Normal file
@@ -0,0 +1,198 @@
|
||||
import os
|
||||
import re
|
||||
|
||||
import cn2an
|
||||
from pypinyin import lazy_pinyin, Style
|
||||
|
||||
from .symbols import punctuation
|
||||
from .tone_sandhi import ToneSandhi
|
||||
|
||||
current_file_path = os.path.dirname(__file__)
|
||||
pinyin_to_symbol_map = {
|
||||
line.split("\t")[0]: line.strip().split("\t")[1]
|
||||
for line in open(os.path.join(current_file_path, "opencpop-strict.txt")).readlines()
|
||||
}
|
||||
|
||||
import jieba.posseg as psg
|
||||
|
||||
|
||||
rep_map = {
|
||||
"：": ",",
|
||||
"；": ",",
|
||||
"，": ",",
|
||||
"。": ".",
|
||||
"！": "!",
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"·": ",",
|
||||
"、": ",",
|
||||
"...": "…",
|
||||
"$": ".",
|
||||
"“": "'",
|
||||
"”": "'",
|
||||
"‘": "'",
|
||||
"’": "'",
|
||||
"（": "'",
|
||||
"）": "'",
|
||||
"(": "'",
|
||||
")": "'",
|
||||
"《": "'",
|
||||
"》": "'",
|
||||
"【": "'",
|
||||
"】": "'",
|
||||
"[": "'",
|
||||
"]": "'",
|
||||
"—": "-",
|
||||
"～": "-",
|
||||
"~": "-",
|
||||
"「": "'",
|
||||
"」": "'",
|
||||
}
|
||||
|
||||
tone_modifier = ToneSandhi()
|
||||
|
||||
|
||||
def replace_punctuation(text):
|
||||
text = text.replace("嗯", "恩").replace("呣", "母")
|
||||
pattern = re.compile("|".join(re.escape(p) for p in rep_map.keys()))
|
||||
|
||||
replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
|
||||
|
||||
replaced_text = re.sub(
|
||||
r"[^\u4e00-\u9fa5" + "".join(punctuation) + r"]+", "", replaced_text
|
||||
)
|
||||
|
||||
return replaced_text
|
||||
|
||||
|
||||
def g2p(text):
|
||||
pattern = r"(?<=[{0}])\s*".format("".join(punctuation))
|
||||
sentences = [i for i in re.split(pattern, text) if i.strip() != ""]
|
||||
phones, tones, word2ph = _g2p(sentences)
|
||||
assert sum(word2ph) == len(phones)
|
||||
assert len(word2ph) == len(text) # Sometimes it will crash,you can add a try-catch.
|
||||
phones = ["_"] + phones + ["_"]
|
||||
tones = [0] + tones + [0]
|
||||
word2ph = [1] + word2ph + [1]
|
||||
return phones, tones, word2ph
|
||||
|
||||
|
||||
def _get_initials_finals(word):
|
||||
initials = []
|
||||
finals = []
|
||||
orig_initials = lazy_pinyin(word, neutral_tone_with_five=True, style=Style.INITIALS)
|
||||
orig_finals = lazy_pinyin(
|
||||
word, neutral_tone_with_five=True, style=Style.FINALS_TONE3
|
||||
)
|
||||
for c, v in zip(orig_initials, orig_finals):
|
||||
initials.append(c)
|
||||
finals.append(v)
|
||||
return initials, finals
|
||||
|
||||
|
||||
def _g2p(segments):
|
||||
phones_list = []
|
||||
tones_list = []
|
||||
word2ph = []
|
||||
for seg in segments:
|
||||
# Replace all English words in the sentence
|
||||
seg = re.sub("[a-zA-Z]+", "", seg)
|
||||
seg_cut = psg.lcut(seg)
|
||||
initials = []
|
||||
finals = []
|
||||
seg_cut = tone_modifier.pre_merge_for_modify(seg_cut)
|
||||
for word, pos in seg_cut:
|
||||
if pos == "eng":
|
||||
continue
|
||||
sub_initials, sub_finals = _get_initials_finals(word)
|
||||
sub_finals = tone_modifier.modified_tone(word, pos, sub_finals)
|
||||
initials.append(sub_initials)
|
||||
finals.append(sub_finals)
|
||||
|
||||
# assert len(sub_initials) == len(sub_finals) == len(word)
|
||||
initials = sum(initials, [])
|
||||
finals = sum(finals, [])
|
||||
#
|
||||
for c, v in zip(initials, finals):
|
||||
raw_pinyin = c + v
|
||||
# NOTE: post process for pypinyin outputs
|
||||
# we discriminate i, ii and iii
|
||||
if c == v:
|
||||
assert c in punctuation
|
||||
phone = [c]
|
||||
tone = "0"
|
||||
word2ph.append(1)
|
||||
else:
|
||||
v_without_tone = v[:-1]
|
||||
tone = v[-1]
|
||||
|
||||
pinyin = c + v_without_tone
|
||||
assert tone in "12345"
|
||||
|
||||
if c:
|
||||
# 多音节
|
||||
v_rep_map = {
|
||||
"uei": "ui",
|
||||
"iou": "iu",
|
||||
"uen": "un",
|
||||
}
|
||||
if v_without_tone in v_rep_map.keys():
|
||||
pinyin = c + v_rep_map[v_without_tone]
|
||||
else:
|
||||
# 单音节
|
||||
pinyin_rep_map = {
|
||||
"ing": "ying",
|
||||
"i": "yi",
|
||||
"in": "yin",
|
||||
"u": "wu",
|
||||
}
|
||||
if pinyin in pinyin_rep_map.keys():
|
||||
pinyin = pinyin_rep_map[pinyin]
|
||||
else:
|
||||
single_rep_map = {
|
||||
"v": "yu",
|
||||
"e": "e",
|
||||
"i": "y",
|
||||
"u": "w",
|
||||
}
|
||||
if pinyin[0] in single_rep_map.keys():
|
||||
pinyin = single_rep_map[pinyin[0]] + pinyin[1:]
|
||||
|
||||
assert pinyin in pinyin_to_symbol_map.keys(), (pinyin, seg, raw_pinyin)
|
||||
phone = pinyin_to_symbol_map[pinyin].split(" ")
|
||||
word2ph.append(len(phone))
|
||||
|
||||
phones_list += phone
|
||||
tones_list += [int(tone)] * len(phone)
|
||||
return phones_list, tones_list, word2ph
|
||||
|
||||
|
||||
def text_normalize(text):
|
||||
numbers = re.findall(r"\d+(?:\.?\d+)?", text)
|
||||
for number in numbers:
|
||||
text = text.replace(number, cn2an.an2cn(number), 1)
|
||||
text = replace_punctuation(text)
|
||||
return text
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph):
|
||||
from text import chinese_bert
|
||||
|
||||
return chinese_bert.get_bert_feature(text, word2ph)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from text.chinese_bert import get_bert_feature
|
||||
|
||||
text = "啊！但是《原神》是由,米哈\游自主， [研发]的一款全.新开放世界.冒险游戏"
|
||||
text = text_normalize(text)
|
||||
print(text)
|
||||
phones, tones, word2ph = g2p(text)
|
||||
bert = get_bert_feature(text, word2ph)
|
||||
|
||||
print(phones, tones, word2ph, bert.shape)
|
||||
|
||||
|
||||
# # 示例用法
|
||||
# text = "这是一个示例文本：,你好！这是一个测试...."
|
||||
# print(g2p_paddle(text)) # 输出: 这是一个示例文本你好这是一个测试
|
||||
101
oldVersion/V200/text/chinese_bert.py
Normal file
101
oldVersion/V200/text/chinese_bert.py
Normal file
@@ -0,0 +1,101 @@
|
||||
import sys
|
||||
|
||||
import torch
|
||||
from transformers import AutoModelForMaskedLM, AutoTokenizer
|
||||
|
||||
from config import config
|
||||
|
||||
LOCAL_PATH = "./bert/chinese-roberta-wwm-ext-large"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(LOCAL_PATH)
|
||||
|
||||
models = dict()
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph, device=config.bert_gen_config.device):
|
||||
if (
|
||||
sys.platform == "darwin"
|
||||
and torch.backends.mps.is_available()
|
||||
and device == "cpu"
|
||||
):
|
||||
device = "mps"
|
||||
if not device:
|
||||
device = "cuda"
|
||||
if device not in models.keys():
|
||||
models[device] = AutoModelForMaskedLM.from_pretrained(LOCAL_PATH).to(device)
|
||||
with torch.no_grad():
|
||||
inputs = tokenizer(text, return_tensors="pt")
|
||||
for i in inputs:
|
||||
inputs[i] = inputs[i].to(device)
|
||||
res = models[device](**inputs, output_hidden_states=True)
|
||||
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
|
||||
|
||||
assert len(word2ph) == len(text) + 2
|
||||
word2phone = word2ph
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
repeat_feature = res[i].repeat(word2phone[i], 1)
|
||||
phone_level_feature.append(repeat_feature)
|
||||
|
||||
phone_level_feature = torch.cat(phone_level_feature, dim=0)
|
||||
|
||||
return phone_level_feature.T
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
word_level_feature = torch.rand(38, 1024) # 12个词,每个词1024维特征
|
||||
word2phone = [
|
||||
1,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
]
|
||||
|
||||
# 计算总帧数
|
||||
total_frames = sum(word2phone)
|
||||
print(word_level_feature.shape)
|
||||
print(word2phone)
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
print(word_level_feature[i].shape)
|
||||
|
||||
# 对每个词重复word2phone[i]次
|
||||
repeat_feature = word_level_feature[i].repeat(word2phone[i], 1)
|
||||
phone_level_feature.append(repeat_feature)
|
||||
|
||||
phone_level_feature = torch.cat(phone_level_feature, dim=0)
|
||||
print(phone_level_feature.shape) # torch.Size([36, 1024])
|
||||
28
oldVersion/V200/text/cleaner.py
Normal file
28
oldVersion/V200/text/cleaner.py
Normal file
@@ -0,0 +1,28 @@
|
||||
from . import chinese, japanese, english, cleaned_text_to_sequence
|
||||
|
||||
|
||||
language_module_map = {"ZH": chinese, "JP": japanese, "EN": english}
|
||||
|
||||
|
||||
def clean_text(text, language):
|
||||
language_module = language_module_map[language]
|
||||
norm_text = language_module.text_normalize(text)
|
||||
phones, tones, word2ph = language_module.g2p(norm_text)
|
||||
return norm_text, phones, tones, word2ph
|
||||
|
||||
|
||||
def clean_text_bert(text, language):
|
||||
language_module = language_module_map[language]
|
||||
norm_text = language_module.text_normalize(text)
|
||||
phones, tones, word2ph = language_module.g2p(norm_text)
|
||||
bert = language_module.get_bert_feature(norm_text, word2ph)
|
||||
return phones, tones, bert
|
||||
|
||||
|
||||
def text_to_sequence(text, language):
|
||||
norm_text, phones, tones, word2ph = clean_text(text, language)
|
||||
return cleaned_text_to_sequence(phones, tones, language)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pass
|
||||
129530
oldVersion/V200/text/cmudict.rep
Normal file
129530
oldVersion/V200/text/cmudict.rep
Normal file
File diff suppressed because it is too large
Load Diff
BIN
oldVersion/V200/text/cmudict_cache.pickle
Normal file
BIN
oldVersion/V200/text/cmudict_cache.pickle
Normal file
Binary file not shown.
362
oldVersion/V200/text/english.py
Normal file
362
oldVersion/V200/text/english.py
Normal file
@@ -0,0 +1,362 @@
|
||||
import pickle
|
||||
import os
|
||||
import re
|
||||
from g2p_en import G2p
|
||||
|
||||
from . import symbols
|
||||
|
||||
current_file_path = os.path.dirname(__file__)
|
||||
CMU_DICT_PATH = os.path.join(current_file_path, "cmudict.rep")
|
||||
CACHE_PATH = os.path.join(current_file_path, "cmudict_cache.pickle")
|
||||
_g2p = G2p()
|
||||
|
||||
arpa = {
|
||||
"AH0",
|
||||
"S",
|
||||
"AH1",
|
||||
"EY2",
|
||||
"AE2",
|
||||
"EH0",
|
||||
"OW2",
|
||||
"UH0",
|
||||
"NG",
|
||||
"B",
|
||||
"G",
|
||||
"AY0",
|
||||
"M",
|
||||
"AA0",
|
||||
"F",
|
||||
"AO0",
|
||||
"ER2",
|
||||
"UH1",
|
||||
"IY1",
|
||||
"AH2",
|
||||
"DH",
|
||||
"IY0",
|
||||
"EY1",
|
||||
"IH0",
|
||||
"K",
|
||||
"N",
|
||||
"W",
|
||||
"IY2",
|
||||
"T",
|
||||
"AA1",
|
||||
"ER1",
|
||||
"EH2",
|
||||
"OY0",
|
||||
"UH2",
|
||||
"UW1",
|
||||
"Z",
|
||||
"AW2",
|
||||
"AW1",
|
||||
"V",
|
||||
"UW2",
|
||||
"AA2",
|
||||
"ER",
|
||||
"AW0",
|
||||
"UW0",
|
||||
"R",
|
||||
"OW1",
|
||||
"EH1",
|
||||
"ZH",
|
||||
"AE0",
|
||||
"IH2",
|
||||
"IH",
|
||||
"Y",
|
||||
"JH",
|
||||
"P",
|
||||
"AY1",
|
||||
"EY0",
|
||||
"OY2",
|
||||
"TH",
|
||||
"HH",
|
||||
"D",
|
||||
"ER0",
|
||||
"CH",
|
||||
"AO1",
|
||||
"AE1",
|
||||
"AO2",
|
||||
"OY1",
|
||||
"AY2",
|
||||
"IH1",
|
||||
"OW0",
|
||||
"L",
|
||||
"SH",
|
||||
}
|
||||
|
||||
|
||||
def post_replace_ph(ph):
|
||||
rep_map = {
|
||||
"：": ",",
|
||||
"；": ",",
|
||||
"，": ",",
|
||||
"。": ".",
|
||||
"！": "!",
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"·": ",",
|
||||
"、": ",",
|
||||
"...": "…",
|
||||
"v": "V",
|
||||
}
|
||||
if ph in rep_map.keys():
|
||||
ph = rep_map[ph]
|
||||
if ph in symbols:
|
||||
return ph
|
||||
if ph not in symbols:
|
||||
ph = "UNK"
|
||||
return ph
|
||||
|
||||
|
||||
def read_dict():
|
||||
g2p_dict = {}
|
||||
start_line = 49
|
||||
with open(CMU_DICT_PATH) as f:
|
||||
line = f.readline()
|
||||
line_index = 1
|
||||
while line:
|
||||
if line_index >= start_line:
|
||||
line = line.strip()
|
||||
word_split = line.split(" ")
|
||||
word = word_split[0]
|
||||
|
||||
syllable_split = word_split[1].split(" - ")
|
||||
g2p_dict[word] = []
|
||||
for syllable in syllable_split:
|
||||
phone_split = syllable.split(" ")
|
||||
g2p_dict[word].append(phone_split)
|
||||
|
||||
line_index = line_index + 1
|
||||
line = f.readline()
|
||||
|
||||
return g2p_dict
|
||||
|
||||
|
||||
def cache_dict(g2p_dict, file_path):
|
||||
with open(file_path, "wb") as pickle_file:
|
||||
pickle.dump(g2p_dict, pickle_file)
|
||||
|
||||
|
||||
def get_dict():
|
||||
if os.path.exists(CACHE_PATH):
|
||||
with open(CACHE_PATH, "rb") as pickle_file:
|
||||
g2p_dict = pickle.load(pickle_file)
|
||||
else:
|
||||
g2p_dict = read_dict()
|
||||
cache_dict(g2p_dict, CACHE_PATH)
|
||||
|
||||
return g2p_dict
|
||||
|
||||
|
||||
eng_dict = get_dict()
|
||||
|
||||
|
||||
def refine_ph(phn):
|
||||
tone = 0
|
||||
if re.search(r"\d$", phn):
|
||||
tone = int(phn[-1]) + 1
|
||||
phn = phn[:-1]
|
||||
return phn.lower(), tone
|
||||
|
||||
|
||||
def refine_syllables(syllables):
|
||||
tones = []
|
||||
phonemes = []
|
||||
for phn_list in syllables:
|
||||
for i in range(len(phn_list)):
|
||||
phn = phn_list[i]
|
||||
phn, tone = refine_ph(phn)
|
||||
phonemes.append(phn)
|
||||
tones.append(tone)
|
||||
return phonemes, tones
|
||||
|
||||
|
||||
import re
|
||||
import inflect
|
||||
|
||||
_inflect = inflect.engine()
|
||||
_comma_number_re = re.compile(r"([0-9][0-9\,]+[0-9])")
|
||||
_decimal_number_re = re.compile(r"([0-9]+\.[0-9]+)")
|
||||
_pounds_re = re.compile(r"£([0-9\,]*[0-9]+)")
|
||||
_dollars_re = re.compile(r"\$([0-9\.\,]*[0-9]+)")
|
||||
_ordinal_re = re.compile(r"[0-9]+(st|nd|rd|th)")
|
||||
_number_re = re.compile(r"[0-9]+")
|
||||
|
||||
# List of (regular expression, replacement) pairs for abbreviations:
|
||||
_abbreviations = [
|
||||
(re.compile("\\b%s\\." % x[0], re.IGNORECASE), x[1])
|
||||
for x in [
|
||||
("mrs", "misess"),
|
||||
("mr", "mister"),
|
||||
("dr", "doctor"),
|
||||
("st", "saint"),
|
||||
("co", "company"),
|
||||
("jr", "junior"),
|
||||
("maj", "major"),
|
||||
("gen", "general"),
|
||||
("drs", "doctors"),
|
||||
("rev", "reverend"),
|
||||
("lt", "lieutenant"),
|
||||
("hon", "honorable"),
|
||||
("sgt", "sergeant"),
|
||||
("capt", "captain"),
|
||||
("esq", "esquire"),
|
||||
("ltd", "limited"),
|
||||
("col", "colonel"),
|
||||
("ft", "fort"),
|
||||
]
|
||||
]
|
||||
|
||||
|
||||
# List of (ipa, lazy ipa) pairs:
|
||||
_lazy_ipa = [
|
||||
(re.compile("%s" % x[0]), x[1])
|
||||
for x in [
|
||||
("r", "ɹ"),
|
||||
("æ", "e"),
|
||||
("ɑ", "a"),
|
||||
("ɔ", "o"),
|
||||
("ð", "z"),
|
||||
("θ", "s"),
|
||||
("ɛ", "e"),
|
||||
("ɪ", "i"),
|
||||
("ʊ", "u"),
|
||||
("ʒ", "ʥ"),
|
||||
("ʤ", "ʥ"),
|
||||
("ˈ", "↓"),
|
||||
]
|
||||
]
|
||||
|
||||
# List of (ipa, lazy ipa2) pairs:
|
||||
_lazy_ipa2 = [
|
||||
(re.compile("%s" % x[0]), x[1])
|
||||
for x in [
|
||||
("r", "ɹ"),
|
||||
("ð", "z"),
|
||||
("θ", "s"),
|
||||
("ʒ", "ʑ"),
|
||||
("ʤ", "dʑ"),
|
||||
("ˈ", "↓"),
|
||||
]
|
||||
]
|
||||
|
||||
# List of (ipa, ipa2) pairs
|
||||
_ipa_to_ipa2 = [
|
||||
(re.compile("%s" % x[0]), x[1]) for x in [("r", "ɹ"), ("ʤ", "dʒ"), ("ʧ", "tʃ")]
|
||||
]
|
||||
|
||||
|
||||
def _expand_dollars(m):
|
||||
match = m.group(1)
|
||||
parts = match.split(".")
|
||||
if len(parts) > 2:
|
||||
return match + " dollars" # Unexpected format
|
||||
dollars = int(parts[0]) if parts[0] else 0
|
||||
cents = int(parts[1]) if len(parts) > 1 and parts[1] else 0
|
||||
if dollars and cents:
|
||||
dollar_unit = "dollar" if dollars == 1 else "dollars"
|
||||
cent_unit = "cent" if cents == 1 else "cents"
|
||||
return "%s %s, %s %s" % (dollars, dollar_unit, cents, cent_unit)
|
||||
elif dollars:
|
||||
dollar_unit = "dollar" if dollars == 1 else "dollars"
|
||||
return "%s %s" % (dollars, dollar_unit)
|
||||
elif cents:
|
||||
cent_unit = "cent" if cents == 1 else "cents"
|
||||
return "%s %s" % (cents, cent_unit)
|
||||
else:
|
||||
return "zero dollars"
|
||||
|
||||
|
||||
def _remove_commas(m):
|
||||
return m.group(1).replace(",", "")
|
||||
|
||||
|
||||
def _expand_ordinal(m):
|
||||
return _inflect.number_to_words(m.group(0))
|
||||
|
||||
|
||||
def _expand_number(m):
|
||||
num = int(m.group(0))
|
||||
if num > 1000 and num < 3000:
|
||||
if num == 2000:
|
||||
return "two thousand"
|
||||
elif num > 2000 and num < 2010:
|
||||
return "two thousand " + _inflect.number_to_words(num % 100)
|
||||
elif num % 100 == 0:
|
||||
return _inflect.number_to_words(num // 100) + " hundred"
|
||||
else:
|
||||
return _inflect.number_to_words(
|
||||
num, andword="", zero="oh", group=2
|
||||
).replace(", ", " ")
|
||||
else:
|
||||
return _inflect.number_to_words(num, andword="")
|
||||
|
||||
|
||||
def _expand_decimal_point(m):
|
||||
return m.group(1).replace(".", " point ")
|
||||
|
||||
|
||||
def normalize_numbers(text):
|
||||
text = re.sub(_comma_number_re, _remove_commas, text)
|
||||
text = re.sub(_pounds_re, r"\1 pounds", text)
|
||||
text = re.sub(_dollars_re, _expand_dollars, text)
|
||||
text = re.sub(_decimal_number_re, _expand_decimal_point, text)
|
||||
text = re.sub(_ordinal_re, _expand_ordinal, text)
|
||||
text = re.sub(_number_re, _expand_number, text)
|
||||
return text
|
||||
|
||||
|
||||
def text_normalize(text):
|
||||
text = normalize_numbers(text)
|
||||
return text
|
||||
|
||||
|
||||
def g2p(text):
|
||||
phones = []
|
||||
tones = []
|
||||
word2ph = []
|
||||
words = re.split(r"([,;.\-\?\!\s+])", text)
|
||||
words = [word for word in words if word.strip() != ""]
|
||||
for word in words:
|
||||
if word.upper() in eng_dict:
|
||||
phns, tns = refine_syllables(eng_dict[word.upper()])
|
||||
phones += phns
|
||||
tones += tns
|
||||
word2ph.append(len(phns))
|
||||
else:
|
||||
phone_list = list(filter(lambda p: p != " ", _g2p(word)))
|
||||
for ph in phone_list:
|
||||
if ph in arpa:
|
||||
ph, tn = refine_ph(ph)
|
||||
phones.append(ph)
|
||||
tones.append(tn)
|
||||
else:
|
||||
phones.append(ph)
|
||||
tones.append(0)
|
||||
word2ph.append(len(phone_list))
|
||||
|
||||
phones = [post_replace_ph(i) for i in phones]
|
||||
|
||||
phones = ["_"] + phones + ["_"]
|
||||
tones = [0] + tones + [0]
|
||||
word2ph = [1] + word2ph + [1]
|
||||
|
||||
return phones, tones, word2ph
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph):
|
||||
from text import english_bert_mock
|
||||
|
||||
return english_bert_mock.get_bert_feature(text, word2ph)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# print(get_dict())
|
||||
# print(eng_word_to_phoneme("hello"))
|
||||
print(g2p("In this paper, we propose 1 DSPGAN, a GAN-based universal vocoder."))
|
||||
# all_phones = set()
|
||||
# for k, syllables in eng_dict.items():
|
||||
# for group in syllables:
|
||||
# for ph in group:
|
||||
# all_phones.add(ph)
|
||||
# print(all_phones)
|
||||
42
oldVersion/V200/text/english_bert_mock.py
Normal file
42
oldVersion/V200/text/english_bert_mock.py
Normal file
@@ -0,0 +1,42 @@
|
||||
import sys
|
||||
|
||||
import torch
|
||||
from transformers import DebertaV2Model, DebertaV2Tokenizer
|
||||
|
||||
from config import config
|
||||
|
||||
|
||||
LOCAL_PATH = "./bert/deberta-v3-large"
|
||||
|
||||
tokenizer = DebertaV2Tokenizer.from_pretrained(LOCAL_PATH)
|
||||
|
||||
models = dict()
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph, device=config.bert_gen_config.device):
|
||||
if (
|
||||
sys.platform == "darwin"
|
||||
and torch.backends.mps.is_available()
|
||||
and device == "cpu"
|
||||
):
|
||||
device = "mps"
|
||||
if not device:
|
||||
device = "cuda"
|
||||
if device not in models.keys():
|
||||
models[device] = DebertaV2Model.from_pretrained(LOCAL_PATH).to(device)
|
||||
with torch.no_grad():
|
||||
inputs = tokenizer(text, return_tensors="pt")
|
||||
for i in inputs:
|
||||
inputs[i] = inputs[i].to(device)
|
||||
res = models[device](**inputs, output_hidden_states=True)
|
||||
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
|
||||
# assert len(word2ph) == len(text)+2
|
||||
word2phone = word2ph
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
repeat_feature = res[i].repeat(word2phone[i], 1)
|
||||
phone_level_feature.append(repeat_feature)
|
||||
|
||||
phone_level_feature = torch.cat(phone_level_feature, dim=0)
|
||||
|
||||
return phone_level_feature.T
|
||||
403
oldVersion/V200/text/japanese.py
Normal file
403
oldVersion/V200/text/japanese.py
Normal file
@@ -0,0 +1,403 @@
|
||||
# Convert Japanese text to phonemes which is
|
||||
# compatible with Julius https://github.com/julius-speech/segmentation-kit
|
||||
import re
|
||||
import unicodedata
|
||||
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
from . import punctuation, symbols
|
||||
|
||||
from num2words import num2words
|
||||
|
||||
import pyopenjtalk
|
||||
import jaconv
|
||||
|
||||
|
||||
def kata2phoneme(text: str) -> str:
|
||||
"""Convert katakana text to phonemes."""
|
||||
text = text.strip()
|
||||
if text == "ー":
|
||||
return ["ー"]
|
||||
elif text.startswith("ー"):
|
||||
return ["ー"] + kata2phoneme(text[1:])
|
||||
res = []
|
||||
prev = None
|
||||
while text:
|
||||
if re.match(_MARKS, text):
|
||||
res.append(text)
|
||||
text = text[1:]
|
||||
continue
|
||||
if text.startswith("ー"):
|
||||
if prev:
|
||||
res.append(prev[-1])
|
||||
text = text[1:]
|
||||
continue
|
||||
res += pyopenjtalk.g2p(text).lower().replace("cl", "q").split(" ")
|
||||
break
|
||||
# res = _COLON_RX.sub(":", res)
|
||||
return res
|
||||
|
||||
|
||||
def hira2kata(text: str) -> str:
|
||||
return jaconv.hira2kata(text)
|
||||
|
||||
|
||||
_SYMBOL_TOKENS = set(list("・、。？！"))
|
||||
_NO_YOMI_TOKENS = set(list("「」『』―（）［］[]"))
|
||||
_MARKS = re.compile(
|
||||
r"[^A-Za-z\d\u3005\u3040-\u30ff\u4e00-\u9fff\uff11-\uff19\uff21-\uff3a\uff41-\uff5a\uff66-\uff9d]"
|
||||
)
|
||||
|
||||
|
||||
def text2kata(text: str) -> str:
|
||||
parsed = pyopenjtalk.run_frontend(text)
|
||||
|
||||
res = []
|
||||
for parts in parsed:
|
||||
word, yomi = replace_punctuation(parts["string"]), parts["pron"].replace(
|
||||
"’", ""
|
||||
)
|
||||
if yomi:
|
||||
if re.match(_MARKS, yomi):
|
||||
if len(word) > 1:
|
||||
word = [replace_punctuation(i) for i in list(word)]
|
||||
yomi = word
|
||||
res += yomi
|
||||
sep += word
|
||||
continue
|
||||
elif word not in rep_map.keys() and word not in rep_map.values():
|
||||
word = ","
|
||||
yomi = word
|
||||
res.append(yomi)
|
||||
else:
|
||||
if word in _SYMBOL_TOKENS:
|
||||
res.append(word)
|
||||
elif word in ("っ", "ッ"):
|
||||
res.append("ッ")
|
||||
elif word in _NO_YOMI_TOKENS:
|
||||
pass
|
||||
else:
|
||||
res.append(word)
|
||||
return hira2kata("".join(res))
|
||||
|
||||
|
||||
def text2sep_kata(text: str) -> (list, list):
|
||||
parsed = pyopenjtalk.run_frontend(text)
|
||||
|
||||
res = []
|
||||
sep = []
|
||||
for parts in parsed:
|
||||
word, yomi = replace_punctuation(parts["string"]), parts["pron"].replace(
|
||||
"’", ""
|
||||
)
|
||||
if yomi:
|
||||
if re.match(_MARKS, yomi):
|
||||
if len(word) > 1:
|
||||
word = [replace_punctuation(i) for i in list(word)]
|
||||
yomi = word
|
||||
res += yomi
|
||||
sep += word
|
||||
continue
|
||||
elif word not in rep_map.keys() and word not in rep_map.values():
|
||||
word = ","
|
||||
yomi = word
|
||||
res.append(yomi)
|
||||
else:
|
||||
if word in _SYMBOL_TOKENS:
|
||||
res.append(word)
|
||||
elif word in ("っ", "ッ"):
|
||||
res.append("ッ")
|
||||
elif word in _NO_YOMI_TOKENS:
|
||||
pass
|
||||
else:
|
||||
res.append(word)
|
||||
sep.append(word)
|
||||
return sep, [hira2kata(i) for i in res], get_accent(parsed)
|
||||
|
||||
|
||||
def get_accent(parsed):
|
||||
labels = pyopenjtalk.make_label(parsed)
|
||||
|
||||
phonemes = []
|
||||
accents = []
|
||||
for n, label in enumerate(labels):
|
||||
phoneme = re.search(r"\-([^\+]*)\+", label).group(1)
|
||||
if phoneme not in ["sil", "pau"]:
|
||||
phonemes.append(phoneme.replace("cl", "q").lower())
|
||||
else:
|
||||
continue
|
||||
a1 = int(re.search(r"/A:(\-?[0-9]+)\+", label).group(1))
|
||||
a2 = int(re.search(r"\+(\d+)\+", label).group(1))
|
||||
if re.search(r"\-([^\+]*)\+", labels[n + 1]).group(1) in ["sil", "pau"]:
|
||||
a2_next = -1
|
||||
else:
|
||||
a2_next = int(re.search(r"\+(\d+)\+", labels[n + 1]).group(1))
|
||||
# Falling
|
||||
if a1 == 0 and a2_next == a2 + 1:
|
||||
accents.append(-1)
|
||||
# Rising
|
||||
elif a2 == 1 and a2_next == 2:
|
||||
accents.append(1)
|
||||
else:
|
||||
accents.append(0)
|
||||
return list(zip(phonemes, accents))
|
||||
|
||||
|
||||
_ALPHASYMBOL_YOMI = {
|
||||
"#": "シャープ",
|
||||
"%": "パーセント",
|
||||
"&": "アンド",
|
||||
"+": "プラス",
|
||||
"-": "マイナス",
|
||||
":": "コロン",
|
||||
";": "セミコロン",
|
||||
"<": "小なり",
|
||||
"=": "イコール",
|
||||
">": "大なり",
|
||||
"@": "アット",
|
||||
"a": "エー",
|
||||
"b": "ビー",
|
||||
"c": "シー",
|
||||
"d": "ディー",
|
||||
"e": "イー",
|
||||
"f": "エフ",
|
||||
"g": "ジー",
|
||||
"h": "エイチ",
|
||||
"i": "アイ",
|
||||
"j": "ジェー",
|
||||
"k": "ケー",
|
||||
"l": "エル",
|
||||
"m": "エム",
|
||||
"n": "エヌ",
|
||||
"o": "オー",
|
||||
"p": "ピー",
|
||||
"q": "キュー",
|
||||
"r": "アール",
|
||||
"s": "エス",
|
||||
"t": "ティー",
|
||||
"u": "ユー",
|
||||
"v": "ブイ",
|
||||
"w": "ダブリュー",
|
||||
"x": "エックス",
|
||||
"y": "ワイ",
|
||||
"z": "ゼット",
|
||||
"α": "アルファ",
|
||||
"β": "ベータ",
|
||||
"γ": "ガンマ",
|
||||
"δ": "デルタ",
|
||||
"ε": "イプシロン",
|
||||
"ζ": "ゼータ",
|
||||
"η": "イータ",
|
||||
"θ": "シータ",
|
||||
"ι": "イオタ",
|
||||
"κ": "カッパ",
|
||||
"λ": "ラムダ",
|
||||
"μ": "ミュー",
|
||||
"ν": "ニュー",
|
||||
"ξ": "クサイ",
|
||||
"ο": "オミクロン",
|
||||
"π": "パイ",
|
||||
"ρ": "ロー",
|
||||
"σ": "シグマ",
|
||||
"τ": "タウ",
|
||||
"υ": "ウプシロン",
|
||||
"φ": "ファイ",
|
||||
"χ": "カイ",
|
||||
"ψ": "プサイ",
|
||||
"ω": "オメガ",
|
||||
}
|
||||
|
||||
|
||||
_NUMBER_WITH_SEPARATOR_RX = re.compile("[0-9]{1,3}(,[0-9]{3})+")
|
||||
_CURRENCY_MAP = {"$": "ドル", "¥": "円", "£": "ポンド", "€": "ユーロ"}
|
||||
_CURRENCY_RX = re.compile(r"([$¥£€])([0-9.]*[0-9])")
|
||||
_NUMBER_RX = re.compile(r"[0-9]+(\.[0-9]+)?")
|
||||
|
||||
|
||||
def japanese_convert_numbers_to_words(text: str) -> str:
|
||||
res = _NUMBER_WITH_SEPARATOR_RX.sub(lambda m: m[0].replace(",", ""), text)
|
||||
res = _CURRENCY_RX.sub(lambda m: m[2] + _CURRENCY_MAP.get(m[1], m[1]), res)
|
||||
res = _NUMBER_RX.sub(lambda m: num2words(m[0], lang="ja"), res)
|
||||
return res
|
||||
|
||||
|
||||
def japanese_convert_alpha_symbols_to_words(text: str) -> str:
|
||||
return "".join([_ALPHASYMBOL_YOMI.get(ch, ch) for ch in text.lower()])
|
||||
|
||||
|
||||
def japanese_text_to_phonemes(text: str) -> str:
|
||||
"""Convert Japanese text to phonemes."""
|
||||
res = unicodedata.normalize("NFKC", text)
|
||||
res = japanese_convert_numbers_to_words(res)
|
||||
# res = japanese_convert_alpha_symbols_to_words(res)
|
||||
res = text2kata(res)
|
||||
res = kata2phoneme(res)
|
||||
return res
|
||||
|
||||
|
||||
def is_japanese_character(char):
|
||||
# 定义日语文字系统的 Unicode 范围
|
||||
japanese_ranges = [
|
||||
(0x3040, 0x309F), # 平假名
|
||||
(0x30A0, 0x30FF), # 片假名
|
||||
(0x4E00, 0x9FFF), # 汉字 (CJK Unified Ideographs)
|
||||
(0x3400, 0x4DBF), # 汉字扩展 A
|
||||
(0x20000, 0x2A6DF), # 汉字扩展 B
|
||||
# 可以根据需要添加其他汉字扩展范围
|
||||
]
|
||||
|
||||
# 将字符的 Unicode 编码转换为整数
|
||||
char_code = ord(char)
|
||||
|
||||
# 检查字符是否在任何一个日语范围内
|
||||
for start, end in japanese_ranges:
|
||||
if start <= char_code <= end:
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
rep_map = {
|
||||
"：": ",",
|
||||
"；": ",",
|
||||
"，": ",",
|
||||
"。": ".",
|
||||
"！": "!",
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"．": ".",
|
||||
"...": "…",
|
||||
"···": "…",
|
||||
"・・・": "…",
|
||||
"·": ",",
|
||||
"・": ",",
|
||||
"、": ",",
|
||||
"$": ".",
|
||||
"“": "'",
|
||||
"”": "'",
|
||||
"‘": "'",
|
||||
"’": "'",
|
||||
"（": "'",
|
||||
"）": "'",
|
||||
"(": "'",
|
||||
")": "'",
|
||||
"《": "'",
|
||||
"》": "'",
|
||||
"【": "'",
|
||||
"】": "'",
|
||||
"[": "'",
|
||||
"]": "'",
|
||||
"—": "-",
|
||||
"−": "-",
|
||||
"～": "-",
|
||||
"~": "-",
|
||||
"「": "'",
|
||||
"」": "'",
|
||||
}
|
||||
|
||||
|
||||
def replace_punctuation(text):
|
||||
pattern = re.compile("|".join(re.escape(p) for p in rep_map.keys()))
|
||||
|
||||
replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
|
||||
|
||||
replaced_text = re.sub(
|
||||
r"[^\u3040-\u309F\u30A0-\u30FF\u4E00-\u9FFF\u3400-\u4DBF\u3005"
|
||||
+ "".join(punctuation)
|
||||
+ r"]+",
|
||||
"",
|
||||
replaced_text,
|
||||
)
|
||||
|
||||
return replaced_text
|
||||
|
||||
|
||||
def text_normalize(text):
|
||||
res = unicodedata.normalize("NFKC", text)
|
||||
res = japanese_convert_numbers_to_words(res)
|
||||
# res = "".join([i for i in res if is_japanese_character(i)])
|
||||
res = replace_punctuation(res)
|
||||
return res
|
||||
|
||||
|
||||
def distribute_phone(n_phone, n_word):
|
||||
phones_per_word = [0] * n_word
|
||||
for task in range(n_phone):
|
||||
min_tasks = min(phones_per_word)
|
||||
min_index = phones_per_word.index(min_tasks)
|
||||
phones_per_word[min_index] += 1
|
||||
return phones_per_word
|
||||
|
||||
|
||||
def handle_long(sep_phonemes):
|
||||
for i in range(len(sep_phonemes)):
|
||||
if sep_phonemes[i][0] == "ー":
|
||||
sep_phonemes[i][0] = sep_phonemes[i - 1][-1]
|
||||
if "ー" in sep_phonemes[i]:
|
||||
for j in range(len(sep_phonemes[i])):
|
||||
if sep_phonemes[i][j] == "ー":
|
||||
sep_phonemes[i][j] = sep_phonemes[i][j - 1][-1]
|
||||
return sep_phonemes
|
||||
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/deberta-v2-large-japanese")
|
||||
|
||||
|
||||
def align_tones(phones, tones):
|
||||
res = []
|
||||
for pho in phones:
|
||||
temp = [0] * len(pho)
|
||||
for idx, p in enumerate(pho):
|
||||
if len(tones) == 0:
|
||||
break
|
||||
if p == tones[0][0]:
|
||||
temp[idx] = tones[0][1]
|
||||
if idx > 0:
|
||||
temp[idx] += temp[idx - 1]
|
||||
tones.pop(0)
|
||||
temp = [0] + temp
|
||||
temp = temp[:-1]
|
||||
if -1 in temp:
|
||||
temp = [i + 1 for i in temp]
|
||||
res.append(temp)
|
||||
res = [i for j in res for i in j]
|
||||
assert not any([i < 0 for i in res]) and not any([i > 1 for i in res])
|
||||
return res
|
||||
|
||||
|
||||
def g2p(norm_text):
|
||||
sep_text, sep_kata, acc = text2sep_kata(norm_text)
|
||||
sep_tokenized = [tokenizer.tokenize(i) for i in sep_text]
|
||||
sep_phonemes = handle_long([kata2phoneme(i) for i in sep_kata])
|
||||
# 异常处理，MeCab不认识的词的话会一路传到这里来，然后炸掉。目前来看只有那些超级稀有的生僻词会出现这种情况
|
||||
for i in sep_phonemes:
|
||||
for j in i:
|
||||
assert j in symbols, (sep_text, sep_kata, sep_phonemes)
|
||||
tones = align_tones(sep_phonemes, acc)
|
||||
|
||||
word2ph = []
|
||||
for token, phoneme in zip(sep_tokenized, sep_phonemes):
|
||||
phone_len = len(phoneme)
|
||||
word_len = len(token)
|
||||
|
||||
aaa = distribute_phone(phone_len, word_len)
|
||||
word2ph += aaa
|
||||
phones = ["_"] + [j for i in sep_phonemes for j in i] + ["_"]
|
||||
tones = [0] + tones + [0]
|
||||
word2ph = [1] + word2ph + [1]
|
||||
assert len(phones) == len(tones)
|
||||
return phones, tones, word2ph
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/deberta-v2-large-japanese")
|
||||
text = "hello,こんにちは、世界ー！……"
|
||||
from text.japanese_bert import get_bert_feature
|
||||
|
||||
text = text_normalize(text)
|
||||
print(text)
|
||||
|
||||
phones, tones, word2ph = g2p(text)
|
||||
bert = get_bert_feature(text, word2ph)
|
||||
|
||||
print(phones, tones, word2ph, bert.shape)
|
||||
58
oldVersion/V200/text/japanese_bert.py
Normal file
58
oldVersion/V200/text/japanese_bert.py
Normal file
@@ -0,0 +1,58 @@
|
||||
import sys
|
||||
|
||||
import torch
|
||||
from transformers import AutoModelForMaskedLM, AutoTokenizer
|
||||
|
||||
from config import config
|
||||
from .japanese import text2sep_kata
|
||||
|
||||
LOCAL_PATH = "./bert/deberta-v2-large-japanese"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(LOCAL_PATH)
|
||||
|
||||
models = dict()
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph, device=config.bert_gen_config.device):
|
||||
sep_text, _, _ = text2sep_kata(text)
|
||||
sep_tokens = [tokenizer.tokenize(t) for t in sep_text]
|
||||
sep_ids = [tokenizer.convert_tokens_to_ids(t) for t in sep_tokens]
|
||||
sep_ids = [2] + [item for sublist in sep_ids for item in sublist] + [3]
|
||||
return get_bert_feature_with_token(sep_ids, word2ph, device)
|
||||
|
||||
|
||||
def get_bert_feature_with_token(tokens, word2ph, device=config.bert_gen_config.device):
|
||||
if (
|
||||
sys.platform == "darwin"
|
||||
and torch.backends.mps.is_available()
|
||||
and device == "cpu"
|
||||
):
|
||||
device = "mps"
|
||||
if not device:
|
||||
device = "cuda"
|
||||
if device not in models.keys():
|
||||
models[device] = AutoModelForMaskedLM.from_pretrained(LOCAL_PATH).to(device)
|
||||
with torch.no_grad():
|
||||
inputs = torch.tensor(tokens).to(device).unsqueeze(0)
|
||||
token_type_ids = torch.zeros_like(inputs).to(device)
|
||||
attention_mask = torch.ones_like(inputs).to(device)
|
||||
inputs = {
|
||||
"input_ids": inputs,
|
||||
"token_type_ids": token_type_ids,
|
||||
"attention_mask": attention_mask,
|
||||
}
|
||||
|
||||
# for i in inputs:
|
||||
# inputs[i] = inputs[i].to(device)
|
||||
res = models[device](**inputs, output_hidden_states=True)
|
||||
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
|
||||
assert inputs["input_ids"].shape[-1] == len(word2ph)
|
||||
word2phone = word2ph
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
repeat_feature = res[i].repeat(word2phone[i], 1)
|
||||
phone_level_feature.append(repeat_feature)
|
||||
|
||||
phone_level_feature = torch.cat(phone_level_feature, dim=0)
|
||||
|
||||
return phone_level_feature.T
|
||||
429
oldVersion/V200/text/opencpop-strict.txt
Normal file
429
oldVersion/V200/text/opencpop-strict.txt
Normal file
@@ -0,0 +1,429 @@
|
||||
a AA a
|
||||
ai AA ai
|
||||
an AA an
|
||||
ang AA ang
|
||||
ao AA ao
|
||||
ba b a
|
||||
bai b ai
|
||||
ban b an
|
||||
bang b ang
|
||||
bao b ao
|
||||
bei b ei
|
||||
ben b en
|
||||
beng b eng
|
||||
bi b i
|
||||
bian b ian
|
||||
biao b iao
|
||||
bie b ie
|
||||
bin b in
|
||||
bing b ing
|
||||
bo b o
|
||||
bu b u
|
||||
ca c a
|
||||
cai c ai
|
||||
can c an
|
||||
cang c ang
|
||||
cao c ao
|
||||
ce c e
|
||||
cei c ei
|
||||
cen c en
|
||||
ceng c eng
|
||||
cha ch a
|
||||
chai ch ai
|
||||
chan ch an
|
||||
chang ch ang
|
||||
chao ch ao
|
||||
che ch e
|
||||
chen ch en
|
||||
cheng ch eng
|
||||
chi ch ir
|
||||
chong ch ong
|
||||
chou ch ou
|
||||
chu ch u
|
||||
chua ch ua
|
||||
chuai ch uai
|
||||
chuan ch uan
|
||||
chuang ch uang
|
||||
chui ch ui
|
||||
chun ch un
|
||||
chuo ch uo
|
||||
ci c i0
|
||||
cong c ong
|
||||
cou c ou
|
||||
cu c u
|
||||
cuan c uan
|
||||
cui c ui
|
||||
cun c un
|
||||
cuo c uo
|
||||
da d a
|
||||
dai d ai
|
||||
dan d an
|
||||
dang d ang
|
||||
dao d ao
|
||||
de d e
|
||||
dei d ei
|
||||
den d en
|
||||
deng d eng
|
||||
di d i
|
||||
dia d ia
|
||||
dian d ian
|
||||
diao d iao
|
||||
die d ie
|
||||
ding d ing
|
||||
diu d iu
|
||||
dong d ong
|
||||
dou d ou
|
||||
du d u
|
||||
duan d uan
|
||||
dui d ui
|
||||
dun d un
|
||||
duo d uo
|
||||
e EE e
|
||||
ei EE ei
|
||||
en EE en
|
||||
eng EE eng
|
||||
er EE er
|
||||
fa f a
|
||||
fan f an
|
||||
fang f ang
|
||||
fei f ei
|
||||
fen f en
|
||||
feng f eng
|
||||
fo f o
|
||||
fou f ou
|
||||
fu f u
|
||||
ga g a
|
||||
gai g ai
|
||||
gan g an
|
||||
gang g ang
|
||||
gao g ao
|
||||
ge g e
|
||||
gei g ei
|
||||
gen g en
|
||||
geng g eng
|
||||
gong g ong
|
||||
gou g ou
|
||||
gu g u
|
||||
gua g ua
|
||||
guai g uai
|
||||
guan g uan
|
||||
guang g uang
|
||||
gui g ui
|
||||
gun g un
|
||||
guo g uo
|
||||
ha h a
|
||||
hai h ai
|
||||
han h an
|
||||
hang h ang
|
||||
hao h ao
|
||||
he h e
|
||||
hei h ei
|
||||
hen h en
|
||||
heng h eng
|
||||
hong h ong
|
||||
hou h ou
|
||||
hu h u
|
||||
hua h ua
|
||||
huai h uai
|
||||
huan h uan
|
||||
huang h uang
|
||||
hui h ui
|
||||
hun h un
|
||||
huo h uo
|
||||
ji j i
|
||||
jia j ia
|
||||
jian j ian
|
||||
jiang j iang
|
||||
jiao j iao
|
||||
jie j ie
|
||||
jin j in
|
||||
jing j ing
|
||||
jiong j iong
|
||||
jiu j iu
|
||||
ju j v
|
||||
jv j v
|
||||
juan j van
|
||||
jvan j van
|
||||
jue j ve
|
||||
jve j ve
|
||||
jun j vn
|
||||
jvn j vn
|
||||
ka k a
|
||||
kai k ai
|
||||
kan k an
|
||||
kang k ang
|
||||
kao k ao
|
||||
ke k e
|
||||
kei k ei
|
||||
ken k en
|
||||
keng k eng
|
||||
kong k ong
|
||||
kou k ou
|
||||
ku k u
|
||||
kua k ua
|
||||
kuai k uai
|
||||
kuan k uan
|
||||
kuang k uang
|
||||
kui k ui
|
||||
kun k un
|
||||
kuo k uo
|
||||
la l a
|
||||
lai l ai
|
||||
lan l an
|
||||
lang l ang
|
||||
lao l ao
|
||||
le l e
|
||||
lei l ei
|
||||
leng l eng
|
||||
li l i
|
||||
lia l ia
|
||||
lian l ian
|
||||
liang l iang
|
||||
liao l iao
|
||||
lie l ie
|
||||
lin l in
|
||||
ling l ing
|
||||
liu l iu
|
||||
lo l o
|
||||
long l ong
|
||||
lou l ou
|
||||
lu l u
|
||||
luan l uan
|
||||
lun l un
|
||||
luo l uo
|
||||
lv l v
|
||||
lve l ve
|
||||
ma m a
|
||||
mai m ai
|
||||
man m an
|
||||
mang m ang
|
||||
mao m ao
|
||||
me m e
|
||||
mei m ei
|
||||
men m en
|
||||
meng m eng
|
||||
mi m i
|
||||
mian m ian
|
||||
miao m iao
|
||||
mie m ie
|
||||
min m in
|
||||
ming m ing
|
||||
miu m iu
|
||||
mo m o
|
||||
mou m ou
|
||||
mu m u
|
||||
na n a
|
||||
nai n ai
|
||||
nan n an
|
||||
nang n ang
|
||||
nao n ao
|
||||
ne n e
|
||||
nei n ei
|
||||
nen n en
|
||||
neng n eng
|
||||
ni n i
|
||||
nian n ian
|
||||
niang n iang
|
||||
niao n iao
|
||||
nie n ie
|
||||
nin n in
|
||||
ning n ing
|
||||
niu n iu
|
||||
nong n ong
|
||||
nou n ou
|
||||
nu n u
|
||||
nuan n uan
|
||||
nun n un
|
||||
nuo n uo
|
||||
nv n v
|
||||
nve n ve
|
||||
o OO o
|
||||
ou OO ou
|
||||
pa p a
|
||||
pai p ai
|
||||
pan p an
|
||||
pang p ang
|
||||
pao p ao
|
||||
pei p ei
|
||||
pen p en
|
||||
peng p eng
|
||||
pi p i
|
||||
pian p ian
|
||||
piao p iao
|
||||
pie p ie
|
||||
pin p in
|
||||
ping p ing
|
||||
po p o
|
||||
pou p ou
|
||||
pu p u
|
||||
qi q i
|
||||
qia q ia
|
||||
qian q ian
|
||||
qiang q iang
|
||||
qiao q iao
|
||||
qie q ie
|
||||
qin q in
|
||||
qing q ing
|
||||
qiong q iong
|
||||
qiu q iu
|
||||
qu q v
|
||||
qv q v
|
||||
quan q van
|
||||
qvan q van
|
||||
que q ve
|
||||
qve q ve
|
||||
qun q vn
|
||||
qvn q vn
|
||||
ran r an
|
||||
rang r ang
|
||||
rao r ao
|
||||
re r e
|
||||
ren r en
|
||||
reng r eng
|
||||
ri r ir
|
||||
rong r ong
|
||||
rou r ou
|
||||
ru r u
|
||||
rua r ua
|
||||
ruan r uan
|
||||
rui r ui
|
||||
run r un
|
||||
ruo r uo
|
||||
sa s a
|
||||
sai s ai
|
||||
san s an
|
||||
sang s ang
|
||||
sao s ao
|
||||
se s e
|
||||
sen s en
|
||||
seng s eng
|
||||
sha sh a
|
||||
shai sh ai
|
||||
shan sh an
|
||||
shang sh ang
|
||||
shao sh ao
|
||||
she sh e
|
||||
shei sh ei
|
||||
shen sh en
|
||||
sheng sh eng
|
||||
shi sh ir
|
||||
shou sh ou
|
||||
shu sh u
|
||||
shua sh ua
|
||||
shuai sh uai
|
||||
shuan sh uan
|
||||
shuang sh uang
|
||||
shui sh ui
|
||||
shun sh un
|
||||
shuo sh uo
|
||||
si s i0
|
||||
song s ong
|
||||
sou s ou
|
||||
su s u
|
||||
suan s uan
|
||||
sui s ui
|
||||
sun s un
|
||||
suo s uo
|
||||
ta t a
|
||||
tai t ai
|
||||
tan t an
|
||||
tang t ang
|
||||
tao t ao
|
||||
te t e
|
||||
tei t ei
|
||||
teng t eng
|
||||
ti t i
|
||||
tian t ian
|
||||
tiao t iao
|
||||
tie t ie
|
||||
ting t ing
|
||||
tong t ong
|
||||
tou t ou
|
||||
tu t u
|
||||
tuan t uan
|
||||
tui t ui
|
||||
tun t un
|
||||
tuo t uo
|
||||
wa w a
|
||||
wai w ai
|
||||
wan w an
|
||||
wang w ang
|
||||
wei w ei
|
||||
wen w en
|
||||
weng w eng
|
||||
wo w o
|
||||
wu w u
|
||||
xi x i
|
||||
xia x ia
|
||||
xian x ian
|
||||
xiang x iang
|
||||
xiao x iao
|
||||
xie x ie
|
||||
xin x in
|
||||
xing x ing
|
||||
xiong x iong
|
||||
xiu x iu
|
||||
xu x v
|
||||
xv x v
|
||||
xuan x van
|
||||
xvan x van
|
||||
xue x ve
|
||||
xve x ve
|
||||
xun x vn
|
||||
xvn x vn
|
||||
ya y a
|
||||
yan y En
|
||||
yang y ang
|
||||
yao y ao
|
||||
ye y E
|
||||
yi y i
|
||||
yin y in
|
||||
ying y ing
|
||||
yo y o
|
||||
yong y ong
|
||||
you y ou
|
||||
yu y v
|
||||
yv y v
|
||||
yuan y van
|
||||
yvan y van
|
||||
yue y ve
|
||||
yve y ve
|
||||
yun y vn
|
||||
yvn y vn
|
||||
za z a
|
||||
zai z ai
|
||||
zan z an
|
||||
zang z ang
|
||||
zao z ao
|
||||
ze z e
|
||||
zei z ei
|
||||
zen z en
|
||||
zeng z eng
|
||||
zha zh a
|
||||
zhai zh ai
|
||||
zhan zh an
|
||||
zhang zh ang
|
||||
zhao zh ao
|
||||
zhe zh e
|
||||
zhei zh ei
|
||||
zhen zh en
|
||||
zheng zh eng
|
||||
zhi zh ir
|
||||
zhong zh ong
|
||||
zhou zh ou
|
||||
zhu zh u
|
||||
zhua zh ua
|
||||
zhuai zh uai
|
||||
zhuan zh uan
|
||||
zhuang zh uang
|
||||
zhui zh ui
|
||||
zhun zh un
|
||||
zhuo zh uo
|
||||
zi z i0
|
||||
zong z ong
|
||||
zou z ou
|
||||
zu z u
|
||||
zuan z uan
|
||||
zui z ui
|
||||
zun z un
|
||||
zuo z uo
|
||||
187
oldVersion/V200/text/symbols.py
Normal file
187
oldVersion/V200/text/symbols.py
Normal file
@@ -0,0 +1,187 @@
|
||||
punctuation = ["!", "?", "…", ",", ".", "'", "-"]
|
||||
pu_symbols = punctuation + ["SP", "UNK"]
|
||||
pad = "_"
|
||||
|
||||
# chinese
|
||||
zh_symbols = [
|
||||
"E",
|
||||
"En",
|
||||
"a",
|
||||
"ai",
|
||||
"an",
|
||||
"ang",
|
||||
"ao",
|
||||
"b",
|
||||
"c",
|
||||
"ch",
|
||||
"d",
|
||||
"e",
|
||||
"ei",
|
||||
"en",
|
||||
"eng",
|
||||
"er",
|
||||
"f",
|
||||
"g",
|
||||
"h",
|
||||
"i",
|
||||
"i0",
|
||||
"ia",
|
||||
"ian",
|
||||
"iang",
|
||||
"iao",
|
||||
"ie",
|
||||
"in",
|
||||
"ing",
|
||||
"iong",
|
||||
"ir",
|
||||
"iu",
|
||||
"j",
|
||||
"k",
|
||||
"l",
|
||||
"m",
|
||||
"n",
|
||||
"o",
|
||||
"ong",
|
||||
"ou",
|
||||
"p",
|
||||
"q",
|
||||
"r",
|
||||
"s",
|
||||
"sh",
|
||||
"t",
|
||||
"u",
|
||||
"ua",
|
||||
"uai",
|
||||
"uan",
|
||||
"uang",
|
||||
"ui",
|
||||
"un",
|
||||
"uo",
|
||||
"v",
|
||||
"van",
|
||||
"ve",
|
||||
"vn",
|
||||
"w",
|
||||
"x",
|
||||
"y",
|
||||
"z",
|
||||
"zh",
|
||||
"AA",
|
||||
"EE",
|
||||
"OO",
|
||||
]
|
||||
num_zh_tones = 6
|
||||
|
||||
# japanese
|
||||
ja_symbols = [
|
||||
"N",
|
||||
"a",
|
||||
"a:",
|
||||
"b",
|
||||
"by",
|
||||
"ch",
|
||||
"d",
|
||||
"dy",
|
||||
"e",
|
||||
"e:",
|
||||
"f",
|
||||
"g",
|
||||
"gy",
|
||||
"h",
|
||||
"hy",
|
||||
"i",
|
||||
"i:",
|
||||
"j",
|
||||
"k",
|
||||
"ky",
|
||||
"m",
|
||||
"my",
|
||||
"n",
|
||||
"ny",
|
||||
"o",
|
||||
"o:",
|
||||
"p",
|
||||
"py",
|
||||
"q",
|
||||
"r",
|
||||
"ry",
|
||||
"s",
|
||||
"sh",
|
||||
"t",
|
||||
"ts",
|
||||
"ty",
|
||||
"u",
|
||||
"u:",
|
||||
"w",
|
||||
"y",
|
||||
"z",
|
||||
"zy",
|
||||
]
|
||||
num_ja_tones = 2
|
||||
|
||||
# English
|
||||
en_symbols = [
|
||||
"aa",
|
||||
"ae",
|
||||
"ah",
|
||||
"ao",
|
||||
"aw",
|
||||
"ay",
|
||||
"b",
|
||||
"ch",
|
||||
"d",
|
||||
"dh",
|
||||
"eh",
|
||||
"er",
|
||||
"ey",
|
||||
"f",
|
||||
"g",
|
||||
"hh",
|
||||
"ih",
|
||||
"iy",
|
||||
"jh",
|
||||
"k",
|
||||
"l",
|
||||
"m",
|
||||
"n",
|
||||
"ng",
|
||||
"ow",
|
||||
"oy",
|
||||
"p",
|
||||
"r",
|
||||
"s",
|
||||
"sh",
|
||||
"t",
|
||||
"th",
|
||||
"uh",
|
||||
"uw",
|
||||
"V",
|
||||
"w",
|
||||
"y",
|
||||
"z",
|
||||
"zh",
|
||||
]
|
||||
num_en_tones = 4
|
||||
|
||||
# combine all symbols
|
||||
normal_symbols = sorted(set(zh_symbols + ja_symbols + en_symbols))
|
||||
symbols = [pad] + normal_symbols + pu_symbols
|
||||
sil_phonemes_ids = [symbols.index(i) for i in pu_symbols]
|
||||
|
||||
# combine all tones
|
||||
num_tones = num_zh_tones + num_ja_tones + num_en_tones
|
||||
|
||||
# language maps
|
||||
language_id_map = {"ZH": 0, "JP": 1, "EN": 2}
|
||||
num_languages = len(language_id_map.keys())
|
||||
|
||||
language_tone_start_map = {
|
||||
"ZH": 0,
|
||||
"JP": num_zh_tones,
|
||||
"EN": num_zh_tones + num_ja_tones,
|
||||
}
|
||||
|
||||
if __name__ == "__main__":
|
||||
a = set(zh_symbols)
|
||||
b = set(en_symbols)
|
||||
print(sorted(a & b))
|
||||
769
oldVersion/V200/text/tone_sandhi.py
Normal file
769
oldVersion/V200/text/tone_sandhi.py
Normal file
@@ -0,0 +1,769 @@
|
||||
# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
from typing import List
|
||||
from typing import Tuple
|
||||
|
||||
import jieba
|
||||
from pypinyin import lazy_pinyin
|
||||
from pypinyin import Style
|
||||
|
||||
|
||||
class ToneSandhi:
|
||||
def __init__(self):
|
||||
self.must_neural_tone_words = {
|
||||
"麻烦",
|
||||
"麻利",
|
||||
"鸳鸯",
|
||||
"高粱",
|
||||
"骨头",
|
||||
"骆驼",
|
||||
"马虎",
|
||||
"首饰",
|
||||
"馒头",
|
||||
"馄饨",
|
||||
"风筝",
|
||||
"难为",
|
||||
"队伍",
|
||||
"阔气",
|
||||
"闺女",
|
||||
"门道",
|
||||
"锄头",
|
||||
"铺盖",
|
||||
"铃铛",
|
||||
"铁匠",
|
||||
"钥匙",
|
||||
"里脊",
|
||||
"里头",
|
||||
"部分",
|
||||
"那么",
|
||||
"道士",
|
||||
"造化",
|
||||
"迷糊",
|
||||
"连累",
|
||||
"这么",
|
||||
"这个",
|
||||
"运气",
|
||||
"过去",
|
||||
"软和",
|
||||
"转悠",
|
||||
"踏实",
|
||||
"跳蚤",
|
||||
"跟头",
|
||||
"趔趄",
|
||||
"财主",
|
||||
"豆腐",
|
||||
"讲究",
|
||||
"记性",
|
||||
"记号",
|
||||
"认识",
|
||||
"规矩",
|
||||
"见识",
|
||||
"裁缝",
|
||||
"补丁",
|
||||
"衣裳",
|
||||
"衣服",
|
||||
"衙门",
|
||||
"街坊",
|
||||
"行李",
|
||||
"行当",
|
||||
"蛤蟆",
|
||||
"蘑菇",
|
||||
"薄荷",
|
||||
"葫芦",
|
||||
"葡萄",
|
||||
"萝卜",
|
||||
"荸荠",
|
||||
"苗条",
|
||||
"苗头",
|
||||
"苍蝇",
|
||||
"芝麻",
|
||||
"舒服",
|
||||
"舒坦",
|
||||
"舌头",
|
||||
"自在",
|
||||
"膏药",
|
||||
"脾气",
|
||||
"脑袋",
|
||||
"脊梁",
|
||||
"能耐",
|
||||
"胳膊",
|
||||
"胭脂",
|
||||
"胡萝",
|
||||
"胡琴",
|
||||
"胡同",
|
||||
"聪明",
|
||||
"耽误",
|
||||
"耽搁",
|
||||
"耷拉",
|
||||
"耳朵",
|
||||
"老爷",
|
||||
"老实",
|
||||
"老婆",
|
||||
"老头",
|
||||
"老太",
|
||||
"翻腾",
|
||||
"罗嗦",
|
||||
"罐头",
|
||||
"编辑",
|
||||
"结实",
|
||||
"红火",
|
||||
"累赘",
|
||||
"糨糊",
|
||||
"糊涂",
|
||||
"精神",
|
||||
"粮食",
|
||||
"簸箕",
|
||||
"篱笆",
|
||||
"算计",
|
||||
"算盘",
|
||||
"答应",
|
||||
"笤帚",
|
||||
"笑语",
|
||||
"笑话",
|
||||
"窟窿",
|
||||
"窝囊",
|
||||
"窗户",
|
||||
"稳当",
|
||||
"稀罕",
|
||||
"称呼",
|
||||
"秧歌",
|
||||
"秀气",
|
||||
"秀才",
|
||||
"福气",
|
||||
"祖宗",
|
||||
"砚台",
|
||||
"码头",
|
||||
"石榴",
|
||||
"石头",
|
||||
"石匠",
|
||||
"知识",
|
||||
"眼睛",
|
||||
"眯缝",
|
||||
"眨巴",
|
||||
"眉毛",
|
||||
"相声",
|
||||
"盘算",
|
||||
"白净",
|
||||
"痢疾",
|
||||
"痛快",
|
||||
"疟疾",
|
||||
"疙瘩",
|
||||
"疏忽",
|
||||
"畜生",
|
||||
"生意",
|
||||
"甘蔗",
|
||||
"琵琶",
|
||||
"琢磨",
|
||||
"琉璃",
|
||||
"玻璃",
|
||||
"玫瑰",
|
||||
"玄乎",
|
||||
"狐狸",
|
||||
"状元",
|
||||
"特务",
|
||||
"牲口",
|
||||
"牙碜",
|
||||
"牌楼",
|
||||
"爽快",
|
||||
"爱人",
|
||||
"热闹",
|
||||
"烧饼",
|
||||
"烟筒",
|
||||
"烂糊",
|
||||
"点心",
|
||||
"炊帚",
|
||||
"灯笼",
|
||||
"火候",
|
||||
"漂亮",
|
||||
"滑溜",
|
||||
"溜达",
|
||||
"温和",
|
||||
"清楚",
|
||||
"消息",
|
||||
"浪头",
|
||||
"活泼",
|
||||
"比方",
|
||||
"正经",
|
||||
"欺负",
|
||||
"模糊",
|
||||
"槟榔",
|
||||
"棺材",
|
||||
"棒槌",
|
||||
"棉花",
|
||||
"核桃",
|
||||
"栅栏",
|
||||
"柴火",
|
||||
"架势",
|
||||
"枕头",
|
||||
"枇杷",
|
||||
"机灵",
|
||||
"本事",
|
||||
"木头",
|
||||
"木匠",
|
||||
"朋友",
|
||||
"月饼",
|
||||
"月亮",
|
||||
"暖和",
|
||||
"明白",
|
||||
"时候",
|
||||
"新鲜",
|
||||
"故事",
|
||||
"收拾",
|
||||
"收成",
|
||||
"提防",
|
||||
"挖苦",
|
||||
"挑剔",
|
||||
"指甲",
|
||||
"指头",
|
||||
"拾掇",
|
||||
"拳头",
|
||||
"拨弄",
|
||||
"招牌",
|
||||
"招呼",
|
||||
"抬举",
|
||||
"护士",
|
||||
"折腾",
|
||||
"扫帚",
|
||||
"打量",
|
||||
"打算",
|
||||
"打点",
|
||||
"打扮",
|
||||
"打听",
|
||||
"打发",
|
||||
"扎实",
|
||||
"扁担",
|
||||
"戒指",
|
||||
"懒得",
|
||||
"意识",
|
||||
"意思",
|
||||
"情形",
|
||||
"悟性",
|
||||
"怪物",
|
||||
"思量",
|
||||
"怎么",
|
||||
"念头",
|
||||
"念叨",
|
||||
"快活",
|
||||
"忙活",
|
||||
"志气",
|
||||
"心思",
|
||||
"得罪",
|
||||
"张罗",
|
||||
"弟兄",
|
||||
"开通",
|
||||
"应酬",
|
||||
"庄稼",
|
||||
"干事",
|
||||
"帮手",
|
||||
"帐篷",
|
||||
"希罕",
|
||||
"师父",
|
||||
"师傅",
|
||||
"巴结",
|
||||
"巴掌",
|
||||
"差事",
|
||||
"工夫",
|
||||
"岁数",
|
||||
"屁股",
|
||||
"尾巴",
|
||||
"少爷",
|
||||
"小气",
|
||||
"小伙",
|
||||
"将就",
|
||||
"对头",
|
||||
"对付",
|
||||
"寡妇",
|
||||
"家伙",
|
||||
"客气",
|
||||
"实在",
|
||||
"官司",
|
||||
"学问",
|
||||
"学生",
|
||||
"字号",
|
||||
"嫁妆",
|
||||
"媳妇",
|
||||
"媒人",
|
||||
"婆家",
|
||||
"娘家",
|
||||
"委屈",
|
||||
"姑娘",
|
||||
"姐夫",
|
||||
"妯娌",
|
||||
"妥当",
|
||||
"妖精",
|
||||
"奴才",
|
||||
"女婿",
|
||||
"头发",
|
||||
"太阳",
|
||||
"大爷",
|
||||
"大方",
|
||||
"大意",
|
||||
"大夫",
|
||||
"多少",
|
||||
"多么",
|
||||
"外甥",
|
||||
"壮实",
|
||||
"地道",
|
||||
"地方",
|
||||
"在乎",
|
||||
"困难",
|
||||
"嘴巴",
|
||||
"嘱咐",
|
||||
"嘟囔",
|
||||
"嘀咕",
|
||||
"喜欢",
|
||||
"喇嘛",
|
||||
"喇叭",
|
||||
"商量",
|
||||
"唾沫",
|
||||
"哑巴",
|
||||
"哈欠",
|
||||
"哆嗦",
|
||||
"咳嗽",
|
||||
"和尚",
|
||||
"告诉",
|
||||
"告示",
|
||||
"含糊",
|
||||
"吓唬",
|
||||
"后头",
|
||||
"名字",
|
||||
"名堂",
|
||||
"合同",
|
||||
"吆喝",
|
||||
"叫唤",
|
||||
"口袋",
|
||||
"厚道",
|
||||
"厉害",
|
||||
"千斤",
|
||||
"包袱",
|
||||
"包涵",
|
||||
"匀称",
|
||||
"勤快",
|
||||
"动静",
|
||||
"动弹",
|
||||
"功夫",
|
||||
"力气",
|
||||
"前头",
|
||||
"刺猬",
|
||||
"刺激",
|
||||
"别扭",
|
||||
"利落",
|
||||
"利索",
|
||||
"利害",
|
||||
"分析",
|
||||
"出息",
|
||||
"凑合",
|
||||
"凉快",
|
||||
"冷战",
|
||||
"冤枉",
|
||||
"冒失",
|
||||
"养活",
|
||||
"关系",
|
||||
"先生",
|
||||
"兄弟",
|
||||
"便宜",
|
||||
"使唤",
|
||||
"佩服",
|
||||
"作坊",
|
||||
"体面",
|
||||
"位置",
|
||||
"似的",
|
||||
"伙计",
|
||||
"休息",
|
||||
"什么",
|
||||
"人家",
|
||||
"亲戚",
|
||||
"亲家",
|
||||
"交情",
|
||||
"云彩",
|
||||
"事情",
|
||||
"买卖",
|
||||
"主意",
|
||||
"丫头",
|
||||
"丧气",
|
||||
"两口",
|
||||
"东西",
|
||||
"东家",
|
||||
"世故",
|
||||
"不由",
|
||||
"不在",
|
||||
"下水",
|
||||
"下巴",
|
||||
"上头",
|
||||
"上司",
|
||||
"丈夫",
|
||||
"丈人",
|
||||
"一辈",
|
||||
"那个",
|
||||
"菩萨",
|
||||
"父亲",
|
||||
"母亲",
|
||||
"咕噜",
|
||||
"邋遢",
|
||||
"费用",
|
||||
"冤家",
|
||||
"甜头",
|
||||
"介绍",
|
||||
"荒唐",
|
||||
"大人",
|
||||
"泥鳅",
|
||||
"幸福",
|
||||
"熟悉",
|
||||
"计划",
|
||||
"扑腾",
|
||||
"蜡烛",
|
||||
"姥爷",
|
||||
"照顾",
|
||||
"喉咙",
|
||||
"吉他",
|
||||
"弄堂",
|
||||
"蚂蚱",
|
||||
"凤凰",
|
||||
"拖沓",
|
||||
"寒碜",
|
||||
"糟蹋",
|
||||
"倒腾",
|
||||
"报复",
|
||||
"逻辑",
|
||||
"盘缠",
|
||||
"喽啰",
|
||||
"牢骚",
|
||||
"咖喱",
|
||||
"扫把",
|
||||
"惦记",
|
||||
}
|
||||
self.must_not_neural_tone_words = {
|
||||
"男子",
|
||||
"女子",
|
||||
"分子",
|
||||
"原子",
|
||||
"量子",
|
||||
"莲子",
|
||||
"石子",
|
||||
"瓜子",
|
||||
"电子",
|
||||
"人人",
|
||||
"虎虎",
|
||||
}
|
||||
self.punc = "：，；。？！“”‘’':,;.?!"
|
||||
|
||||
# the meaning of jieba pos tag: https://blog.csdn.net/weixin_44174352/article/details/113731041
|
||||
# e.g.
|
||||
# word: "家里"
|
||||
# pos: "s"
|
||||
# finals: ['ia1', 'i3']
|
||||
def _neural_sandhi(self, word: str, pos: str, finals: List[str]) -> List[str]:
|
||||
# reduplication words for n. and v. e.g. 奶奶, 试试, 旺旺
|
||||
for j, item in enumerate(word):
|
||||
if (
|
||||
j - 1 >= 0
|
||||
and item == word[j - 1]
|
||||
and pos[0] in {"n", "v", "a"}
|
||||
and word not in self.must_not_neural_tone_words
|
||||
):
|
||||
finals[j] = finals[j][:-1] + "5"
|
||||
ge_idx = word.find("个")
|
||||
if len(word) >= 1 and word[-1] in "吧呢啊呐噻嘛吖嗨呐哦哒额滴哩哟喽啰耶喔诶":
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
elif len(word) >= 1 and word[-1] in "的地得":
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
# e.g. 走了, 看着, 去过
|
||||
# elif len(word) == 1 and word in "了着过" and pos in {"ul", "uz", "ug"}:
|
||||
# finals[-1] = finals[-1][:-1] + "5"
|
||||
elif (
|
||||
len(word) > 1
|
||||
and word[-1] in "们子"
|
||||
and pos in {"r", "n"}
|
||||
and word not in self.must_not_neural_tone_words
|
||||
):
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
# e.g. 桌上, 地下, 家里
|
||||
elif len(word) > 1 and word[-1] in "上下里" and pos in {"s", "l", "f"}:
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
# e.g. 上来, 下去
|
||||
elif len(word) > 1 and word[-1] in "来去" and word[-2] in "上下进出回过起开":
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
# 个做量词
|
||||
elif (
|
||||
ge_idx >= 1
|
||||
and (word[ge_idx - 1].isnumeric() or word[ge_idx - 1] in "几有两半多各整每做是")
|
||||
) or word == "个":
|
||||
finals[ge_idx] = finals[ge_idx][:-1] + "5"
|
||||
else:
|
||||
if (
|
||||
word in self.must_neural_tone_words
|
||||
or word[-2:] in self.must_neural_tone_words
|
||||
):
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
|
||||
word_list = self._split_word(word)
|
||||
finals_list = [finals[: len(word_list[0])], finals[len(word_list[0]) :]]
|
||||
for i, word in enumerate(word_list):
|
||||
# conventional neural in Chinese
|
||||
if (
|
||||
word in self.must_neural_tone_words
|
||||
or word[-2:] in self.must_neural_tone_words
|
||||
):
|
||||
finals_list[i][-1] = finals_list[i][-1][:-1] + "5"
|
||||
finals = sum(finals_list, [])
|
||||
return finals
|
||||
|
||||
def _bu_sandhi(self, word: str, finals: List[str]) -> List[str]:
|
||||
# e.g. 看不懂
|
||||
if len(word) == 3 and word[1] == "不":
|
||||
finals[1] = finals[1][:-1] + "5"
|
||||
else:
|
||||
for i, char in enumerate(word):
|
||||
# "不" before tone4 should be bu2, e.g. 不怕
|
||||
if char == "不" and i + 1 < len(word) and finals[i + 1][-1] == "4":
|
||||
finals[i] = finals[i][:-1] + "2"
|
||||
return finals
|
||||
|
||||
def _yi_sandhi(self, word: str, finals: List[str]) -> List[str]:
|
||||
# "一" in number sequences, e.g. 一零零, 二一零
|
||||
if word.find("一") != -1 and all(
|
||||
[item.isnumeric() for item in word if item != "一"]
|
||||
):
|
||||
return finals
|
||||
# "一" between reduplication words should be yi5, e.g. 看一看
|
||||
elif len(word) == 3 and word[1] == "一" and word[0] == word[-1]:
|
||||
finals[1] = finals[1][:-1] + "5"
|
||||
# when "一" is ordinal word, it should be yi1
|
||||
elif word.startswith("第一"):
|
||||
finals[1] = finals[1][:-1] + "1"
|
||||
else:
|
||||
for i, char in enumerate(word):
|
||||
if char == "一" and i + 1 < len(word):
|
||||
# "一" before tone4 should be yi2, e.g. 一段
|
||||
if finals[i + 1][-1] == "4":
|
||||
finals[i] = finals[i][:-1] + "2"
|
||||
# "一" before non-tone4 should be yi4, e.g. 一天
|
||||
else:
|
||||
# "一" 后面如果是标点，还读一声
|
||||
if word[i + 1] not in self.punc:
|
||||
finals[i] = finals[i][:-1] + "4"
|
||||
return finals
|
||||
|
||||
def _split_word(self, word: str) -> List[str]:
|
||||
word_list = jieba.cut_for_search(word)
|
||||
word_list = sorted(word_list, key=lambda i: len(i), reverse=False)
|
||||
first_subword = word_list[0]
|
||||
first_begin_idx = word.find(first_subword)
|
||||
if first_begin_idx == 0:
|
||||
second_subword = word[len(first_subword) :]
|
||||
new_word_list = [first_subword, second_subword]
|
||||
else:
|
||||
second_subword = word[: -len(first_subword)]
|
||||
new_word_list = [second_subword, first_subword]
|
||||
return new_word_list
|
||||
|
||||
def _three_sandhi(self, word: str, finals: List[str]) -> List[str]:
|
||||
if len(word) == 2 and self._all_tone_three(finals):
|
||||
finals[0] = finals[0][:-1] + "2"
|
||||
elif len(word) == 3:
|
||||
word_list = self._split_word(word)
|
||||
if self._all_tone_three(finals):
|
||||
# disyllabic + monosyllabic, e.g. 蒙古/包
|
||||
if len(word_list[0]) == 2:
|
||||
finals[0] = finals[0][:-1] + "2"
|
||||
finals[1] = finals[1][:-1] + "2"
|
||||
# monosyllabic + disyllabic, e.g. 纸/老虎
|
||||
elif len(word_list[0]) == 1:
|
||||
finals[1] = finals[1][:-1] + "2"
|
||||
else:
|
||||
finals_list = [finals[: len(word_list[0])], finals[len(word_list[0]) :]]
|
||||
if len(finals_list) == 2:
|
||||
for i, sub in enumerate(finals_list):
|
||||
# e.g. 所有/人
|
||||
if self._all_tone_three(sub) and len(sub) == 2:
|
||||
finals_list[i][0] = finals_list[i][0][:-1] + "2"
|
||||
# e.g. 好/喜欢
|
||||
elif (
|
||||
i == 1
|
||||
and not self._all_tone_three(sub)
|
||||
and finals_list[i][0][-1] == "3"
|
||||
and finals_list[0][-1][-1] == "3"
|
||||
):
|
||||
finals_list[0][-1] = finals_list[0][-1][:-1] + "2"
|
||||
finals = sum(finals_list, [])
|
||||
# split idiom into two words who's length is 2
|
||||
elif len(word) == 4:
|
||||
finals_list = [finals[:2], finals[2:]]
|
||||
finals = []
|
||||
for sub in finals_list:
|
||||
if self._all_tone_three(sub):
|
||||
sub[0] = sub[0][:-1] + "2"
|
||||
finals += sub
|
||||
|
||||
return finals
|
||||
|
||||
def _all_tone_three(self, finals: List[str]) -> bool:
|
||||
return all(x[-1] == "3" for x in finals)
|
||||
|
||||
# merge "不" and the word behind it
|
||||
# if don't merge, "不" sometimes appears alone according to jieba, which may occur sandhi error
|
||||
def _merge_bu(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
last_word = ""
|
||||
for word, pos in seg:
|
||||
if last_word == "不":
|
||||
word = last_word + word
|
||||
if word != "不":
|
||||
new_seg.append((word, pos))
|
||||
last_word = word[:]
|
||||
if last_word == "不":
|
||||
new_seg.append((last_word, "d"))
|
||||
last_word = ""
|
||||
return new_seg
|
||||
|
||||
# function 1: merge "一" and reduplication words in it's left and right, e.g. "听","一","听" ->"听一听"
|
||||
# function 2: merge single "一" and the word behind it
|
||||
# if don't merge, "一" sometimes appears alone according to jieba, which may occur sandhi error
|
||||
# e.g.
|
||||
# input seg: [('听', 'v'), ('一', 'm'), ('听', 'v')]
|
||||
# output seg: [['听一听', 'v']]
|
||||
def _merge_yi(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
# function 1
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if (
|
||||
i - 1 >= 0
|
||||
and word == "一"
|
||||
and i + 1 < len(seg)
|
||||
and seg[i - 1][0] == seg[i + 1][0]
|
||||
and seg[i - 1][1] == "v"
|
||||
):
|
||||
new_seg[i - 1][0] = new_seg[i - 1][0] + "一" + new_seg[i - 1][0]
|
||||
else:
|
||||
if (
|
||||
i - 2 >= 0
|
||||
and seg[i - 1][0] == "一"
|
||||
and seg[i - 2][0] == word
|
||||
and pos == "v"
|
||||
):
|
||||
continue
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
seg = new_seg
|
||||
new_seg = []
|
||||
# function 2
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if new_seg and new_seg[-1][0] == "一":
|
||||
new_seg[-1][0] = new_seg[-1][0] + word
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
return new_seg
|
||||
|
||||
# the first and the second words are all_tone_three
|
||||
def _merge_continuous_three_tones(
|
||||
self, seg: List[Tuple[str, str]]
|
||||
) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
sub_finals_list = [
|
||||
lazy_pinyin(word, neutral_tone_with_five=True, style=Style.FINALS_TONE3)
|
||||
for (word, pos) in seg
|
||||
]
|
||||
assert len(sub_finals_list) == len(seg)
|
||||
merge_last = [False] * len(seg)
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if (
|
||||
i - 1 >= 0
|
||||
and self._all_tone_three(sub_finals_list[i - 1])
|
||||
and self._all_tone_three(sub_finals_list[i])
|
||||
and not merge_last[i - 1]
|
||||
):
|
||||
# if the last word is reduplication, not merge, because reduplication need to be _neural_sandhi
|
||||
if (
|
||||
not self._is_reduplication(seg[i - 1][0])
|
||||
and len(seg[i - 1][0]) + len(seg[i][0]) <= 3
|
||||
):
|
||||
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
|
||||
merge_last[i] = True
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
|
||||
return new_seg
|
||||
|
||||
def _is_reduplication(self, word: str) -> bool:
|
||||
return len(word) == 2 and word[0] == word[1]
|
||||
|
||||
# the last char of first word and the first char of second word is tone_three
|
||||
def _merge_continuous_three_tones_2(
|
||||
self, seg: List[Tuple[str, str]]
|
||||
) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
sub_finals_list = [
|
||||
lazy_pinyin(word, neutral_tone_with_five=True, style=Style.FINALS_TONE3)
|
||||
for (word, pos) in seg
|
||||
]
|
||||
assert len(sub_finals_list) == len(seg)
|
||||
merge_last = [False] * len(seg)
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if (
|
||||
i - 1 >= 0
|
||||
and sub_finals_list[i - 1][-1][-1] == "3"
|
||||
and sub_finals_list[i][0][-1] == "3"
|
||||
and not merge_last[i - 1]
|
||||
):
|
||||
# if the last word is reduplication, not merge, because reduplication need to be _neural_sandhi
|
||||
if (
|
||||
not self._is_reduplication(seg[i - 1][0])
|
||||
and len(seg[i - 1][0]) + len(seg[i][0]) <= 3
|
||||
):
|
||||
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
|
||||
merge_last[i] = True
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
return new_seg
|
||||
|
||||
def _merge_er(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if i - 1 >= 0 and word == "儿" and seg[i - 1][0] != "#":
|
||||
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
return new_seg
|
||||
|
||||
def _merge_reduplication(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if new_seg and word == new_seg[-1][0]:
|
||||
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
return new_seg
|
||||
|
||||
def pre_merge_for_modify(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
seg = self._merge_bu(seg)
|
||||
try:
|
||||
seg = self._merge_yi(seg)
|
||||
except:
|
||||
print("_merge_yi failed")
|
||||
seg = self._merge_reduplication(seg)
|
||||
seg = self._merge_continuous_three_tones(seg)
|
||||
seg = self._merge_continuous_three_tones_2(seg)
|
||||
seg = self._merge_er(seg)
|
||||
return seg
|
||||
|
||||
def modified_tone(self, word: str, pos: str, finals: List[str]) -> List[str]:
|
||||
finals = self._bu_sandhi(word, finals)
|
||||
finals = self._yi_sandhi(word, finals)
|
||||
finals = self._neural_sandhi(word, pos, finals)
|
||||
finals = self._three_sandhi(word, finals)
|
||||
return finals
|
||||
@@ -176,6 +176,80 @@ if __name__ == "__main__":
|
||||
class Text(BaseModel):
|
||||
text: str
|
||||
|
||||
def _voice(
|
||||
text: str,
|
||||
model_id: int,
|
||||
speaker_name: str,
|
||||
speaker_id: int,
|
||||
sdp_ratio: float,
|
||||
noise: float,
|
||||
noisew: float,
|
||||
length: float,
|
||||
language: str,
|
||||
auto_translate: bool,
|
||||
auto_split: bool,
|
||||
) -> Response | Dict[str, any]:
|
||||
# 检查模型是否存在
|
||||
if model_id not in loaded_models.models.keys():
|
||||
return {"status": 10, "detail": f"模型model_id={model_id}未加载"}
|
||||
# 检查是否提供speaker
|
||||
if speaker_name is None and speaker_id is None:
|
||||
return {"status": 11, "detail": "请提供speaker_name或speaker_id"}
|
||||
elif speaker_name is None:
|
||||
# 检查speaker_id是否存在
|
||||
if speaker_id not in loaded_models.models[model_id].id2spk.keys():
|
||||
return {"status": 12, "detail": f"角色speaker_id={speaker_id}不存在"}
|
||||
speaker_name = loaded_models.models[model_id].id2spk[speaker_id]
|
||||
# 检查speaker_name是否存在
|
||||
if speaker_name not in loaded_models.models[model_id].spk2id.keys():
|
||||
return {"status": 13, "detail": f"角色speaker_name={speaker_name}不存在"}
|
||||
if language is None:
|
||||
language = loaded_models.models[model_id].language
|
||||
if auto_translate:
|
||||
text = trans.translate(Sentence=text, to_Language=language.lower())
|
||||
if not auto_split:
|
||||
with torch.no_grad():
|
||||
audio = infer(
|
||||
text=text,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise,
|
||||
noise_scale_w=noisew,
|
||||
length_scale=length,
|
||||
sid=speaker_name,
|
||||
language=language,
|
||||
hps=loaded_models.models[model_id].hps,
|
||||
net_g=loaded_models.models[model_id].net_g,
|
||||
device=loaded_models.models[model_id].device,
|
||||
)
|
||||
else:
|
||||
texts = cut_sent(text)
|
||||
audios = []
|
||||
with torch.no_grad():
|
||||
for t in texts:
|
||||
audios.append(
|
||||
infer(
|
||||
text=t,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise,
|
||||
noise_scale_w=noisew,
|
||||
length_scale=length,
|
||||
sid=speaker_name,
|
||||
language=language,
|
||||
hps=loaded_models.models[model_id].hps,
|
||||
net_g=loaded_models.models[model_id].net_g,
|
||||
device=loaded_models.models[model_id].device,
|
||||
)
|
||||
)
|
||||
audios.append(np.zeros(int(44100 * 0.2)))
|
||||
audio = np.concatenate(audios)
|
||||
audio = gradio.processing_utils.convert_to_16_bit_wav(audio)
|
||||
wavContent = BytesIO()
|
||||
wavfile.write(
|
||||
wavContent, loaded_models.models[model_id].hps.data.sampling_rate, audio
|
||||
)
|
||||
response = Response(content=wavContent.getvalue(), media_type="audio/wav")
|
||||
return response
|
||||
|
||||
@app.post("/voice")
|
||||
def voice(
|
||||
request: Request, # fastapi自动注入
|
||||
@@ -198,66 +272,19 @@ if __name__ == "__main__":
|
||||
logger.info(
|
||||
f"{request.client.host}:{request.client.port}/voice { unquote(str(request.query_params) )} text={text}"
|
||||
)
|
||||
# 检查模型是否存在
|
||||
if model_id not in loaded_models.models.keys():
|
||||
return {"status": 10, "detail": f"模型model_id={model_id}未加载"}
|
||||
# 检查是否提供speaker
|
||||
if speaker_name is None and speaker_id is None:
|
||||
return {"status": 11, "detail": "请提供speaker_name或speaker_id"}
|
||||
elif speaker_name is None:
|
||||
# 检查speaker_id是否存在
|
||||
if speaker_id not in loaded_models.models[model_id].id2spk.keys():
|
||||
return {"status": 12, "detail": f"角色speaker_id={speaker_id}不存在"}
|
||||
speaker_name = loaded_models.models[model_id].id2spk[speaker_id]
|
||||
# 检查speaker_name是否存在
|
||||
if speaker_name not in loaded_models.models[model_id].spk2id.keys():
|
||||
return {"status": 13, "detail": f"角色speaker_name={speaker_name}不存在"}
|
||||
if language is None:
|
||||
language = loaded_models.models[model_id].language
|
||||
if auto_translate:
|
||||
text = trans.translate(Sentence=text, to_Language=language.lower())
|
||||
if not auto_split:
|
||||
with torch.no_grad():
|
||||
audio = infer(
|
||||
text=text,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise,
|
||||
noise_scale_w=noisew,
|
||||
length_scale=length,
|
||||
sid=speaker_name,
|
||||
language=language,
|
||||
hps=loaded_models.models[model_id].hps,
|
||||
net_g=loaded_models.models[model_id].net_g,
|
||||
device=loaded_models.models[model_id].device,
|
||||
)
|
||||
else:
|
||||
texts = cut_sent(text)
|
||||
audios = []
|
||||
with torch.no_grad():
|
||||
for t in texts:
|
||||
audios.append(
|
||||
infer(
|
||||
text=t,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise,
|
||||
noise_scale_w=noisew,
|
||||
length_scale=length,
|
||||
sid=speaker_name,
|
||||
language=language,
|
||||
hps=loaded_models.models[model_id].hps,
|
||||
net_g=loaded_models.models[model_id].net_g,
|
||||
device=loaded_models.models[model_id].device,
|
||||
)
|
||||
)
|
||||
audios.append(np.zeros((int)(44100 * 0.3)))
|
||||
audio = np.concatenate(audios)
|
||||
audio = gradio.processing_utils.convert_to_16_bit_wav(audio)
|
||||
wavContent = BytesIO()
|
||||
wavfile.write(
|
||||
wavContent, loaded_models.models[model_id].hps.data.sampling_rate, audio
|
||||
return _voice(
|
||||
text=text,
|
||||
model_id=model_id,
|
||||
speaker_name=speaker_name,
|
||||
speaker_id=speaker_id,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise=noise,
|
||||
noisew=noisew,
|
||||
length=length,
|
||||
language=language,
|
||||
auto_translate=auto_translate,
|
||||
auto_split=auto_split,
|
||||
)
|
||||
response = Response(content=wavContent.getvalue(), media_type="audio/wav")
|
||||
return response
|
||||
|
||||
@app.get("/voice")
|
||||
def voice(
|
||||
@@ -280,66 +307,19 @@ if __name__ == "__main__":
|
||||
logger.info(
|
||||
f"{request.client.host}:{request.client.port}/voice { unquote(str(request.query_params) )}"
|
||||
)
|
||||
# 检查模型是否存在
|
||||
if model_id not in loaded_models.models.keys():
|
||||
return {"status": 10, "detail": f"模型model_id={model_id}未加载"}
|
||||
# 检查是否提供speaker
|
||||
if speaker_name is None and speaker_id is None:
|
||||
return {"status": 11, "detail": "请提供speaker_name或speaker_id"}
|
||||
elif speaker_name is None:
|
||||
# 检查speaker_id是否存在
|
||||
if speaker_id not in loaded_models.models[model_id].id2spk.keys():
|
||||
return {"status": 12, "detail": f"角色speaker_id={speaker_id}不存在"}
|
||||
speaker_name = loaded_models.models[model_id].id2spk[speaker_id]
|
||||
# 检查speaker_name是否存在
|
||||
if speaker_name not in loaded_models.models[model_id].spk2id.keys():
|
||||
return {"status": 13, "detail": f"角色speaker_name={speaker_name}不存在"}
|
||||
if language is None:
|
||||
language = loaded_models.models[model_id].language
|
||||
if auto_translate:
|
||||
text = trans.translate(Sentence=text, to_Language=language.lower())
|
||||
if not auto_split:
|
||||
with torch.no_grad():
|
||||
audio = infer(
|
||||
text=text,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise,
|
||||
noise_scale_w=noisew,
|
||||
length_scale=length,
|
||||
sid=speaker_name,
|
||||
language=language,
|
||||
hps=loaded_models.models[model_id].hps,
|
||||
net_g=loaded_models.models[model_id].net_g,
|
||||
device=loaded_models.models[model_id].device,
|
||||
)
|
||||
else:
|
||||
texts = cut_sent(text)
|
||||
audios = []
|
||||
with torch.no_grad():
|
||||
for t in texts:
|
||||
audios.append(
|
||||
infer(
|
||||
text=t,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise,
|
||||
noise_scale_w=noisew,
|
||||
length_scale=length,
|
||||
sid=speaker_name,
|
||||
language=language,
|
||||
hps=loaded_models.models[model_id].hps,
|
||||
net_g=loaded_models.models[model_id].net_g,
|
||||
device=loaded_models.models[model_id].device,
|
||||
)
|
||||
)
|
||||
audios.append(np.zeros((int)(44100 * 0.3)))
|
||||
audio = np.concatenate(audios)
|
||||
audio = gradio.processing_utils.convert_to_16_bit_wav(audio)
|
||||
wavContent = BytesIO()
|
||||
wavfile.write(
|
||||
wavContent, loaded_models.models[model_id].hps.data.sampling_rate, audio
|
||||
return _voice(
|
||||
text=text,
|
||||
model_id=model_id,
|
||||
speaker_name=speaker_name,
|
||||
speaker_id=speaker_id,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise=noise,
|
||||
noisew=noisew,
|
||||
length=length,
|
||||
language=language,
|
||||
auto_translate=auto_translate,
|
||||
auto_split=auto_split,
|
||||
)
|
||||
response = Response(content=wavContent.getvalue(), media_type="audio/wav")
|
||||
return response
|
||||
|
||||
@app.get("/models/info")
|
||||
def get_loaded_models_info(request: Request):
|
||||
|
||||
87
spec_gen.py
Normal file
87
spec_gen.py
Normal file
@@ -0,0 +1,87 @@
|
||||
import torch
|
||||
from tqdm import tqdm
|
||||
from multiprocessing import Pool
|
||||
from mel_processing import spectrogram_torch, mel_spectrogram_torch
|
||||
from utils import load_wav_to_torch
|
||||
|
||||
|
||||
class AudioProcessor:
|
||||
def __init__(
|
||||
self,
|
||||
max_wav_value,
|
||||
use_mel_spec_posterior,
|
||||
filter_length,
|
||||
n_mel_channels,
|
||||
sampling_rate,
|
||||
hop_length,
|
||||
win_length,
|
||||
mel_fmin,
|
||||
mel_fmax,
|
||||
):
|
||||
self.max_wav_value = max_wav_value
|
||||
self.use_mel_spec_posterior = use_mel_spec_posterior
|
||||
self.filter_length = filter_length
|
||||
self.n_mel_channels = n_mel_channels
|
||||
self.sampling_rate = sampling_rate
|
||||
self.hop_length = hop_length
|
||||
self.win_length = win_length
|
||||
self.mel_fmin = mel_fmin
|
||||
self.mel_fmax = mel_fmax
|
||||
|
||||
def process_audio(self, filename):
|
||||
audio, sampling_rate = load_wav_to_torch(filename)
|
||||
audio_norm = audio / self.max_wav_value
|
||||
audio_norm = audio_norm.unsqueeze(0)
|
||||
spec_filename = filename.replace(".wav", ".spec.pt")
|
||||
if self.use_mel_spec_posterior:
|
||||
spec_filename = spec_filename.replace(".spec.pt", ".mel.pt")
|
||||
try:
|
||||
spec = torch.load(spec_filename)
|
||||
except:
|
||||
if self.use_mel_spec_posterior:
|
||||
spec = mel_spectrogram_torch(
|
||||
audio_norm,
|
||||
self.filter_length,
|
||||
self.n_mel_channels,
|
||||
self.sampling_rate,
|
||||
self.hop_length,
|
||||
self.win_length,
|
||||
self.mel_fmin,
|
||||
self.mel_fmax,
|
||||
center=False,
|
||||
)
|
||||
else:
|
||||
spec = spectrogram_torch(
|
||||
audio_norm,
|
||||
self.filter_length,
|
||||
self.sampling_rate,
|
||||
self.hop_length,
|
||||
self.win_length,
|
||||
center=False,
|
||||
)
|
||||
spec = torch.squeeze(spec, 0)
|
||||
torch.save(spec, spec_filename)
|
||||
return spec, audio_norm
|
||||
|
||||
|
||||
# 使用示例
|
||||
processor = AudioProcessor(
|
||||
max_wav_value=32768.0,
|
||||
use_mel_spec_posterior=False,
|
||||
filter_length=2048,
|
||||
n_mel_channels=128,
|
||||
sampling_rate=44100,
|
||||
hop_length=512,
|
||||
win_length=2048,
|
||||
mel_fmin=0.0,
|
||||
mel_fmax="null",
|
||||
)
|
||||
|
||||
with open("filelists/train.list", "r") as f:
|
||||
filepaths = [line.split("|")[0] for line in f] # 取每一行的第一部分作为audiopath
|
||||
|
||||
# 使用多进程处理
|
||||
with Pool(processes=32) as pool: # 使用4个进程
|
||||
with tqdm(total=len(filepaths)) as pbar:
|
||||
for i, _ in enumerate(pool.imap_unordered(processor.process_audio, filepaths)):
|
||||
pbar.update()
|
||||
@@ -30,6 +30,7 @@ rep_map = {
|
||||
"$": ".",
|
||||
"“": "'",
|
||||
"”": "'",
|
||||
'"': "'",
|
||||
"‘": "'",
|
||||
"’": "'",
|
||||
"（": "'",
|
||||
|
||||
121
text/english.py
121
text/english.py
@@ -2,6 +2,7 @@ import pickle
|
||||
import os
|
||||
import re
|
||||
from g2p_en import G2p
|
||||
from transformers import DebertaV2Tokenizer
|
||||
|
||||
from text import symbols
|
||||
|
||||
@@ -9,6 +10,8 @@ current_file_path = os.path.dirname(__file__)
|
||||
CMU_DICT_PATH = os.path.join(current_file_path, "cmudict.rep")
|
||||
CACHE_PATH = os.path.join(current_file_path, "cmudict_cache.pickle")
|
||||
_g2p = G2p()
|
||||
LOCAL_PATH = "./bert/deberta-v3-large"
|
||||
tokenizer = DebertaV2Tokenizer.from_pretrained(LOCAL_PATH)
|
||||
|
||||
arpa = {
|
||||
"AH0",
|
||||
@@ -96,7 +99,9 @@ def post_replace_ph(ph):
|
||||
"\n": ".",
|
||||
"·": ",",
|
||||
"、": ",",
|
||||
"...": "…",
|
||||
"…": "...",
|
||||
"···": "...",
|
||||
"・・・": "...",
|
||||
"v": "V",
|
||||
}
|
||||
if ph in rep_map.keys():
|
||||
@@ -108,6 +113,62 @@ def post_replace_ph(ph):
|
||||
return ph
|
||||
|
||||
|
||||
rep_map = {
|
||||
"：": ",",
|
||||
"；": ",",
|
||||
"，": ",",
|
||||
"。": ".",
|
||||
"！": "!",
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"．": ".",
|
||||
"…": "...",
|
||||
"···": "...",
|
||||
"・・・": "...",
|
||||
"·": ",",
|
||||
"・": ",",
|
||||
"、": ",",
|
||||
"$": ".",
|
||||
"“": "'",
|
||||
"”": "'",
|
||||
'"': "'",
|
||||
"‘": "'",
|
||||
"’": "'",
|
||||
"（": "'",
|
||||
"）": "'",
|
||||
"(": "'",
|
||||
")": "'",
|
||||
"《": "'",
|
||||
"》": "'",
|
||||
"【": "'",
|
||||
"】": "'",
|
||||
"[": "'",
|
||||
"]": "'",
|
||||
"—": "-",
|
||||
"−": "-",
|
||||
"～": "-",
|
||||
"~": "-",
|
||||
"「": "'",
|
||||
"」": "'",
|
||||
}
|
||||
|
||||
|
||||
def replace_punctuation(text):
|
||||
pattern = re.compile("|".join(re.escape(p) for p in rep_map.keys()))
|
||||
|
||||
replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
|
||||
|
||||
# replaced_text = re.sub(
|
||||
# r"[^\u3040-\u309F\u30A0-\u30FF\u4E00-\u9FFF\u3400-\u4DBF\u3005"
|
||||
# + "".join(punctuation)
|
||||
# + r"]+",
|
||||
# "",
|
||||
# replaced_text,
|
||||
# )
|
||||
|
||||
return replaced_text
|
||||
|
||||
|
||||
def read_dict():
|
||||
g2p_dict = {}
|
||||
start_line = 49
|
||||
@@ -308,38 +369,68 @@ def normalize_numbers(text):
|
||||
|
||||
def text_normalize(text):
|
||||
text = normalize_numbers(text)
|
||||
text = replace_punctuation(text)
|
||||
text = re.sub(r"([,;.\?\!])([\w])", r"\1 \2", text)
|
||||
return text
|
||||
|
||||
|
||||
def distribute_phone(n_phone, n_word):
|
||||
phones_per_word = [0] * n_word
|
||||
for task in range(n_phone):
|
||||
min_tasks = min(phones_per_word)
|
||||
min_index = phones_per_word.index(min_tasks)
|
||||
phones_per_word[min_index] += 1
|
||||
return phones_per_word
|
||||
|
||||
|
||||
def sep_text(text):
|
||||
words = re.split(r"([,;.\?\!\s+])", text)
|
||||
words = [word for word in words if word.strip() != ""]
|
||||
return words
|
||||
|
||||
|
||||
def g2p(text):
|
||||
phones = []
|
||||
tones = []
|
||||
word2ph = []
|
||||
words = re.split(r"([,;.\-\?\!\s+])", text)
|
||||
words = [word for word in words if word.strip() != ""]
|
||||
# word2ph = []
|
||||
words = sep_text(text)
|
||||
tokens = [tokenizer.tokenize(i) for i in words]
|
||||
for word in words:
|
||||
if word.upper() in eng_dict:
|
||||
phns, tns = refine_syllables(eng_dict[word.upper()])
|
||||
phones += phns
|
||||
tones += tns
|
||||
word2ph.append(len(phns))
|
||||
phones.append([post_replace_ph(i) for i in phns])
|
||||
tones.append(tns)
|
||||
# word2ph.append(len(phns))
|
||||
else:
|
||||
phone_list = list(filter(lambda p: p != " ", _g2p(word)))
|
||||
phns = []
|
||||
tns = []
|
||||
for ph in phone_list:
|
||||
if ph in arpa:
|
||||
ph, tn = refine_ph(ph)
|
||||
phones.append(ph)
|
||||
tones.append(tn)
|
||||
phns.append(ph)
|
||||
tns.append(tn)
|
||||
else:
|
||||
phones.append(ph)
|
||||
tones.append(0)
|
||||
word2ph.append(len(phone_list))
|
||||
phns.append(ph)
|
||||
tns.append(0)
|
||||
phones.append([post_replace_ph(i) for i in phns])
|
||||
tones.append(tns)
|
||||
# word2ph.append(len(phns))
|
||||
# phones = [post_replace_ph(i) for i in phones]
|
||||
|
||||
phones = [post_replace_ph(i) for i in phones]
|
||||
word2ph = []
|
||||
for token, phoneme in zip(tokens, phones):
|
||||
phone_len = len(phoneme)
|
||||
word_len = len(token)
|
||||
|
||||
phones = ["_"] + phones + ["_"]
|
||||
tones = [0] + tones + [0]
|
||||
aaa = distribute_phone(phone_len, word_len)
|
||||
word2ph += aaa
|
||||
|
||||
phones = ["_"] + [j for i in phones for j in i] + ["_"]
|
||||
tones = [0] + [j for i in tones for j in i] + [0]
|
||||
word2ph = [1] + word2ph + [1]
|
||||
assert len(phones) == len(tones), text
|
||||
assert len(phones) == sum(word2ph), text
|
||||
|
||||
return phones, tones, word2ph
|
||||
|
||||
|
||||
@@ -30,7 +30,7 @@ def get_bert_feature(text, word2ph, device=config.bert_gen_config.device):
|
||||
inputs[i] = inputs[i].to(device)
|
||||
res = models[device](**inputs, output_hidden_states=True)
|
||||
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
|
||||
# assert len(word2ph) == len(text)+2
|
||||
assert len(word2ph) == res.shape[0], (text, res.shape[0], len(word2ph))
|
||||
word2phone = word2ph
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
|
||||
@@ -266,15 +266,16 @@ rep_map = {
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"．": ".",
|
||||
"...": "…",
|
||||
"···": "…",
|
||||
"・・・": "…",
|
||||
"…": "...",
|
||||
"···": "...",
|
||||
"・・・": "...",
|
||||
"·": ",",
|
||||
"・": ",",
|
||||
"、": ",",
|
||||
"$": ".",
|
||||
"“": "'",
|
||||
"”": "'",
|
||||
'"': "'",
|
||||
"‘": "'",
|
||||
"’": "'",
|
||||
"（": "'",
|
||||
@@ -317,6 +318,7 @@ def text_normalize(text):
|
||||
res = japanese_convert_numbers_to_words(res)
|
||||
# res = "".join([i for i in res if is_japanese_character(i)])
|
||||
res = replace_punctuation(res)
|
||||
res = res.replace("゙", "")
|
||||
return res
|
||||
|
||||
|
||||
@@ -340,7 +342,7 @@ def handle_long(sep_phonemes):
|
||||
return sep_phonemes
|
||||
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/deberta-v2-large-japanese")
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/deberta-v2-large-japanese-char-wwm")
|
||||
|
||||
|
||||
def align_tones(phones, tones):
|
||||
@@ -365,9 +367,35 @@ def align_tones(phones, tones):
|
||||
return res
|
||||
|
||||
|
||||
def rearrange_tones(tones, phones):
|
||||
res = [0] * len(tones)
|
||||
for i in range(len(tones)):
|
||||
if i == 0:
|
||||
if tones[i] not in punctuation:
|
||||
res[i] = 1
|
||||
elif tones[i] == prev:
|
||||
if phones[i] in punctuation:
|
||||
res[i] = 0
|
||||
else:
|
||||
res[i] = 1
|
||||
elif tones[i] > prev:
|
||||
res[i] = 2
|
||||
elif tones[i] < prev:
|
||||
res[i - 1] = 3
|
||||
res[i] = 1
|
||||
prev = tones[i]
|
||||
return res
|
||||
|
||||
|
||||
def g2p(norm_text):
|
||||
sep_text, sep_kata, acc = text2sep_kata(norm_text)
|
||||
sep_tokenized = [tokenizer.tokenize(i) for i in sep_text]
|
||||
sep_tokenized = []
|
||||
for i in sep_text:
|
||||
if i not in punctuation:
|
||||
sep_tokenized.append(tokenizer.tokenize(i))
|
||||
else:
|
||||
sep_tokenized.append([i])
|
||||
|
||||
sep_phonemes = handle_long([kata2phoneme(i) for i in sep_kata])
|
||||
# 异常处理，MeCab不认识的词的话会一路传到这里来，然后炸掉。目前来看只有那些超级稀有的生僻词会出现这种情况
|
||||
for i in sep_phonemes:
|
||||
@@ -383,6 +411,7 @@ def g2p(norm_text):
|
||||
aaa = distribute_phone(phone_len, word_len)
|
||||
word2ph += aaa
|
||||
phones = ["_"] + [j for i in sep_phonemes for j in i] + ["_"]
|
||||
# tones = [0] + rearrange_tones(tones, phones[1:-1]) + [0]
|
||||
tones = [0] + tones + [0]
|
||||
word2ph = [1] + word2ph + [1]
|
||||
assert len(phones) == len(tones)
|
||||
|
||||
@@ -6,7 +6,7 @@ from transformers import AutoModelForMaskedLM, AutoTokenizer
|
||||
from config import config
|
||||
from text.japanese import text2sep_kata
|
||||
|
||||
LOCAL_PATH = "./bert/deberta-v2-large-japanese"
|
||||
LOCAL_PATH = "./bert/deberta-v2-large-japanese-char-wwm"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(LOCAL_PATH)
|
||||
|
||||
@@ -14,14 +14,7 @@ models = dict()
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph, device=config.bert_gen_config.device):
|
||||
sep_text, _, _ = text2sep_kata(text)
|
||||
sep_tokens = [tokenizer.tokenize(t) for t in sep_text]
|
||||
sep_ids = [tokenizer.convert_tokens_to_ids(t) for t in sep_tokens]
|
||||
sep_ids = [2] + [item for sublist in sep_ids for item in sublist] + [3]
|
||||
return get_bert_feature_with_token(sep_ids, word2ph, device)
|
||||
|
||||
|
||||
def get_bert_feature_with_token(tokens, word2ph, device=config.bert_gen_config.device):
|
||||
text = "".join(text2sep_kata(text)[0])
|
||||
if (
|
||||
sys.platform == "darwin"
|
||||
and torch.backends.mps.is_available()
|
||||
@@ -33,20 +26,13 @@ def get_bert_feature_with_token(tokens, word2ph, device=config.bert_gen_config.d
|
||||
if device not in models.keys():
|
||||
models[device] = AutoModelForMaskedLM.from_pretrained(LOCAL_PATH).to(device)
|
||||
with torch.no_grad():
|
||||
inputs = torch.tensor(tokens).to(device).unsqueeze(0)
|
||||
token_type_ids = torch.zeros_like(inputs).to(device)
|
||||
attention_mask = torch.ones_like(inputs).to(device)
|
||||
inputs = {
|
||||
"input_ids": inputs,
|
||||
"token_type_ids": token_type_ids,
|
||||
"attention_mask": attention_mask,
|
||||
}
|
||||
|
||||
# for i in inputs:
|
||||
# inputs[i] = inputs[i].to(device)
|
||||
inputs = tokenizer(text, return_tensors="pt")
|
||||
for i in inputs:
|
||||
inputs[i] = inputs[i].to(device)
|
||||
res = models[device](**inputs, output_hidden_states=True)
|
||||
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
|
||||
assert inputs["input_ids"].shape[-1] == len(word2ph)
|
||||
|
||||
assert len(word2ph) == len(text) + 2
|
||||
word2phone = word2ph
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
|
||||
1387
train_ms.py
1387
train_ms.py
File diff suppressed because it is too large
Load Diff
52
utils.py
52
utils.py
@@ -9,6 +9,7 @@ import numpy as np
|
||||
from huggingface_hub import hf_hub_download
|
||||
from scipy.io.wavfile import read
|
||||
import torch
|
||||
import re
|
||||
|
||||
MATPLOTLIB_FLAG = False
|
||||
|
||||
@@ -385,3 +386,54 @@ class HParams:
|
||||
|
||||
def __repr__(self):
|
||||
return self.__dict__.__repr__()
|
||||
|
||||
|
||||
def load_model(model_path, config_path):
|
||||
hps = get_hparams_from_file(config_path)
|
||||
net = SynthesizerTrn(
|
||||
# len(symbols),
|
||||
108,
|
||||
hps.data.filter_length // 2 + 1,
|
||||
hps.train.segment_size // hps.data.hop_length,
|
||||
n_speakers=hps.data.n_speakers,
|
||||
**hps.model,
|
||||
).to("cpu")
|
||||
_ = net.eval()
|
||||
_ = load_checkpoint(model_path, net, None, skip_optimizer=True)
|
||||
return net
|
||||
|
||||
|
||||
def mix_model(
|
||||
network1, network2, output_path, voice_ratio=(0.5, 0.5), tone_ratio=(0.5, 0.5)
|
||||
):
|
||||
if hasattr(network1, "module"):
|
||||
state_dict1 = network1.module.state_dict()
|
||||
state_dict2 = network2.module.state_dict()
|
||||
else:
|
||||
state_dict1 = network1.state_dict()
|
||||
state_dict2 = network2.state_dict()
|
||||
for k in state_dict1.keys():
|
||||
if k not in state_dict2.keys():
|
||||
continue
|
||||
if "enc_p" in k:
|
||||
state_dict1[k] = (
|
||||
state_dict1[k].clone() * tone_ratio[0]
|
||||
+ state_dict2[k].clone() * tone_ratio[1]
|
||||
)
|
||||
else:
|
||||
state_dict1[k] = (
|
||||
state_dict1[k].clone() * voice_ratio[0]
|
||||
+ state_dict2[k].clone() * voice_ratio[1]
|
||||
)
|
||||
for k in state_dict2.keys():
|
||||
if k not in state_dict1.keys():
|
||||
state_dict1[k] = state_dict2[k].clone()
|
||||
torch.save(
|
||||
{"model": state_dict1, "iteration": 0, "optimizer": None, "learning_rate": 0},
|
||||
output_path,
|
||||
)
|
||||
|
||||
|
||||
def get_steps(model_path):
|
||||
matches = re.findall(r"\d+", model_path)
|
||||
return matches[-1] if matches else None
|
||||
|
||||
46
webui.py
46
webui.py
@@ -1,7 +1,6 @@
|
||||
# flake8: noqa: E402
|
||||
import os
|
||||
import logging
|
||||
|
||||
import re_matching
|
||||
from tools.sentence import split_by_language
|
||||
|
||||
@@ -24,6 +23,7 @@ import webbrowser
|
||||
import numpy as np
|
||||
from config import config
|
||||
from tools.translate import translate
|
||||
import librosa
|
||||
|
||||
net_g = None
|
||||
|
||||
@@ -40,6 +40,8 @@ def generate_audio(
|
||||
length_scale,
|
||||
speaker,
|
||||
language,
|
||||
reference_audio,
|
||||
emotion,
|
||||
skip_start=False,
|
||||
skip_end=False,
|
||||
):
|
||||
@@ -51,6 +53,8 @@ def generate_audio(
|
||||
skip_end = (idx != len(slices) - 1) and skip_end
|
||||
audio = infer(
|
||||
piece,
|
||||
reference_audio=reference_audio,
|
||||
emotion=emotion,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise_scale,
|
||||
noise_scale_w=noise_scale_w,
|
||||
@@ -117,6 +121,8 @@ def tts_split(
|
||||
cut_by_sent,
|
||||
interval_between_para,
|
||||
interval_between_sent,
|
||||
reference_audio,
|
||||
emotion,
|
||||
):
|
||||
if language == "mix":
|
||||
return ("invalid", None)
|
||||
@@ -130,6 +136,8 @@ def tts_split(
|
||||
skip_end = idx != len(para_list) - 1
|
||||
audio = infer(
|
||||
p,
|
||||
reference_audio=reference_audio,
|
||||
emotion=emotion,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise_scale,
|
||||
noise_scale_w=noise_scale_w,
|
||||
@@ -157,6 +165,8 @@ def tts_split(
|
||||
skip_end = (idx != len(sent_list) - 1) and skip_end
|
||||
audio = infer(
|
||||
s,
|
||||
reference_audio=reference_audio,
|
||||
emotion=emotion,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise_scale,
|
||||
noise_scale_w=noise_scale_w,
|
||||
@@ -193,6 +203,8 @@ def tts_fn(
|
||||
noise_scale_w,
|
||||
length_scale,
|
||||
language,
|
||||
reference_audio,
|
||||
emotion,
|
||||
):
|
||||
audio_list = []
|
||||
if language == "mix":
|
||||
@@ -261,6 +273,8 @@ def tts_fn(
|
||||
length_scale,
|
||||
_speaker,
|
||||
lang_to_generate,
|
||||
reference_audio,
|
||||
emotion,
|
||||
skip_start,
|
||||
skip_end,
|
||||
)
|
||||
@@ -305,6 +319,8 @@ def tts_fn(
|
||||
noise_scale,
|
||||
noise_scale_w,
|
||||
length_scale,
|
||||
reference_audio,
|
||||
emotion,
|
||||
speaker,
|
||||
lang_to_generate,
|
||||
skip_start,
|
||||
@@ -322,6 +338,8 @@ def tts_fn(
|
||||
length_scale,
|
||||
speaker,
|
||||
language,
|
||||
reference_audio,
|
||||
emotion,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -360,22 +378,25 @@ if __name__ == "__main__":
|
||||
trans = gr.Button("中翻日", variant="primary")
|
||||
slicer = gr.Button("快速切分", variant="primary")
|
||||
speaker = gr.Dropdown(
|
||||
choices=speakers, value=speakers[0], label="选择说话人"
|
||||
choices=speakers, value=speakers[0], label="Speaker"
|
||||
)
|
||||
emotion = gr.Slider(
|
||||
minimum=0, maximum=9, value=0, step=1, label="Emotion"
|
||||
)
|
||||
sdp_ratio = gr.Slider(
|
||||
minimum=0, maximum=1, value=0.2, step=0.1, label="SDP/DP混合比"
|
||||
minimum=0, maximum=1, value=0.2, step=0.1, label="SDP Ratio"
|
||||
)
|
||||
noise_scale = gr.Slider(
|
||||
minimum=0.1, maximum=2, value=0.6, step=0.1, label="感情"
|
||||
minimum=0.1, maximum=2, value=0.6, step=0.1, label="Noise"
|
||||
)
|
||||
noise_scale_w = gr.Slider(
|
||||
minimum=0.1, maximum=2, value=0.8, step=0.1, label="音素长度"
|
||||
minimum=0.1, maximum=2, value=0.8, step=0.1, label="Noise_W"
|
||||
)
|
||||
length_scale = gr.Slider(
|
||||
minimum=0.1, maximum=2, value=1.0, step=0.1, label="语速"
|
||||
minimum=0.1, maximum=2, value=1.0, step=0.1, label="Length"
|
||||
)
|
||||
language = gr.Dropdown(
|
||||
choices=languages, value=languages[0], label="选择语言(新增mix混合选项)"
|
||||
choices=languages, value=languages[0], label="Language"
|
||||
)
|
||||
btn = gr.Button("生成音频！", variant="primary")
|
||||
with gr.Column():
|
||||
@@ -408,6 +429,8 @@ if __name__ == "__main__":
|
||||
# show_download_button=False,
|
||||
# value=os.path.abspath("./img/参数说明.png"),
|
||||
# )
|
||||
reference_text = gr.Markdown(value="## 情感参考音频（WAV 格式）：用于生成语音的情感参考。")
|
||||
reference_audio = gr.Audio(label="情感参考音频（WAV 格式）", type="filepath")
|
||||
btn.click(
|
||||
tts_fn,
|
||||
inputs=[
|
||||
@@ -418,6 +441,8 @@ if __name__ == "__main__":
|
||||
noise_scale_w,
|
||||
length_scale,
|
||||
language,
|
||||
reference_audio,
|
||||
emotion,
|
||||
],
|
||||
outputs=[text_output, audio_output],
|
||||
)
|
||||
@@ -440,10 +465,17 @@ if __name__ == "__main__":
|
||||
opt_cut_by_sent,
|
||||
interval_between_para,
|
||||
interval_between_sent,
|
||||
reference_audio,
|
||||
emotion,
|
||||
],
|
||||
outputs=[text_output, audio_output],
|
||||
)
|
||||
|
||||
reference_audio.upload(
|
||||
lambda x: librosa.load(x, 16000)[::-1],
|
||||
inputs=[reference_audio],
|
||||
outputs=[reference_audio],
|
||||
)
|
||||
print("推理页面已开启!")
|
||||
webbrowser.open(f"http://127.0.0.1:{config.webui_config.port}")
|
||||
app.launch(share=config.webui_config.share, server_port=config.webui_config.port)
|
||||
|
||||
Reference in New Issue
Block a user