tuna2134/sbv2-v2
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Use clap to achieve prompt controlled generation (#223)

Browse Source 
* 快速分类音频并把yml格式结果存在训练根目录里 (#190)

* Add files via upload

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update models.py

* Update webui.py

* Update infer.py

* Create compress_model.py

* 重新提交，更新Gradio推理UI (#193)

* Update webui.py

* Update webui.py

* 更新 train_ms.py

* 更新 models.py

* 更新 models.py

* 更新 models.py

* 更新 train_ms.py

* 更新 train_ms.py

* 更新 models.py

* Update preprocess_text.py

* Update config.json

* Update train_ms.py

* Update webui.py (#206)

* Add files via upload (#209)

* Update train_ms.py

* Update train_ms.py

* Update preprocess_text.py

* Update train_ms.py

* fix (#211)

* Update emotion_clustering.py

* Add files via upload

* Update emotion_clustering.py

* add cluster center save

* Add files via upload

* Update config.py

* Update default_config.yml

* Update config.py

* Update config.py

* Update emotion_clustering.py

* Update emotion_clustering.py

* Update config.py

* Update emotion_clustering.py

* Update emotion_clustering.py

* Update webui.py

* Update emotion_clustering.py

* Update commons.py

* Update emotion_clustering.py

* Update webui.py

* Update webui.py

* Add files via upload

* Update train_ms.py

* Update train_ms.py

* Update train_ms.py

* Update train_ms.py

* Update train_ms.py

* Update webui.py

* Update emotion_clustering.py

* Update emotion_clustering.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix default_config.yml.

* Update infer.py

* feat: support infer 2.1 models

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: support infer 2.1 models 兼容bug修复

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update train_ms.py

* Add CLAP

* Fix data loader

* Fix infer.py

* Fix webui.py

* Add prompt template

* Update clap_gen.py

* Fix wrong environ value

* Add g for dur disc

* Update clap_gen.py

* Fix multilang generation

* Update config.json

* Prompt mode

* Improve slice segments performance

* Add preprocess webui

* Update webui_preprocess.py

* Update webui_preprocess.py

* Update config.py

* Update default_config.yml

* Update config.py

* Update clap_gen.py

* Delete emo_gen.py

* Delete get_emo.py

* Delete emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim directory

* Update README.md

* Update README

* Split val per lang

* Delete emotion_clustering.py

* Update default_config.yml

* Update default_config.yml

* Update config.py

* Update preprocess_text.py

* Update webui_preprocess.py

* Update defalut_config.yml

* Update webui_preprocess.py

* Update preprocess_text.py

* Random augmentation for CLAP

* Update data_utils.py

* Update preprocess_text.py

* Add vq for CLAP features to avoid overfitting

* Random dummy inputs

* Update webui.py

* Update models.py

* Update infer.py

* Apply Code Formatter Change

* Update config.json

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: YYuX-1145 <138500330+YYuX-1145@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sora <654163754@qq.com>
Co-authored-by: Sihan Wang <wangsihan1995@gmail.com>
Co-authored-by: Stardust-minus <Stardust-minus@users.noreply.github.com>
This commit is contained in:
Stardust·减
2023-12-12 19:19:50 +08:00
committed by GitHub
parent 9cc786d781
commit eaefc57d71
49 changed files with 287404 additions and 3426 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
README.md
View File

@@ -5,6 +5,11 @@
# Bert-VITS2
VITS2 Backbone with multilingual bert
For quick guide, please refer to `webui_preprocess.py`.
简易教程请参见 `webui_preprocess.py`。
## 请注意，本项目核心思路来源于[anyvoiceai/MassTTS](https://github.com/anyvoiceai/MassTTS) 一个非常好的tts项目
## MassTTS的演示demo为[ai版峰哥锐评峰哥本人,并找回了在金三角失落的腰子](https://www.bilibili.com/video/BV1w24y1c7z9)

64
clap_gen.py Normal file
View File

@@ -0,0 +1,64 @@
import argparse
from multiprocessing import Pool, cpu_count
import torch
import torch.multiprocessing as mp
from tqdm import tqdm
import utils
from config import config
from clap_wrapper import get_clap_audio_feature
import librosa
import os
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
def process_line(line):
device = config.emo_gen_config.device
if config.emo_gen_config.use_multi_device:
rank = mp.current_process()._identity
rank = rank[0] if len(rank) > 0 else 0
if torch.cuda.is_available():
gpu_id = rank % torch.cuda.device_count()
device = torch.device(f"cuda:{gpu_id}")
else:
device = torch.device("cpu")
wav_path, _, language_str, text, phones, tone, word2ph = line.strip().split("|")
clap_path = wav_path.replace(".WAV", ".wav").replace(".wav", ".emo.npy")
if os.path.isfile(clap_path):
return
audio = librosa.load(wav_path, 48000)[0]
# audio = librosa.resample(audio, 44100, 48000)
clap = get_clap_audio_feature(audio, device)
torch.save(clap, clap_path)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"-c", "--config", type=str, default=config.emo_gen_config.config_path
)
parser.add_argument(
"--num_processes", type=int, default=config.emo_gen_config.num_processes
)
args, _ = parser.parse_known_args()
config_path = args.config
hps = utils.get_hparams_from_file(config_path)
lines = []
with open(hps.data.training_files, encoding="utf-8") as f:
lines.extend(f.readlines())
with open(hps.data.validation_files, encoding="utf-8") as f:
lines.extend(f.readlines())
if len(lines) != 0:
num_processes = min(args.num_processes, cpu_count())
with Pool(processes=num_processes) as pool:
for _ in tqdm(pool.imap_unordered(process_line, lines), total=len(lines)):
pass
print(f"clap生成完毕!, 共有{len(lines)}个emo.pt生成!")

49
clap_wrapper.py Normal file
View File

@@ -0,0 +1,49 @@
import sys
import torch
from transformers import ClapModel, ClapProcessor
from config import config
models = dict()
processor = ClapProcessor.from_pretrained("./emotional/clap-htsat-fused")
def get_clap_audio_feature(audio_data, device=config.bert_gen_config.device):
if (
sys.platform == "darwin"
and torch.backends.mps.is_available()
and device == "cpu"
):
device = "mps"
if not device:
device = "cuda"
if device not in models.keys():
models[device] = ClapModel.from_pretrained("./emotional/clap-htsat-fused").to(
device
)
with torch.no_grad():
inputs = processor(
audios=audio_data, return_tensors="pt", sampling_rate=48000
).to(device)
emb = models[device].get_audio_features(**inputs)
return emb.T
def get_clap_text_feature(text, device=config.bert_gen_config.device):
if (
sys.platform == "darwin"
and torch.backends.mps.is_available()
and device == "cpu"
):
device = "mps"
if not device:
device = "cuda"
if device not in models.keys():
models[device] = ClapModel.from_pretrained("./emotional/clap-htsat-fused").to(
device
)
with torch.no_grad():
inputs = processor(text=text, return_tensors="pt").to(device)
emb = models[device].get_text_features(**inputs)
return emb.T

20
commons.py
View File

@@ -46,26 +46,18 @@ def rand_gumbel_like(x):
def slice_segments(x, ids_str, segment_size=4):
ret = torch.zeros_like(x[:, :, :segment_size])
for i in range(x.size(0)):
idx_str = ids_str[i]
idx_end = idx_str + segment_size
if idx_str < 0:
i1 = x.size(2) + idx_str
r1 = x[i, :, i1:]
r2 = x[i, :, :idx_end]
ret[i] = torch.cat([r1, r2], dim=1)
else:
ret[i] = x[i, :, idx_str:idx_end]
return ret
gather_indices = ids_str.view(x.size(0), 1, 1).repeat(
1, x.size(1), 1
) + torch.arange(segment_size, device=x.device)
return torch.gather(x, 2, gather_indices)
def rand_slice_segments(x, x_lengths=None, segment_size=4):
b, d, t = x.size()
if x_lengths is None:
x_lengths = t
ids_str_max = x_lengths - segment_size + 1
ids_str = (torch.rand([b]).to(device=x.device) * ids_str_max).to(dtype=torch.long)
ids_str_max = torch.clamp(x_lengths - segment_size + 1, min=0)
ids_str = (torch.rand([b], device=x.device) * ids_str_max).to(dtype=torch.long)
ret = slice_segments(x, ids_str, segment_size)
return ret, ids_str

1
compress_model.py
View File

@@ -1,6 +1,7 @@
from collections import OrderedDict
from text.symbols import symbols
import torch
from tools.log import logger
import utils
from models import SynthesizerTrn

492
config.py
View File

@@ -1,244 +1,248 @@
"""
@Desc: 全局配置文件读取
"""
import argparse
import yaml
from typing import Dict, List
import os
import shutil
import sys
class Resample_config:
"""重采样配置"""
def __init__(self, in_dir: str, out_dir: str, sampling_rate: int = 44100):
self.sampling_rate: int = sampling_rate # 目标采样率
self.in_dir: str = in_dir # 待处理音频目录路径
self.out_dir: str = out_dir # 重采样输出路径
@classmethod
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
"""从字典中生成实例"""
# 不检查路径是否有效，此逻辑在resample.py中处理
data["in_dir"] = os.path.join(dataset_path, data["in_dir"])
data["out_dir"] = os.path.join(dataset_path, data["out_dir"])
return cls(**data)
class Preprocess_text_config:
"""数据预处理配置"""
def __init__(
self,
transcription_path: str,
cleaned_path: str,
train_path: str,
val_path: str,
config_path: str,
val_per_spk: int = 5,
max_val_total: int = 10000,
clean: bool = True,
):
self.transcription_path: str = transcription_path # 原始文本文件路径，文本格式应为{wav_path}|{speaker_name}|{language}|{text}。
self.cleaned_path: str = cleaned_path # 数据清洗后文本路径，可以不填。不填则将在原始文本目录生成
self.train_path: str = train_path # 训练集路径，可以不填。不填则将在原始文本目录生成
self.val_path: str = val_path # 验证集路径，可以不填。不填则将在原始文本目录生成
self.config_path: str = config_path # 配置文件路径
self.val_per_spk: int = val_per_spk # 每个speaker的验证集条数
self.max_val_total: int = max_val_total # 验证集最大条数，多于的会被截断并放到训练集中
self.clean: bool = clean # 是否进行数据清洗
@classmethod
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
"""从字典中生成实例"""
data["transcription_path"] = os.path.join(
dataset_path, data["transcription_path"]
)
if data["cleaned_path"] == "" or data["cleaned_path"] is None:
data["cleaned_path"] = None
else:
data["cleaned_path"] = os.path.join(dataset_path, data["cleaned_path"])
data["train_path"] = os.path.join(dataset_path, data["train_path"])
data["val_path"] = os.path.join(dataset_path, data["val_path"])
data["config_path"] = os.path.join(dataset_path, data["config_path"])
return cls(**data)
class Bert_gen_config:
"""bert_gen 配置"""
def __init__(
self,
config_path: str,
num_processes: int = 2,
device: str = "cuda",
use_multi_device: bool = False,
):
self.config_path = config_path
self.num_processes = num_processes
self.device = device
self.use_multi_device = use_multi_device
@classmethod
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
data["config_path"] = os.path.join(dataset_path, data["config_path"])
return cls(**data)
class Emo_gen_config:
"""emo_gen 配置"""
def __init__(
self,
config_path: str,
num_processes: int = 2,
device: str = "cuda",
):
self.config_path = config_path
self.num_processes = num_processes
self.device = device
@classmethod
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
data["config_path"] = os.path.join(dataset_path, data["config_path"])
return cls(**data)
class Train_ms_config:
"""训练配置"""
def __init__(
self,
config_path: str,
env: Dict[str, any],
base: Dict[str, any],
model: str,
num_workers: int,
spec_cache: bool,
keep_ckpts: int,
):
self.env = env # 需要加载的环境变量
self.base = base # 底模配置
self.model = model # 训练模型存储目录，该路径为相对于dataset_path的路径，而非项目根目录
self.config_path = config_path # 配置文件路径
self.num_workers = num_workers # worker数量
self.spec_cache = spec_cache # 是否启用spec缓存
self.keep_ckpts = keep_ckpts # ckpt数量
@classmethod
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
# data["model"] = os.path.join(dataset_path, data["model"])
data["config_path"] = os.path.join(dataset_path, data["config_path"])
return cls(**data)
class Webui_config:
"""webui 配置"""
def __init__(
self,
device: str,
model: str,
config_path: str,
language_identification_library: str,
port: int = 7860,
share: bool = False,
debug: bool = False,
):
self.device: str = device
self.model: str = model # 端口号
self.config_path: str = config_path # 是否公开部署，对外网开放
self.port: int = port # 是否开启debug模式
self.share: bool = share # 模型路径
self.debug: bool = debug # 配置文件路径
self.language_identification_library: str = (
language_identification_library # 语种识别库
)
@classmethod
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
data["config_path"] = os.path.join(dataset_path, data["config_path"])
data["model"] = os.path.join(dataset_path, data["model"])
return cls(**data)
class Server_config:
def __init__(
self, models: List[Dict[str, any]], port: int = 5000, device: str = "cuda"
):
self.models: List[Dict[str, any]] = models # 需要加载的所有模型的配置
self.port: int = port # 端口号
self.device: str = device # 模型默认使用设备
@classmethod
def from_dict(cls, data: Dict[str, any]):
return cls(**data)
class Translate_config:
"""翻译api配置"""
def __init__(self, app_key: str, secret_key: str):
self.app_key = app_key
self.secret_key = secret_key
@classmethod
def from_dict(cls, data: Dict[str, any]):
return cls(**data)
class Config:
def __init__(self, config_path: str):
if not os.path.isfile(config_path) and os.path.isfile("default_config.yml"):
shutil.copy(src="default_config.yml", dst=config_path)
print(
f"已根据默认配置文件default_config.yml生成配置文件{config_path}。请按该配置文件的说明进行配置后重新运行。"
)
print("如无特殊需求，请勿修改default_config.yml或备份该文件。")
sys.exit(0)
with open(file=config_path, mode="r", encoding="utf-8") as file:
yaml_config: Dict[str, any] = yaml.safe_load(file.read())
dataset_path: str = yaml_config["dataset_path"]
openi_token: str = yaml_config["openi_token"]
self.dataset_path: str = dataset_path
self.mirror: str = yaml_config["mirror"]
self.openi_token: str = openi_token
self.resample_config: Resample_config = Resample_config.from_dict(
dataset_path, yaml_config["resample"]
)
self.preprocess_text_config: Preprocess_text_config = (
Preprocess_text_config.from_dict(
dataset_path, yaml_config["preprocess_text"]
)
)
self.bert_gen_config: Bert_gen_config = Bert_gen_config.from_dict(
dataset_path, yaml_config["bert_gen"]
)
self.train_ms_config: Train_ms_config = Train_ms_config.from_dict(
dataset_path, yaml_config["train_ms"]
)
self.webui_config: Webui_config = Webui_config.from_dict(
dataset_path, yaml_config["webui"]
)
self.server_config: Server_config = Server_config.from_dict(
yaml_config["server"]
)
self.translate_config: Translate_config = Translate_config.from_dict(
yaml_config["translate"]
)
parser = argparse.ArgumentParser()
# 为避免与以前的config.json起冲突，将其更名如下
parser.add_argument("-y", "--yml_config", type=str, default="config.yml")
args, _ = parser.parse_known_args()
config = Config(args.yml_config)
yml_config = args.yml_config
"""
@Desc: 全局配置文件读取
"""
import argparse
import yaml
from typing import Dict, List
import os
import shutil
import sys
class Resample_config:
"""重采样配置"""
def __init__(self, in_dir: str, out_dir: str, sampling_rate: int = 44100):
self.sampling_rate: int = sampling_rate # 目标采样率
self.in_dir: str = in_dir # 待处理音频目录路径
self.out_dir: str = out_dir # 重采样输出路径
@classmethod
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
"""从字典中生成实例"""
# 不检查路径是否有效，此逻辑在resample.py中处理
data["in_dir"] = os.path.join(dataset_path, data["in_dir"])
data["out_dir"] = os.path.join(dataset_path, data["out_dir"])
return cls(**data)
class Preprocess_text_config:
"""数据预处理配置"""
def __init__(
self,
transcription_path: str,
cleaned_path: str,
train_path: str,
val_path: str,
config_path: str,
val_per_lang: int = 5,
max_val_total: int = 10000,
clean: bool = True,
):
self.transcription_path: str = transcription_path # 原始文本文件路径，文本格式应为{wav_path}|{speaker_name}|{language}|{text}。
self.cleaned_path: str = cleaned_path # 数据清洗后文本路径，可以不填。不填则将在原始文本目录生成
self.train_path: str = train_path # 训练集路径，可以不填。不填则将在原始文本目录生成
self.val_path: str = val_path # 验证集路径，可以不填。不填则将在原始文本目录生成
self.config_path: str = config_path # 配置文件路径
self.val_per_lang: int = val_per_lang # 每个speaker的验证集条数
self.max_val_total: int = max_val_total # 验证集最大条数，多于的会被截断并放到训练集中
self.clean: bool = clean # 是否进行数据清洗
@classmethod
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
"""从字典中生成实例"""
data["transcription_path"] = os.path.join(
dataset_path, data["transcription_path"]
)
if data["cleaned_path"] == "" or data["cleaned_path"] is None:
data["cleaned_path"] = None
else:
data["cleaned_path"] = os.path.join(dataset_path, data["cleaned_path"])
data["train_path"] = os.path.join(dataset_path, data["train_path"])
data["val_path"] = os.path.join(dataset_path, data["val_path"])
data["config_path"] = os.path.join(dataset_path, data["config_path"])
return cls(**data)
class Bert_gen_config:
"""bert_gen 配置"""
def __init__(
self,
config_path: str,
num_processes: int = 2,
device: str = "cuda",
use_multi_device: bool = False,
):
self.config_path = config_path
self.num_processes = num_processes
self.device = device
self.use_multi_device = use_multi_device
@classmethod
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
data["config_path"] = os.path.join(dataset_path, data["config_path"])
return cls(**data)
class Emo_gen_config:
"""emo_gen 配置"""
def __init__(
self,
config_path: str,
num_processes: int = 2,
device: str = "cuda",
use_multi_device: bool = False,
):
self.config_path = config_path
self.num_processes = num_processes
self.device = device
self.use_multi_device = use_multi_device
@classmethod
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
data["config_path"] = os.path.join(dataset_path, data["config_path"])
return cls(**data)
class Train_ms_config:
"""训练配置"""
def __init__(
self,
config_path: str,
env: Dict[str, any],
base: Dict[str, any],
model: str,
num_workers: int,
spec_cache: bool,
keep_ckpts: int,
):
self.env = env # 需要加载的环境变量
self.base = base # 底模配置
self.model = model # 训练模型存储目录，该路径为相对于dataset_path的路径，而非项目根目录
self.config_path = config_path # 配置文件路径
self.num_workers = num_workers # worker数量
self.spec_cache = spec_cache # 是否启用spec缓存
self.keep_ckpts = keep_ckpts # ckpt数量
@classmethod
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
# data["model"] = os.path.join(dataset_path, data["model"])
data["config_path"] = os.path.join(dataset_path, data["config_path"])
return cls(**data)
class Webui_config:
"""webui 配置"""
def __init__(
self,
device: str,
model: str,
config_path: str,
language_identification_library: str,
port: int = 7860,
share: bool = False,
debug: bool = False,
):
self.device: str = device
self.model: str = model # 端口号
self.config_path: str = config_path # 是否公开部署，对外网开放
self.port: int = port # 是否开启debug模式
self.share: bool = share # 模型路径
self.debug: bool = debug # 配置文件路径
self.language_identification_library: str = (
language_identification_library # 语种识别库
)
@classmethod
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
data["config_path"] = os.path.join(dataset_path, data["config_path"])
data["model"] = os.path.join(dataset_path, data["model"])
return cls(**data)
class Server_config:
def __init__(
self, models: List[Dict[str, any]], port: int = 5000, device: str = "cuda"
):
self.models: List[Dict[str, any]] = models # 需要加载的所有模型的配置
self.port: int = port # 端口号
self.device: str = device # 模型默认使用设备
@classmethod
def from_dict(cls, data: Dict[str, any]):
return cls(**data)
class Translate_config:
"""翻译api配置"""
def __init__(self, app_key: str, secret_key: str):
self.app_key = app_key
self.secret_key = secret_key
@classmethod
def from_dict(cls, data: Dict[str, any]):
return cls(**data)
class Config:
def __init__(self, config_path: str):
if not os.path.isfile(config_path) and os.path.isfile("default_config.yml"):
shutil.copy(src="default_config.yml", dst=config_path)
print(
f"已根据默认配置文件default_config.yml生成配置文件{config_path}。请按该配置文件的说明进行配置后重新运行。"
)
print("如无特殊需求，请勿修改default_config.yml或备份该文件。")
sys.exit(0)
with open(file=config_path, mode="r", encoding="utf-8") as file:
yaml_config: Dict[str, any] = yaml.safe_load(file.read())
dataset_path: str = yaml_config["dataset_path"]
openi_token: str = yaml_config["openi_token"]
self.dataset_path: str = dataset_path
self.mirror: str = yaml_config["mirror"]
self.openi_token: str = openi_token
self.resample_config: Resample_config = Resample_config.from_dict(
dataset_path, yaml_config["resample"]
)
self.preprocess_text_config: Preprocess_text_config = (
Preprocess_text_config.from_dict(
dataset_path, yaml_config["preprocess_text"]
)
)
self.bert_gen_config: Bert_gen_config = Bert_gen_config.from_dict(
dataset_path, yaml_config["bert_gen"]
)
self.emo_gen_config: Emo_gen_config = Emo_gen_config.from_dict(
dataset_path, yaml_config["emo_gen"]
)
self.train_ms_config: Train_ms_config = Train_ms_config.from_dict(
dataset_path, yaml_config["train_ms"]
)
self.webui_config: Webui_config = Webui_config.from_dict(
dataset_path, yaml_config["webui"]
)
self.server_config: Server_config = Server_config.from_dict(
yaml_config["server"]
)
self.translate_config: Translate_config = Translate_config.from_dict(
yaml_config["translate"]
)
parser = argparse.ArgumentParser()
# 为避免与以前的config.json起冲突，将其更名如下
parser.add_argument("-y", "--yml_config", type=str, default="config.yml")
args, _ = parser.parse_known_args()
config = Config(args.yml_config)

437
configs/config.json
View File

@@ -10,7 +10,7 @@
0.99
],
"eps": 1e-09,
"batch_size": 24,
"batch_size": 12,
"fp16_run": false,
"lr_decay": 0.99995,
"segment_size": 16384,
@@ -18,7 +18,10 @@
"warmup_epochs": 0,
"c_mel": 45,
"c_kl": 1.0,
"skip_optimizer": true
"skip_optimizer": true,
"freeze_ZH_bert": false,
"freeze_JP_bert": false,
"freeze_EN_bert": false
},
"data": {
"training_files": "filelists/train.list",
@@ -676,220 +679,220 @@
"埃舍尔_EN": 638,
"萨齐因_EN": 639,
"古田_EN": 640,
"陆景和": 641,
"莫弈": 642,
"左然": 643,
"夏彦": 644,
"三月七_ZH": 645,
"丹恒_ZH": 646,
"希儿_ZH": 647,
"娜塔莎_ZH": 648,
"希露瓦_ZH": 649,
"瓦尔特_ZH": 650,
"佩拉_ZH": 651,
"布洛妮娅_ZH": 652,
"虎克_ZH": 653,
"素裳_ZH": 654,
"克拉拉_ZH": 655,
"符玄_ZH": 656,
"白露_ZH": 657,
"杰帕德_ZH": 658,
"景元_ZH": 659,
"藿藿_ZH": 660,
"姬子_ZH": 661,
"穹_ZH": 662,
"星_ZH": 663,
"卡芙卡_ZH": 664,
"桂乃芬_ZH": 665,
"艾丝妲_ZH": 666,
"玲可_ZH": 667,
"彦卿_ZH": 668,
"托帕_ZH": 669,
"驭空_ZH": 670,
"浮烟_ZH": 671,
"停云_ZH": 672,
"镜流_ZH": 673,
"罗刹_ZH": 674,
"卢卡_ZH": 675,
"史瓦罗_ZH": 676,
"黑塔_ZH": 677,
"桑博_ZH": 678,
"伦纳德_ZH": 679,
"明曦_ZH": 680,
"银狼_ZH": 681,
"帕姆_ZH": 682,
"青雀_ZH": 683,
"乔瓦尼_ZH": 684,
"公输师傅_ZH": 685,
"晴霓_ZH": 686,
"螺丝咕姆_ZH": 687,
"阿兰_ZH": 688,
"奥列格_ZH": 689,
"丹枢_ZH": 690,
"尾巴_ZH": 691,
"寒鸦_ZH": 692,
"雪衣_ZH": 693,
"可可利亚_ZH": 694,
"青镞_ZH": 695,
"半夏_ZH": 696,
"银枝_ZH": 697,
"大毫_ZH": 698,
"霄翰_ZH": 699,
"信使_ZH": 700,
"费斯曼_ZH": 701,
"绿芙蓉_ZH": 702,
"dev_成男_ZH": 703,
"金人会长_ZH": 704,
"维利特_ZH": 705,
"维尔德_ZH": 706,
"斯科特_ZH": 707,
"卡波特_ZH": 708,
"刃_ZH": 709,
"岩明_ZH": 710,
"浣溪_ZH": 711,
"三月七_JP": 712,
"丹恒_JP": 713,
"希儿_JP": 714,
"娜塔莎_JP": 715,
"希露瓦_JP": 716,
"瓦尔特_JP": 717,
"佩拉_JP": 718,
"布洛妮娅_JP": 719,
"虎克_JP": 720,
"素裳_JP": 721,
"克拉拉_JP": 722,
"符玄_JP": 723,
"白露_JP": 724,
"杰帕德_JP": 725,
"景元_JP": 726,
"藿藿_JP": 727,
"姬子_JP": 728,
"卡芙卡_JP": 729,
"穹_JP": 730,
"星_JP": 731,
"桂乃芬_JP": 732,
"艾丝妲_JP": 733,
"彦卿_JP": 734,
"玲可_JP": 735,
"托帕_JP": 736,
"驭空_JP": 737,
"浮烟_JP": 738,
"停云_JP": 739,
"镜流_JP": 740,
"罗刹_JP": 741,
"卢卡_JP": 742,
"史瓦罗_JP": 743,
"黑塔_JP": 744,
"桑博_JP": 745,
"伦纳德_JP": 746,
"明曦_JP": 747,
"银狼_JP": 748,
"帕姆_JP": 749,
"青雀_JP": 750,
"乔瓦尼_JP": 751,
"公输师傅_JP": 752,
"晴霓_JP": 753,
"螺丝咕姆_JP": 754,
"阿兰_JP": 755,
"奥列格_JP": 756,
"丹枢_JP": 757,
"尾巴_JP": 758,
"寒鸦_JP": 759,
"雪衣_JP": 760,
"可可利亚_JP": 761,
"青镞_JP": 762,
"半夏_JP": 763,
"银枝_JP": 764,
"大毫_JP": 765,
"霄翰_JP": 766,
"信使_JP": 767,
"费斯曼_JP": 768,
"绿芙蓉_JP": 769,
"dev_成男_JP": 770,
"金人会长_JP": 771,
"维利特_JP": 772,
"维尔德_JP": 773,
"斯科特_JP": 774,
"刃_JP": 775,
"卡波特_JP": 776,
"岩明_JP": 777,
"浣溪_JP": 778,
"净砚_JP": 779,
"紫月季_JP": 780,
"歌蒂_JP": 781,
"奇怪的云骑_JP": 782,
"幻胧_JP": 783,
"斯薇塔_JP": 784,
"隐书_JP": 785,
"三月七_EN": 786,
"丹恒_EN": 787,
"希儿_EN": 788,
"娜塔莎_EN": 789,
"希露瓦_EN": 790,
"瓦尔特_EN": 791,
"佩拉_EN": 792,
"布洛妮娅_EN": 793,
"虎克_EN": 794,
"素裳_EN": 795,
"克拉拉_EN": 796,
"符玄_EN": 797,
"白露_EN": 798,
"杰帕德_EN": 799,
"景元_EN": 800,
"藿藿_EN": 801,
"姬子_EN": 802,
"卡芙卡_EN": 803,
"穹_EN": 804,
"星_EN": 805,
"桂乃芬_EN": 806,
"艾丝妲_EN": 807,
"彦卿_EN": 808,
"玲可_EN": 809,
"托帕_EN": 810,
"驭空_EN": 811,
"浮烟_EN": 812,
"停云_EN": 813,
"镜流_EN": 814,
"罗刹_EN": 815,
"卢卡_EN": 816,
"史瓦罗_EN": 817,
"黑塔_EN": 818,
"桑博_EN": 819,
"伦纳德_EN": 820,
"明曦_EN": 821,
"银狼_EN": 822,
"帕姆_EN": 823,
"青雀_EN": 824,
"乔瓦尼_EN": 825,
"公输师傅_EN": 826,
"晴霓_EN": 827,
"螺丝咕姆_EN": 828,
"阿兰_EN": 829,
"奥列格_EN": 830,
"丹枢_EN": 831,
"尾巴_EN": 832,
"寒鸦_EN": 833,
"雪衣_EN": 834,
"可可利亚_EN": 835,
"青镞_EN": 836,
"半夏_EN": 837,
"银枝_EN": 838,
"大毫_EN": 839,
"霄翰_EN": 840,
"信使_EN": 841,
"费斯曼_EN": 842,
"绿芙蓉_EN": 843,
"dev_成男_EN": 844,
"金人会长_EN": 845,
"维利特_EN": 846,
"维尔德_EN": 847,
"刃_EN": 848,
"卡波特_EN": 849,
"岩明_EN": 850,
"浣溪_EN": 851,
"紫月季_EN": 852,
"幻胧_EN": 853,
"女声_EN": 854
"三月七_ZH": 641,
"丹恒_ZH": 642,
"希儿_ZH": 643,
"娜塔莎_ZH": 644,
"希露瓦_ZH": 645,
"瓦尔特_ZH": 646,
"佩拉_ZH": 647,
"布洛妮娅_ZH": 648,
"虎克_ZH": 649,
"素裳_ZH": 650,
"克拉拉_ZH": 651,
"符玄_ZH": 652,
"白露_ZH": 653,
"杰帕德_ZH": 654,
"景元_ZH": 655,
"藿藿_ZH": 656,
"姬子_ZH": 657,
"穹_ZH": 658,
"星_ZH": 659,
"卡芙卡_ZH": 660,
"桂乃芬_ZH": 661,
"艾丝妲_ZH": 662,
"玲可_ZH": 663,
"彦卿_ZH": 664,
"托帕_ZH": 665,
"驭空_ZH": 666,
"浮烟_ZH": 667,
"停云_ZH": 668,
"镜流_ZH": 669,
"罗刹_ZH": 670,
"卢卡_ZH": 671,
"史瓦罗_ZH": 672,
"黑塔_ZH": 673,
"桑博_ZH": 674,
"伦纳德_ZH": 675,
"明曦_ZH": 676,
"银狼_ZH": 677,
"帕姆_ZH": 678,
"青雀_ZH": 679,
"乔瓦尼_ZH": 680,
"公输师傅_ZH": 681,
"晴霓_ZH": 682,
"螺丝咕姆_ZH": 683,
"阿兰_ZH": 684,
"奥列格_ZH": 685,
"丹枢_ZH": 686,
"尾巴_ZH": 687,
"寒鸦_ZH": 688,
"雪衣_ZH": 689,
"可可利亚_ZH": 690,
"青镞_ZH": 691,
"半夏_ZH": 692,
"银枝_ZH": 693,
"大毫_ZH": 694,
"霄翰_ZH": 695,
"信使_ZH": 696,
"费斯曼_ZH": 697,
"绿芙蓉_ZH": 698,
"dev_成男_ZH": 699,
"金人会长_ZH": 700,
"维利特_ZH": 701,
"维尔德_ZH": 702,
"斯科特_ZH": 703,
"卡波特_ZH": 704,
"刃_ZH": 705,
"岩明_ZH": 706,
"浣溪_ZH": 707,
"三月七_JP": 708,
"丹恒_JP": 709,
"希儿_JP": 710,
"娜塔莎_JP": 711,
"希露瓦_JP": 712,
"瓦尔特_JP": 713,
"佩拉_JP": 714,
"布洛妮娅_JP": 715,
"虎克_JP": 716,
"素裳_JP": 717,
"克拉拉_JP": 718,
"符玄_JP": 719,
"白露_JP": 720,
"杰帕德_JP": 721,
"景元_JP": 722,
"藿藿_JP": 723,
"姬子_JP": 724,
"卡芙卡_JP": 725,
"穹_JP": 726,
"星_JP": 727,
"桂乃芬_JP": 728,
"艾丝妲_JP": 729,
"彦卿_JP": 730,
"玲可_JP": 731,
"托帕_JP": 732,
"驭空_JP": 733,
"浮烟_JP": 734,
"停云_JP": 735,
"镜流_JP": 736,
"罗刹_JP": 737,
"卢卡_JP": 738,
"史瓦罗_JP": 739,
"黑塔_JP": 740,
"桑博_JP": 741,
"伦纳德_JP": 742,
"明曦_JP": 743,
"银狼_JP": 744,
"帕姆_JP": 745,
"青雀_JP": 746,
"乔瓦尼_JP": 747,
"公输师傅_JP": 748,
"晴霓_JP": 749,
"螺丝咕姆_JP": 750,
"阿兰_JP": 751,
"奥列格_JP": 752,
"丹枢_JP": 753,
"尾巴_JP": 754,
"寒鸦_JP": 755,
"雪衣_JP": 756,
"可可利亚_JP": 757,
"青镞_JP": 758,
"半夏_JP": 759,
"银枝_JP": 760,
"大毫_JP": 761,
"霄翰_JP": 762,
"信使_JP": 763,
"费斯曼_JP": 764,
"绿芙蓉_JP": 765,
"dev_成男_JP": 766,
"金人会长_JP": 767,
"维利特_JP": 768,
"维尔德_JP": 769,
"斯科特_JP": 770,
"刃_JP": 771,
"卡波特_JP": 772,
"岩明_JP": 773,
"浣溪_JP": 774,
"净砚_JP": 775,
"紫月季_JP": 776,
"歌蒂_JP": 777,
"奇怪的云骑_JP": 778,
"幻胧_JP": 779,
"斯薇塔_JP": 780,
"隐书_JP": 781,
"三月七_EN": 782,
"丹恒_EN": 783,
"希儿_EN": 784,
"娜塔莎_EN": 785,
"希露瓦_EN": 786,
"瓦尔特_EN": 787,
"佩拉_EN": 788,
"布洛妮娅_EN": 789,
"虎克_EN": 790,
"素裳_EN": 791,
"克拉拉_EN": 792,
"符玄_EN": 793,
"白露_EN": 794,
"杰帕德_EN": 795,
"景元_EN": 796,
"藿藿_EN": 797,
"姬子_EN": 798,
"卡芙卡_EN": 799,
"穹_EN": 800,
"星_EN": 801,
"桂乃芬_EN": 802,
"艾丝妲_EN": 803,
"彦卿_EN": 804,
"玲可_EN": 805,
"托帕_EN": 806,
"驭空_EN": 807,
"浮烟_EN": 808,
"停云_EN": 809,
"镜流_EN": 810,
"罗刹_EN": 811,
"卢卡_EN": 812,
"史瓦罗_EN": 813,
"黑塔_EN": 814,
"桑博_EN": 815,
"伦纳德_EN": 816,
"明曦_EN": 817,
"银狼_EN": 818,
"帕姆_EN": 819,
"青雀_EN": 820,
"乔瓦尼_EN": 821,
"公输师傅_EN": 822,
"晴霓_EN": 823,
"螺丝咕姆_EN": 824,
"阿兰_EN": 825,
"奥列格_EN": 826,
"丹枢_EN": 827,
"尾巴_EN": 828,
"寒鸦_EN": 829,
"雪衣_EN": 830,
"可可利亚_EN": 831,
"青镞_EN": 832,
"半夏_EN": 833,
"银枝_EN": 834,
"大毫_EN": 835,
"霄翰_EN": 836,
"信使_EN": 837,
"费斯曼_EN": 838,
"绿芙蓉_EN": 839,
"dev_成男_EN": 840,
"金人会长_EN": 841,
"维利特_EN": 842,
"维尔德_EN": 843,
"刃_EN": 844,
"卡波特_EN": 845,
"岩明_EN": 846,
"浣溪_EN": 847,
"紫月季_EN": 848,
"幻胧_EN": 849,
"女声_EN": 850,
"陆景和": 851,
"莫弈": 852,
"左然": 853,
"夏彦": 854
}
},
"model": {
@@ -946,5 +949,5 @@
"use_spectral_norm": false,
"gin_channels": 256
},
"version": "2.1"
"version": "2.2"
}

27
data_utils.py
View File

@@ -44,6 +44,10 @@ class TextAudioSpeakerLoader(torch.utils.data.Dataset):
self.min_text_len = getattr(hparams, "min_text_len", 1)
self.max_text_len = getattr(hparams, "max_text_len", 384)
self.empty_emo = torch.squeeze(
torch.load("empty_emo.npy", map_location="cpu"), dim=1
)
random.seed(1234)
random.shuffle(self.audiopaths_sid_text)
self._filter()
@@ -93,7 +97,14 @@ class TextAudioSpeakerLoader(torch.utils.data.Dataset):
spec, wav = self.get_audio(audiopath)
sid = torch.LongTensor([int(self.spk_map[sid])])
emo = torch.FloatTensor(np.load(audiopath.replace(".wav", ".emo.npy")))
if np.random.rand() > 0.1:
emo = torch.squeeze(
torch.load(audiopath.replace(".wav", ".emo.npy"), map_location="cpu"),
dim=1,
)
else:
emo = self.empty_emo
return (phones, spec, wav, sid, tone, language, bert, ja_bert, en_bert, emo)
def get_audio(self, filename):
@@ -157,15 +168,15 @@ class TextAudioSpeakerLoader(torch.utils.data.Dataset):
if language_str == "ZH":
bert = bert_ori
ja_bert = torch.zeros(1024, len(phone))
en_bert = torch.zeros(1024, len(phone))
ja_bert = torch.rand(1024, len(phone))
en_bert = torch.rand(1024, len(phone))
elif language_str == "JP":
bert = torch.zeros(1024, len(phone))
bert = torch.rand(1024, len(phone))
ja_bert = bert_ori
en_bert = torch.zeros(1024, len(phone))
en_bert = torch.rand(1024, len(phone))
elif language_str == "EN":
bert = torch.zeros(1024, len(phone))
ja_bert = torch.zeros(1024, len(phone))
bert = torch.rand(1024, len(phone))
ja_bert = torch.rand(1024, len(phone))
en_bert = bert_ori
phone = torch.LongTensor(phone)
tone = torch.LongTensor(tone)
@@ -215,7 +226,7 @@ class TextAudioSpeakerCollate:
bert_padded = torch.FloatTensor(len(batch), 1024, max_text_len)
ja_bert_padded = torch.FloatTensor(len(batch), 1024, max_text_len)
en_bert_padded = torch.FloatTensor(len(batch), 1024, max_text_len)
emo = torch.FloatTensor(len(batch), 1024)
emo = torch.FloatTensor(len(batch), 512)
spec_padded = torch.FloatTensor(len(batch), batch[0][1].size(0), max_spec_len)
wav_padded = torch.FloatTensor(len(batch), 1, max_wav_len)

353
default_config.yml
View File

@@ -1,176 +1,177 @@
# 全局配置
# 对于希望在同一时间使用多个配置文件的情况，例如两个GPU同时跑两个训练集：通过环境变量指定配置文件，不指定则默认为./config.yml
# 拟提供通用路径配置，统一存放数据，避免数据放得很乱
# 每个数据集与其对应的模型存放至统一路径下，后续所有的路径配置均为相对于datasetPath的路径
# 不填或者填空则路径为相对于项目根目录的路径
dataset_path: "Data/"
# 模型镜像源，默认huggingface，使用openi镜像源需指定openi_token
mirror: ""
openi_token: "" # openi token
# resample 音频重采样配置
# 注意， “:” 后需要加空格
resample:
# 目标重采样率
sampling_rate: 44100
# 音频文件输入路径，重采样会将该路径下所有.wav音频文件重采样
# 请填入相对于datasetPath的相对路径
in_dir: "audios/raw" # 相对于根目录的路径为 /datasetPath/in_dir
# 音频文件重采样后输出路径
out_dir: "audios/wavs"
# preprocess_text 数据集预处理相关配置
# 注意， “:” 后需要加空格
preprocess_text:
# 原始文本文件路径，文本格式应为{wav_path}|{speaker_name}|{language}|{text}。
transcription_path: "filelists/你的数据集文本.list"
# 数据清洗后文本路径，可以不填。不填则将在原始文本目录生成
cleaned_path: ""
# 训练集路径
train_path: "filelists/train.list"
# 验证集路径
val_path: "filelists/val.list"
# 配置文件路径
config_path: "config.json"
# 每个speaker的验证集条数
val_per_spk: 4
# 验证集最大条数，多于的会被截断并放到训练集中
max_val_total: 8
# 是否进行数据清洗
clean: true
# bert_gen 相关配置
# 注意， “:” 后需要加空格
bert_gen:
# 训练数据集配置文件路径
config_path: "config.json"
# 并行数
num_processes: 2
# 使用设备：可选项 "cuda" 显卡推理，"cpu" cpu推理
# 该选项同时决定了get_bert_feature的默认设备
device: "cuda"
# 使用多卡推理
use_multi_device: false
# emo_gen 相关配置
# 注意， “:” 后需要加空格
emo_gen:
# 训练数据集配置文件路径
config_path: "config.json"
# 并行数
num_processes: 2
# 使用设备：可选项 "cuda" 显卡推理，"cpu" cpu推理
device: "cuda"
# train 训练配置
# 注意， “:” 后需要加空格
train_ms:
env:
MASTER_ADDR: "localhost"
MASTER_PORT: 10086
WORLD_SIZE: 1
LOCAL_RANK: 0
RANK: 0
# 可以填写任意名的环境变量
# THE_ENV_VAR_YOU_NEED_TO_USE: "1234567"
# 底模设置
base:
use_base_model: false
repo_id: "Stardust_minus/Bert-VITS2"
model_image: "Bert-VITS2_2.1-Emo底模" # openi网页的模型名
# 训练模型存储目录：与旧版本的区别，原先数据集是存放在logs/model_name下的，现在改为统一存放在Data/你的数据集/models下
model: "models"
# 配置文件路径
config_path: "config.json"
# 训练使用的worker，不建议超过CPU核心数
num_workers: 16
# 关闭此项可以节约接近50%的磁盘空间，但是可能导致实际训练速度变慢和更高的CPU使用率。
spec_cache: True
# 保存的检查点数量，多于此数目的权重会被删除来节省空间。
keep_ckpts: 8
# webui webui配置
# 注意， “:” 后需要加空格
webui:
# 推理设备
device: "cuda"
# 模型路径
model: "genshin/models/G_8000.pth"
# 配置文件路径
config_path: "config.json"
# 端口号
port: 7860
# 是否公开部署，对外网开放
share: false
# 是否开启debug模式
debug: false
# 语种识别库，可选langid, fastlid
language_identification_library: "langid"
# server api配置
# 注意， “:” 后需要加空格
# 注意，本配置下的所有配置均为相对于根目录的路径
server:
# 端口号
port: 5000
# 模型默认使用设备：但是当前并没有实现这个配置。
device: "cuda"
# 需要加载的所有模型的配置，可以填多个模型，也可以不填模型，等网页成功后手动加载模型
# 不加载模型的配置格式：删除默认给的两个模型配置，给models赋值 [ ]，也就是空列表。参考模型2的speakers 即 models: [ ]
# 注意，所有模型都必须正确配置model与config的路径，空路径会导致加载错误。
# 也可以不填模型，等网页加载成功后手动填写models。
models:
- # 模型的路径
model: ""
# 模型config.json的路径
config: ""
# 模型使用设备，若填写则会覆盖默认配置
device: "cuda"
# 模型默认使用的语言
language: "ZH"
# 模型人物默认参数
# 不必填写所有人物，不填的使用默认值
# 暂时不用填写，当前尚未实现按人区分配置
speakers:
- speaker: "科比"
sdp_ratio: 0.2
noise_scale: 0.6
noise_scale_w: 0.8
length_scale: 1
- speaker: "五条悟"
sdp_ratio: 0.3
noise_scale: 0.7
noise_scale_w: 0.8
length_scale: 0.5
- speaker: "安倍晋三"
sdp_ratio: 0.2
noise_scale: 0.6
noise_scale_w: 0.8
length_scale: 1.2
- # 模型的路径
model: ""
# 模型config.json的路径
config: ""
# 模型使用设备，若填写则会覆盖默认配置
device: "cpu"
# 模型默认使用的语言
language: "JP"
# 模型人物默认参数
# 不必填写所有人物，不填的使用默认值
speakers: [ ] # 也可以不填
# 百度翻译开放平台 api配置
# api接入文档 https://api.fanyi.baidu.com/doc/21
# 请不要在github等网站公开分享你的app id 与 key
translate:
# 你的APPID
"app_key": ""
# 你的密钥
"secret_key": ""
# 全局配置
# 对于希望在同一时间使用多个配置文件的情况，例如两个GPU同时跑两个训练集：通过环境变量指定配置文件，不指定则默认为./config.yml
# 拟提供通用路径配置，统一存放数据，避免数据放得很乱
# 每个数据集与其对应的模型存放至统一路径下，后续所有的路径配置均为相对于datasetPath的路径
# 不填或者填空则路径为相对于项目根目录的路径
dataset_path: "Data/"
# 模型镜像源，默认huggingface，使用openi镜像源需指定openi_token
mirror: ""
openi_token: "" # openi token
# resample 音频重采样配置
# 注意， “:” 后需要加空格
resample:
# 目标重采样率
sampling_rate: 44100
# 音频文件输入路径，重采样会将该路径下所有.wav音频文件重采样
# 请填入相对于datasetPath的相对路径
in_dir: "audios/raw" # 相对于根目录的路径为 /datasetPath/in_dir
# 音频文件重采样后输出路径
out_dir: "audios/wavs"
# preprocess_text 数据集预处理相关配置
# 注意， “:” 后需要加空格
preprocess_text:
# 原始文本文件路径，文本格式应为{wav_path}|{speaker_name}|{language}|{text}。
transcription_path: "filelists/你的数据集文本.list"
# 数据清洗后文本路径，可以不填。不填则将在原始文本目录生成
cleaned_path: ""
# 训练集路径
train_path: "filelists/train.list"
# 验证集路径
val_path: "filelists/val.list"
# 配置文件路径
config_path: "config.json"
# 每个语言的验证集条数
val_per_lang: 4
# 验证集最大条数，多于的会被截断并放到训练集中
max_val_total: 12
# 是否进行数据清洗
clean: true
# bert_gen 相关配置
# 注意， “:” 后需要加空格
bert_gen:
# 训练数据集配置文件路径
config_path: "config.json"
# 并行数
num_processes: 4
# 使用设备：可选项 "cuda" 显卡推理，"cpu" cpu推理
# 该选项同时决定了get_bert_feature的默认设备
device: "cuda"
# 使用多卡推理
use_multi_device: false
# emo_gen 相关配置
# 注意， “:” 后需要加空格
emo_gen:
# 训练数据集配置文件路径
config_path: "config.json"
# 并行数
num_processes: 4
# 使用设备：可选项 "cuda" 显卡推理，"cpu" cpu推理
device: "cuda"
# 使用多卡推理
use_multi_device: false
# train 训练配置
# 注意， “:” 后需要加空格
train_ms:
env:
MASTER_ADDR: "localhost"
MASTER_PORT: 10086
WORLD_SIZE: 1
LOCAL_RANK: 0
RANK: 0
# 可以填写任意名的环境变量
# THE_ENV_VAR_YOU_NEED_TO_USE: "1234567"
# 底模设置
base:
use_base_model: false
repo_id: "Stardust_minus/Bert-VITS2"
model_image: "Bert-VITS2_2.1-Emo底模" # openi网页的模型名
# 训练模型存储目录：与旧版本的区别，原先数据集是存放在logs/model_name下的，现在改为统一存放在Data/你的数据集/models下
model: "models"
# 配置文件路径
config_path: "configs/config.json"
# 训练使用的worker，不建议超过CPU核心数
num_workers: 16
# 关闭此项可以节约接近50%的磁盘空间，但是可能导致实际训练速度变慢和更高的CPU使用率。
spec_cache: True
# 保存的检查点数量，多于此数目的权重会被删除来节省空间。
keep_ckpts: 8
# webui webui配置
# 注意， “:” 后需要加空格
webui:
# 推理设备
device: "cuda"
# 模型路径
model: "models/G_8000.pth"
# 配置文件路径
config_path: "configs/config.json"
# 端口号
port: 7860
# 是否公开部署，对外网开放
share: false
# 是否开启debug模式
debug: false
# 语种识别库，可选langid, fastlid
language_identification_library: "langid"
# server-fastapi配置
# 注意， “:” 后需要加空格
# 注意，本配置下的所有配置均为相对于根目录的路径
server:
# 端口号
port: 5000
# 模型默认使用设备：但是当前并没有实现这个配置。
device: "cuda"
# 需要加载的所有模型的配置，可以填多个模型，也可以不填模型，等网页成功后手动加载模型
# 不加载模型的配置格式：删除默认给的两个模型配置，给models赋值 [ ]，也就是空列表。参考模型2的speakers 即 models: [ ]
# 注意，所有模型都必须正确配置model与config的路径，空路径会导致加载错误。
# 也可以不填模型，等网页加载成功后手动填写models。
models:
- # 模型的路径
model: ""
# 模型config.json的路径
config: ""
# 模型使用设备，若填写则会覆盖默认配置
device: "cuda"
# 模型默认使用的语言
language: "ZH"
# 模型人物默认参数
# 不必填写所有人物，不填的使用默认值
# 暂时不用填写，当前尚未实现按人区分配置
speakers:
- speaker: "科比"
sdp_ratio: 0.2
noise_scale: 0.6
noise_scale_w: 0.8
length_scale: 1
- speaker: "五条悟"
sdp_ratio: 0.3
noise_scale: 0.7
noise_scale_w: 0.8
length_scale: 0.5
- speaker: "安倍晋三"
sdp_ratio: 0.2
noise_scale: 0.6
noise_scale_w: 0.8
length_scale: 1.2
- # 模型的路径
model: ""
# 模型config.json的路径
config: ""
# 模型使用设备，若填写则会覆盖默认配置
device: "cpu"
# 模型默认使用的语言
language: "JP"
# 模型人物默认参数
# 不必填写所有人物，不填的使用默认值
speakers: [ ] # 也可以不填
# 百度翻译开放平台 api配置
# api接入文档 https://api.fanyi.baidu.com/doc/21
# 请不要在github等网站公开分享你的app id 与 key
translate:
# 你的APPID
"app_key": ""
# 你的密钥
"secret_key": ""

10
emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim/.gitattributes → emotional/clap-htsat-fused/.gitattributes vendored
View File

@@ -1,22 +1,28 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bin.* filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
@@ -24,5 +30,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zstandard filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

107
emotional/clap-htsat-fused/README.md Normal file
View File

@@ -0,0 +1,107 @@
---
license: apache-2.0
---
# Model card for CLAP
Model card for CLAP: Contrastive Language-Audio Pretraining
![clap_image](https://s3.amazonaws.com/moonup/production/uploads/1678811100805-62441d1d9fdefb55a0b7d12c.png)
# Table of Contents
0. [TL;DR](#TL;DR)
1. [Model Details](#model-details)
2. [Usage](#usage)
3. [Uses](#uses)
4. [Citation](#citation)
# TL;DR
The abstract of the paper states that:
> Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zero-shot setting and is able to obtain performance comparable to models' results in the non-zero-shot setting. LAION-Audio-630K and the proposed model are both available to the public.
# Usage
You can use this model for zero shot audio classification or extracting audio and/or textual features.
# Uses
## Perform zero-shot audio classification
### Using `pipeline`
```python
from datasets import load_dataset
from transformers import pipeline
dataset = load_dataset("ashraq/esc50")
audio = dataset["train"]["audio"][-1]["array"]
audio_classifier = pipeline(task="zero-shot-audio-classification", model="laion/clap-htsat-fused")
output = audio_classifier(audio, candidate_labels=["Sound of a dog", "Sound of vaccum cleaner"])
print(output)
>>> [{"score": 0.999, "label": "Sound of a dog"}, {"score": 0.001, "label": "Sound of vaccum cleaner"}]
```
## Run the model:
You can also get the audio and text embeddings using `ClapModel`
### Run the model on CPU:
```python
from datasets import load_dataset
from transformers import ClapModel, ClapProcessor
librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
audio_sample = librispeech_dummy[0]
model = ClapModel.from_pretrained("laion/clap-htsat-fused")
processor = ClapProcessor.from_pretrained("laion/clap-htsat-fused")
inputs = processor(audios=audio_sample["audio"]["array"], return_tensors="pt")
audio_embed = model.get_audio_features(**inputs)
```
### Run the model on GPU:
```python
from datasets import load_dataset
from transformers import ClapModel, ClapProcessor
librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
audio_sample = librispeech_dummy[0]
model = ClapModel.from_pretrained("laion/clap-htsat-fused").to(0)
processor = ClapProcessor.from_pretrained("laion/clap-htsat-fused")
inputs = processor(audios=audio_sample["audio"]["array"], return_tensors="pt").to(0)
audio_embed = model.get_audio_features(**inputs)
```
# Citation
If you are using this model for your work, please consider citing the original paper:
```
@misc{https://doi.org/10.48550/arxiv.2211.06687,
doi = {10.48550/ARXIV.2211.06687},
url = {https://arxiv.org/abs/2211.06687},
author = {Wu, Yusong and Chen, Ke and Zhang, Tianyu and Hui, Yuchen and Berg-Kirkpatrick, Taylor and Dubnov, Shlomo},
keywords = {Sound (cs.SD), Audio and Speech Processing (eess.AS), FOS: Computer and information sciences, FOS: Computer and information sciences, FOS: Electrical engineering, electronic engineering, information engineering, FOS: Electrical engineering, electronic engineering, information engineering},
title = {Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution 4.0 International}
}
```

207
emotional/clap-htsat-fused/config.json Normal file
View File

@@ -0,0 +1,207 @@
{
"_commit_hash": null,
"architectures": [
"ClapModel"
],
"audio_config": {
"_name_or_path": "",
"add_cross_attention": false,
"aff_block_r": 4,
"architectures": null,
"attention_probs_dropout_prob": 0.0,
"bad_words_ids": null,
"begin_suppress_tokens": null,
"bos_token_id": null,
"chunk_size_feed_forward": 0,
"cross_attention_hidden_size": null,
"decoder_start_token_id": null,
"depths": [
2,
2,
6,
2
],
"diversity_penalty": 0.0,
"do_sample": false,
"drop_path_rate": 0.0,
"early_stopping": false,
"enable_fusion": true,
"enable_patch_fusion": true,
"enable_patch_layer_norm": true,
"encoder_no_repeat_ngram_size": 0,
"eos_token_id": null,
"exponential_decay_length_penalty": null,
"finetuning_task": null,
"flatten_patch_embeds": true,
"forced_bos_token_id": null,
"forced_eos_token_id": null,
"fusion_num_hidden_layers": 2,
"fusion_type": null,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"id2label": {
"0": "LABEL_0",
"1": "LABEL_1"
},
"initializer_factor": 1.0,
"is_decoder": false,
"is_encoder_decoder": false,
"label2id": {
"LABEL_0": 0,
"LABEL_1": 1
},
"layer_norm_eps": 1e-05,
"length_penalty": 1.0,
"max_length": 20,
"min_length": 0,
"mlp_ratio": 4.0,
"model_type": "clap_audio_model",
"no_repeat_ngram_size": 0,
"num_attention_heads": [
4,
8,
16,
32
],
"num_beam_groups": 1,
"num_beams": 1,
"num_classes": 527,
"num_hidden_layers": 4,
"num_mel_bins": 64,
"num_return_sequences": 1,
"output_attentions": false,
"output_hidden_states": false,
"output_scores": false,
"pad_token_id": null,
"patch_embed_input_channels": 1,
"patch_embeds_hidden_size": 96,
"patch_size": 4,
"patch_stride": [
4,
4
],
"prefix": null,
"problem_type": null,
"projection_dim": 512,
"projection_hidden_act": "relu",
"projection_hidden_size": 768,
"pruned_heads": {},
"qkv_bias": true,
"remove_invalid_values": false,
"repetition_penalty": 1.0,
"return_dict": true,
"return_dict_in_generate": false,
"sep_token_id": null,
"spec_size": 256,
"suppress_tokens": null,
"task_specific_params": null,
"temperature": 1.0,
"tf_legacy_loss": false,
"tie_encoder_decoder": false,
"tie_word_embeddings": true,
"tokenizer_class": null,
"top_k": 50,
"top_p": 1.0,
"torch_dtype": null,
"torchscript": false,
"transformers_version": "4.27.0.dev0",
"typical_p": 1.0,
"use_bfloat16": false,
"window_size": 8
},
"hidden_size": 768,
"initializer_factor": 1.0,
"logit_scale_init_value": 14.285714285714285,
"model_type": "clap",
"num_hidden_layers": 16,
"projection_dim": 512,
"projection_hidden_act": "relu",
"text_config": {
"_name_or_path": "",
"add_cross_attention": false,
"architectures": null,
"attention_probs_dropout_prob": 0.1,
"bad_words_ids": null,
"begin_suppress_tokens": null,
"bos_token_id": 0,
"chunk_size_feed_forward": 0,
"classifier_dropout": null,
"cross_attention_hidden_size": null,
"decoder_start_token_id": null,
"diversity_penalty": 0.0,
"do_sample": false,
"early_stopping": false,
"encoder_no_repeat_ngram_size": 0,
"eos_token_id": 2,
"exponential_decay_length_penalty": null,
"finetuning_task": null,
"forced_bos_token_id": null,
"forced_eos_token_id": null,
"fusion_hidden_size": 768,
"fusion_num_hidden_layers": 2,
"hidden_act": "gelu",
"hidden_dropout_prob": 0.1,
"hidden_size": 768,
"id2label": {
"0": "LABEL_0",
"1": "LABEL_1"
},
"initializer_factor": 1.0,
"initializer_range": 0.02,
"intermediate_size": 3072,
"is_decoder": false,
"is_encoder_decoder": false,
"label2id": {
"LABEL_0": 0,
"LABEL_1": 1
},
"layer_norm_eps": 1e-12,
"length_penalty": 1.0,
"max_length": 20,
"max_position_embeddings": 514,
"min_length": 0,
"model_type": "clap_text_model",
"no_repeat_ngram_size": 0,
"num_attention_heads": 12,
"num_beam_groups": 1,
"num_beams": 1,
"num_hidden_layers": 12,
"num_return_sequences": 1,
"output_attentions": false,
"output_hidden_states": false,
"output_scores": false,
"pad_token_id": 1,
"position_embedding_type": "absolute",
"prefix": null,
"problem_type": null,
"projection_dim": 512,
"projection_hidden_act": "relu",
"projection_hidden_size": 768,
"pruned_heads": {},
"remove_invalid_values": false,
"repetition_penalty": 1.0,
"return_dict": true,
"return_dict_in_generate": false,
"sep_token_id": null,
"suppress_tokens": null,
"task_specific_params": null,
"temperature": 1.0,
"tf_legacy_loss": false,
"tie_encoder_decoder": false,
"tie_word_embeddings": true,
"tokenizer_class": null,
"top_k": 50,
"top_p": 1.0,
"torch_dtype": null,
"torchscript": false,
"transformers_version": "4.27.0.dev0",
"type_vocab_size": 1,
"typical_p": 1.0,
"use_bfloat16": false,
"use_cache": true,
"vocab_size": 50265
},
"torch_dtype": "float32",
"transformers_version": null
}

50001
emotional/clap-htsat-fused/merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

22
emotional/clap-htsat-fused/preprocessor_config.json Normal file
View File

@@ -0,0 +1,22 @@
{
"chunk_length_s": 10,
"feature_extractor_type": "ClapFeatureExtractor",
"feature_size": 64,
"fft_window_size": 1024,
"frequency_max": 14000,
"frequency_min": 50,
"hop_length": 480,
"max_length_s": 10,
"n_fft": 1024,
"nb_frequency_bins": 513,
"nb_max_frames": 1000,
"nb_max_samples": 480000,
"padding": "repeatpad",
"padding_side": "right",
"padding_value": 0.0,
"processor_class": "ClapProcessor",
"return_attention_mask": false,
"sampling_rate": 48000,
"top_db": null,
"truncation": "fusion"
}

15
emotional/clap-htsat-fused/special_tokens_map.json Normal file
View File

@@ -0,0 +1,15 @@
{
"bos_token": "<s>",
"cls_token": "<s>",
"eos_token": "</s>",
"mask_token": {
"content": "<mask>",
"lstrip": true,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": "<pad>",
"sep_token": "</s>",
"unk_token": "<unk>"
}

100362
emotional/clap-htsat-fused/tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

16
emotional/clap-htsat-fused/tokenizer_config.json Normal file
View File

@@ -0,0 +1,16 @@
{
"add_prefix_space": false,
"bos_token": "<s>",
"cls_token": "<s>",
"eos_token": "</s>",
"errors": "replace",
"mask_token": "<mask>",
"model_max_length": 512,
"pad_token": "<pad>",
"processor_class": "ClapProcessor",
"sep_token": "</s>",
"special_tokens_map_file": null,
"tokenizer_class": "RobertaTokenizer",
"trim_offsets": true,
"unk_token": "<unk>"
}

1
emotional/clap-htsat-fused/vocab.json Normal file
View File

File diff suppressed because one or more lines are too long

437
emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim/LICENSE
View File

@@ -1,437 +0,0 @@
Attribution-NonCommercial-ShareAlike 4.0 International
=======================================================================
Creative Commons Corporation ("Creative Commons") is not a law firm and
does not provide legal services or legal advice. Distribution of
Creative Commons public licenses does not create a lawyer-client or
other relationship. Creative Commons makes its licenses and related
information available on an "as-is" basis. Creative Commons gives no
warranties regarding its licenses, any material licensed under their
terms and conditions, or any related information. Creative Commons
disclaims all liability for damages resulting from their use to the
fullest extent possible.
Using Creative Commons Public Licenses
Creative Commons public licenses provide a standard set of terms and
conditions that creators and other rights holders may use to share
original works of authorship and other material subject to copyright
and certain other rights specified in the public license below. The
following considerations are for informational purposes only, are not
exhaustive, and do not form part of our licenses.
Considerations for licensors: Our public licenses are
intended for use by those authorized to give the public
permission to use material in ways otherwise restricted by
copyright and certain other rights. Our licenses are
irrevocable. Licensors should read and understand the terms
and conditions of the license they choose before applying it.
Licensors should also secure all rights necessary before
applying our licenses so that the public can reuse the
material as expected. Licensors should clearly mark any
material not subject to the license. This includes other CC-
licensed material, or material used under an exception or
limitation to copyright. More considerations for licensors:
wiki.creativecommons.org/Considerations_for_licensors
Considerations for the public: By using one of our public
licenses, a licensor grants the public permission to use the
licensed material under specified terms and conditions. If
the licensor's permission is not necessary for any reason--for
example, because of any applicable exception or limitation to
copyright--then that use is not regulated by the license. Our
licenses grant only permissions under copyright and certain
other rights that a licensor has authority to grant. Use of
the licensed material may still be restricted for other
reasons, including because others have copyright or other
rights in the material. A licensor may make special requests,
such as asking that all changes be marked or described.
Although not required by our licenses, you are encouraged to
respect those requests where reasonable. More considerations
for the public:
wiki.creativecommons.org/Considerations_for_licensees
=======================================================================
Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International
Public License
By exercising the Licensed Rights (defined below), You accept and agree
to be bound by the terms and conditions of this Creative Commons
Attribution-NonCommercial-ShareAlike 4.0 International Public License
("Public License"). To the extent this Public License may be
interpreted as a contract, You are granted the Licensed Rights in
consideration of Your acceptance of these terms and conditions, and the
Licensor grants You such rights in consideration of benefits the
Licensor receives from making the Licensed Material available under
these terms and conditions.
Section 1 -- Definitions.
a. Adapted Material means material subject to Copyright and Similar
Rights that is derived from or based upon the Licensed Material
and in which the Licensed Material is translated, altered,
arranged, transformed, or otherwise modified in a manner requiring
permission under the Copyright and Similar Rights held by the
Licensor. For purposes of this Public License, where the Licensed
Material is a musical work, performance, or sound recording,
Adapted Material is always produced where the Licensed Material is
synched in timed relation with a moving image.
b. Adapter's License means the license You apply to Your Copyright
and Similar Rights in Your contributions to Adapted Material in
accordance with the terms and conditions of this Public License.
c. BY-NC-SA Compatible License means a license listed at
creativecommons.org/compatiblelicenses, approved by Creative
Commons as essentially the equivalent of this Public License.
d. Copyright and Similar Rights means copyright and/or similar rights
closely related to copyright including, without limitation,
performance, broadcast, sound recording, and Sui Generis Database
Rights, without regard to how the rights are labeled or
categorized. For purposes of this Public License, the rights
specified in Section 2(b)(1)-(2) are not Copyright and Similar
Rights.
e. Effective Technological Measures means those measures that, in the
absence of proper authority, may not be circumvented under laws
fulfilling obligations under Article 11 of the WIPO Copyright
Treaty adopted on December 20, 1996, and/or similar international
agreements.
f. Exceptions and Limitations means fair use, fair dealing, and/or
any other exception or limitation to Copyright and Similar Rights
that applies to Your use of the Licensed Material.
g. License Elements means the license attributes listed in the name
of a Creative Commons Public License. The License Elements of this
Public License are Attribution, NonCommercial, and ShareAlike.
h. Licensed Material means the artistic or literary work, database,
or other material to which the Licensor applied this Public
License.
i. Licensed Rights means the rights granted to You subject to the
terms and conditions of this Public License, which are limited to
all Copyright and Similar Rights that apply to Your use of the
Licensed Material and that the Licensor has authority to license.
j. Licensor means the individual(s) or entity(ies) granting rights
under this Public License.
k. NonCommercial means not primarily intended for or directed towards
commercial advantage or monetary compensation. For purposes of
this Public License, the exchange of the Licensed Material for
other material subject to Copyright and Similar Rights by digital
file-sharing or similar means is NonCommercial provided there is
no payment of monetary compensation in connection with the
exchange.
l. Share means to provide material to the public by any means or
process that requires permission under the Licensed Rights, such
as reproduction, public display, public performance, distribution,
dissemination, communication, or importation, and to make material
available to the public including in ways that members of the
public may access the material from a place and at a time
individually chosen by them.
m. Sui Generis Database Rights means rights other than copyright
resulting from Directive 96/9/EC of the European Parliament and of
the Council of 11 March 1996 on the legal protection of databases,
as amended and/or succeeded, as well as other essentially
equivalent rights anywhere in the world.
n. You means the individual or entity exercising the Licensed Rights
under this Public License. Your has a corresponding meaning.
Section 2 -- Scope.
a. License grant.
1. Subject to the terms and conditions of this Public License,
the Licensor hereby grants You a worldwide, royalty-free,
non-sublicensable, non-exclusive, irrevocable license to
exercise the Licensed Rights in the Licensed Material to:
a. reproduce and Share the Licensed Material, in whole or
in part, for NonCommercial purposes only; and
b. produce, reproduce, and Share Adapted Material for
NonCommercial purposes only.
2. Exceptions and Limitations. For the avoidance of doubt, where
Exceptions and Limitations apply to Your use, this Public
License does not apply, and You do not need to comply with
its terms and conditions.
3. Term. The term of this Public License is specified in Section
6(a).
4. Media and formats; technical modifications allowed. The
Licensor authorizes You to exercise the Licensed Rights in
all media and formats whether now known or hereafter created,
and to make technical modifications necessary to do so. The
Licensor waives and/or agrees not to assert any right or
authority to forbid You from making technical modifications
necessary to exercise the Licensed Rights, including
technical modifications necessary to circumvent Effective
Technological Measures. For purposes of this Public License,
simply making modifications authorized by this Section 2(a)
(4) never produces Adapted Material.
5. Downstream recipients.
a. Offer from the Licensor -- Licensed Material. Every
recipient of the Licensed Material automatically
receives an offer from the Licensor to exercise the
Licensed Rights under the terms and conditions of this
Public License.
b. Additional offer from the Licensor -- Adapted Material.
Every recipient of Adapted Material from You
automatically receives an offer from the Licensor to
exercise the Licensed Rights in the Adapted Material
under the conditions of the Adapter's License You apply.
c. No downstream restrictions. You may not offer or impose
any additional or different terms or conditions on, or
apply any Effective Technological Measures to, the
Licensed Material if doing so restricts exercise of the
Licensed Rights by any recipient of the Licensed
Material.
6. No endorsement. Nothing in this Public License constitutes or
may be construed as permission to assert or imply that You
are, or that Your use of the Licensed Material is, connected
with, or sponsored, endorsed, or granted official status by,
the Licensor or others designated to receive attribution as
provided in Section 3(a)(1)(A)(i).
b. Other rights.
1. Moral rights, such as the right of integrity, are not
licensed under this Public License, nor are publicity,
privacy, and/or other similar personality rights; however, to
the extent possible, the Licensor waives and/or agrees not to
assert any such rights held by the Licensor to the limited
extent necessary to allow You to exercise the Licensed
Rights, but not otherwise.
2. Patent and trademark rights are not licensed under this
Public License.
3. To the extent possible, the Licensor waives any right to
collect royalties from You for the exercise of the Licensed
Rights, whether directly or through a collecting society
under any voluntary or waivable statutory or compulsory
licensing scheme. In all other cases the Licensor expressly
reserves any right to collect such royalties, including when
the Licensed Material is used other than for NonCommercial
purposes.
Section 3 -- License Conditions.
Your exercise of the Licensed Rights is expressly made subject to the
following conditions.
a. Attribution.
1. If You Share the Licensed Material (including in modified
form), You must:
a. retain the following if it is supplied by the Licensor
with the Licensed Material:
i. identification of the creator(s) of the Licensed
Material and any others designated to receive
attribution, in any reasonable manner requested by
the Licensor (including by pseudonym if
designated);
ii. a copyright notice;
iii. a notice that refers to this Public License;
iv. a notice that refers to the disclaimer of
warranties;
v. a URI or hyperlink to the Licensed Material to the
extent reasonably practicable;
b. indicate if You modified the Licensed Material and
retain an indication of any previous modifications; and
c. indicate the Licensed Material is licensed under this
Public License, and include the text of, or the URI or
hyperlink to, this Public License.
2. You may satisfy the conditions in Section 3(a)(1) in any
reasonable manner based on the medium, means, and context in
which You Share the Licensed Material. For example, it may be
reasonable to satisfy the conditions by providing a URI or
hyperlink to a resource that includes the required
information.
3. If requested by the Licensor, You must remove any of the
information required by Section 3(a)(1)(A) to the extent
reasonably practicable.
b. ShareAlike.
In addition to the conditions in Section 3(a), if You Share
Adapted Material You produce, the following conditions also apply.
1. The Adapter's License You apply must be a Creative Commons
license with the same License Elements, this version or
later, or a BY-NC-SA Compatible License.
2. You must include the text of, or the URI or hyperlink to, the
Adapter's License You apply. You may satisfy this condition
in any reasonable manner based on the medium, means, and
context in which You Share Adapted Material.
3. You may not offer or impose any additional or different terms
or conditions on, or apply any Effective Technological
Measures to, Adapted Material that restrict exercise of the
rights granted under the Adapter's License You apply.
Section 4 -- Sui Generis Database Rights.
Where the Licensed Rights include Sui Generis Database Rights that
apply to Your use of the Licensed Material:
a. for the avoidance of doubt, Section 2(a)(1) grants You the right
to extract, reuse, reproduce, and Share all or a substantial
portion of the contents of the database for NonCommercial purposes
only;
b. if You include all or a substantial portion of the database
contents in a database in which You have Sui Generis Database
Rights, then the database in which You have Sui Generis Database
Rights (but not its individual contents) is Adapted Material,
including for purposes of Section 3(b); and
c. You must comply with the conditions in Section 3(a) if You Share
all or a substantial portion of the contents of the database.
For the avoidance of doubt, this Section 4 supplements and does not
replace Your obligations under this Public License where the Licensed
Rights include other Copyright and Similar Rights.
Section 5 -- Disclaimer of Warranties and Limitation of Liability.
a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
c. The disclaimer of warranties and limitation of liability provided
above shall be interpreted in a manner that, to the extent
possible, most closely approximates an absolute disclaimer and
waiver of all liability.
Section 6 -- Term and Termination.
a. This Public License applies for the term of the Copyright and
Similar Rights licensed here. However, if You fail to comply with
this Public License, then Your rights under this Public License
terminate automatically.
b. Where Your right to use the Licensed Material has terminated under
Section 6(a), it reinstates:
1. automatically as of the date the violation is cured, provided
it is cured within 30 days of Your discovery of the
violation; or
2. upon express reinstatement by the Licensor.
For the avoidance of doubt, this Section 6(b) does not affect any
right the Licensor may have to seek remedies for Your violations
of this Public License.
c. For the avoidance of doubt, the Licensor may also offer the
Licensed Material under separate terms or conditions or stop
distributing the Licensed Material at any time; however, doing so
will not terminate this Public License.
d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
License.
Section 7 -- Other Terms and Conditions.
a. The Licensor shall not be bound by any additional or different
terms or conditions communicated by You unless expressly agreed.
b. Any arrangements, understandings, or agreements regarding the
Licensed Material not stated herein are separate from and
independent of the terms and conditions of this Public License.
Section 8 -- Interpretation.
a. For the avoidance of doubt, this Public License does not, and
shall not be interpreted to, reduce, limit, restrict, or impose
conditions on any use of the Licensed Material that could lawfully
be made without permission under this Public License.
b. To the extent possible, if any provision of this Public License is
deemed unenforceable, it shall be automatically reformed to the
minimum extent necessary to make it enforceable. If the provision
cannot be reformed, it shall be severed from this Public License
without affecting the enforceability of the remaining terms and
conditions.
c. No term or condition of this Public License will be waived and no
failure to comply consented to unless expressly agreed to by the
Licensor.
d. Nothing in this Public License constitutes or may be interpreted
as a limitation upon, or waiver of, any privileges and immunities
that apply to the Licensor or You, including from the legal
processes of any jurisdiction or authority.
=======================================================================
Creative Commons is not a party to its public
licenses. Notwithstanding, Creative Commons may elect to apply one of
its public licenses to material it publishes and in those instances
will be considered the “Licensor.” The text of the Creative Commons
public licenses is dedicated to the public domain under the CC0 Public
Domain Dedication. Except for the limited purpose of indicating that
material is shared under a Creative Commons public license or as
otherwise permitted by the Creative Commons policies published at
creativecommons.org/policies, Creative Commons does not authorize the
use of the trademark "Creative Commons" or any other trademark or logo
of Creative Commons without its prior written consent including,
without limitation, in connection with any unauthorized modifications
to any of its public licenses or any other arrangements,
understandings, or agreements concerning use of licensed material. For
the avoidance of doubt, this paragraph does not form part of the
public licenses.
Creative Commons may be contacted at creativecommons.org.

127
emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim/README.md
View File

@@ -1,127 +0,0 @@
---
language: en
datasets:
- msp-podcast
inference: true
tags:
- speech
- audio
- wav2vec2
- audio-classification
- emotion-recognition
license: cc-by-nc-sa-4.0
pipeline_tag: audio-classification
---
# Model for Dimensional Speech Emotion Recognition based on Wav2vec 2.0
The model expects a raw audio signal as input and outputs predictions for arousal, dominance and valence in a range of approximately 0...1. In addition, it also provides the pooled states of the last transformer layer. The model was created by fine-tuning [
Wav2Vec2-Large-Robust](https://huggingface.co/facebook/wav2vec2-large-robust) on [MSP-Podcast](https://ecs.utdallas.edu/research/researchlabs/msp-lab/MSP-Podcast.html) (v1.7). The model was pruned from 24 to 12 transformer layers before fine-tuning. An [ONNX](https://onnx.ai/") export of the model is available from [doi:10.5281/zenodo.6221127](https://zenodo.org/record/6221127). Further details are given in the associated [paper](https://arxiv.org/abs/2203.07378) and [tutorial](https://github.com/audeering/w2v2-how-to).
# Usage
```python
import numpy as np
import torch
import torch.nn as nn
from transformers import Wav2Vec2Processor
from transformers.models.wav2vec2.modeling_wav2vec2 import (
Wav2Vec2Model,
Wav2Vec2PreTrainedModel,
)
class RegressionHead(nn.Module):
r"""Classification head."""
def __init__(self, config):
super().__init__()
self.dense = nn.Linear(config.hidden_size, config.hidden_size)
self.dropout = nn.Dropout(config.final_dropout)
self.out_proj = nn.Linear(config.hidden_size, config.num_labels)
def forward(self, features, **kwargs):
x = features
x = self.dropout(x)
x = self.dense(x)
x = torch.tanh(x)
x = self.dropout(x)
x = self.out_proj(x)
return x
class EmotionModel(Wav2Vec2PreTrainedModel):
r"""Speech emotion classifier."""
def __init__(self, config):
super().__init__(config)
self.config = config
self.wav2vec2 = Wav2Vec2Model(config)
self.classifier = RegressionHead(config)
self.init_weights()
def forward(
self,
input_values,
):
outputs = self.wav2vec2(input_values)
hidden_states = outputs[0]
hidden_states = torch.mean(hidden_states, dim=1)
logits = self.classifier(hidden_states)
return hidden_states, logits
# load model from hub
device = 'cpu'
model_name = 'audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim'
processor = Wav2Vec2Processor.from_pretrained(model_name)
model = EmotionModel.from_pretrained(model_name)
# dummy signal
sampling_rate = 16000
signal = np.zeros((1, sampling_rate), dtype=np.float32)
def process_func(
x: np.ndarray,
sampling_rate: int,
embeddings: bool = False,
) -> np.ndarray:
r"""Predict emotions or extract embeddings from raw audio signal."""
# run through processor to normalize signal
# always returns a batch, so we just get the first entry
# then we put it on the device
y = processor(x, sampling_rate=sampling_rate)
y = y['input_values'][0]
y = y.reshape(1, -1)
y = torch.from_numpy(y).to(device)
# run through model
with torch.no_grad():
y = model(y)[0 if embeddings else 1]
# convert to numpy
y = y.detach().cpu().numpy()
return y
print(process_func(signal, sampling_rate))
# Arousal dominance valence
# [[0.5460754 0.6062266 0.40431657]]
print(process_func(signal, sampling_rate, embeddings=True))
# Pooled hidden states of last transformer layer
# [[-0.00752167 0.0065819 -0.00746342 ... 0.00663632 0.00848748
# 0.00599211]]
```

122
emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim/config.json
View File

@@ -1,122 +0,0 @@
{
"_name_or_path": "torch",
"activation_dropout": 0.1,
"adapter_kernel_size": 3,
"adapter_stride": 2,
"add_adapter": false,
"apply_spec_augment": true,
"architectures": [
"Wav2Vec2ForSpeechClassification"
],
"attention_dropout": 0.1,
"bos_token_id": 1,
"classifier_proj_size": 256,
"codevector_dim": 768,
"contrastive_logits_temperature": 0.1,
"conv_bias": true,
"conv_dim": [
512,
512,
512,
512,
512,
512,
512
],
"conv_kernel": [
10,
3,
3,
3,
3,
2,
2
],
"conv_stride": [
5,
2,
2,
2,
2,
2,
2
],
"ctc_loss_reduction": "sum",
"ctc_zero_infinity": false,
"diversity_loss_weight": 0.1,
"do_stable_layer_norm": true,
"eos_token_id": 2,
"feat_extract_activation": "gelu",
"feat_extract_dropout": 0.0,
"feat_extract_norm": "layer",
"feat_proj_dropout": 0.1,
"feat_quantizer_dropout": 0.0,
"final_dropout": 0.1,
"finetuning_task": "wav2vec2_reg",
"gradient_checkpointing": false,
"hidden_act": "gelu",
"hidden_dropout": 0.1,
"hidden_dropout_prob": 0.1,
"hidden_size": 1024,
"id2label": {
"0": "arousal",
"1": "dominance",
"2": "valence"
},
"initializer_range": 0.02,
"intermediate_size": 4096,
"label2id": {
"arousal": 0,
"dominance": 1,
"valence": 2
},
"layer_norm_eps": 1e-05,
"layerdrop": 0.1,
"mask_feature_length": 10,
"mask_feature_min_masks": 0,
"mask_feature_prob": 0.0,
"mask_time_length": 10,
"mask_time_min_masks": 2,
"mask_time_prob": 0.05,
"model_type": "wav2vec2",
"num_adapter_layers": 3,
"num_attention_heads": 16,
"num_codevector_groups": 2,
"num_codevectors_per_group": 320,
"num_conv_pos_embedding_groups": 16,
"num_conv_pos_embeddings": 128,
"num_feat_extract_layers": 7,
"num_hidden_layers": 12,
"num_negatives": 100,
"output_hidden_size": 1024,
"pad_token_id": 0,
"pooling_mode": "mean",
"problem_type": "regression",
"proj_codevector_dim": 768,
"tdnn_dilation": [
1,
2,
3,
1,
1
],
"tdnn_dim": [
512,
512,
512,
512,
1500
],
"tdnn_kernel": [
5,
3,
3,
1,
1
],
"torch_dtype": "float32",
"transformers_version": "4.17.0.dev0",
"use_weighted_layer_sum": false,
"vocab_size": null,
"xvector_output_dim": 512
}

9
emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim/preprocessor_config.json
View File

@@ -1,9 +0,0 @@
{
"do_normalize": true,
"feature_extractor_type": "Wav2Vec2FeatureExtractor",
"feature_size": 1,
"padding_side": "right",
"padding_value": 0.0,
"return_attention_mask": true,
"sampling_rate": 16000
}

1
emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim/vocab.json
View File

@@ -1 +0,0 @@
{}

BIN
empty_emo.npy Normal file
View File

Binary file not shown.

26
get_emo.py
View File

@@ -1,26 +0,0 @@
from emo_gen import EmotionModel, process_func
import librosa
import numpy as np
import torch
from transformers import Wav2Vec2Processor
from config import config
model_name = "./emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim"
device = "cuda" if torch.cuda.is_available() else "cpu"
processor = Wav2Vec2Processor.from_pretrained(model_name)
model = EmotionModel.from_pretrained(model_name).to(device)
def get_emo(path):
wav, sr = librosa.load(path, 16000)
device = config.bert_gen_config.device
return process_func(
np.expand_dims(wav, 0).astype(np.float64),
sr,
model,
processor,
device,
embeddings=True,
).squeeze(0)

722
infer.py
View File

@@ -1,341 +1,381 @@
"""
版本管理、兼容推理及模型加载实现。
版本说明：
1. 版本号与github的release版本号对应，使用哪个release版本训练的模型即对应其版本号
2. 请在模型的config.json中显示声明版本号，添加一个字段"version" : "你的版本号"
特殊版本说明：
1.1.1-fix： 1.1.1版本训练的模型，但是在推理时使用dev的日语修复
1.1.1-dev： dev开发
2.1：当前版本
"""
import torch
import commons
from text import cleaned_text_to_sequence, get_bert
from get_emo import get_emo
from text.cleaner import clean_text
import utils
from models import SynthesizerTrn
from text.symbols import symbols
from oldVersion.V200.models import SynthesizerTrn as V200SynthesizerTrn
from oldVersion.V200.text import symbols as V200symbols
from oldVersion.V111.models import SynthesizerTrn as V111SynthesizerTrn
from oldVersion.V111.text import symbols as V111symbols
from oldVersion.V110.models import SynthesizerTrn as V110SynthesizerTrn
from oldVersion.V110.text import symbols as V110symbols
from oldVersion.V101.models import SynthesizerTrn as V101SynthesizerTrn
from oldVersion.V101.text import symbols as V101symbols
from oldVersion import V111, V110, V101, V200
# 当前版本信息
latest_version = "2.1"
# 版本兼容
SynthesizerTrnMap = {
"2.0.2-fix": V200SynthesizerTrn,
"2.0.1": V200SynthesizerTrn,
"2.0": V200SynthesizerTrn,
"1.1.1-fix": V111SynthesizerTrn,
"1.1.1": V111SynthesizerTrn,
"1.1": V110SynthesizerTrn,
"1.1.0": V110SynthesizerTrn,
"1.0.1": V101SynthesizerTrn,
"1.0": V101SynthesizerTrn,
"1.0.0": V101SynthesizerTrn,
}
symbolsMap = {
"2.0.2-fix": V200symbols,
"2.0.1": V200symbols,
"2.0": V200symbols,
"1.1.1-fix": V111symbols,
"1.1.1": V111symbols,
"1.1": V110symbols,
"1.1.0": V110symbols,
"1.0.1": V101symbols,
"1.0": V101symbols,
"1.0.0": V101symbols,
}
def get_net_g(model_path: str, version: str, device: str, hps):
if version != latest_version:
net_g = SynthesizerTrnMap[version](
len(symbolsMap[version]),
hps.data.filter_length // 2 + 1,
hps.train.segment_size // hps.data.hop_length,
n_speakers=hps.data.n_speakers,
**hps.model,
).to(device)
else:
# 当前版本模型 net_g
net_g = SynthesizerTrn(
len(symbols),
hps.data.filter_length // 2 + 1,
hps.train.segment_size // hps.data.hop_length,
n_speakers=hps.data.n_speakers,
**hps.model,
).to(device)
_ = net_g.eval()
_ = utils.load_checkpoint(model_path, net_g, None, skip_optimizer=True)
return net_g
def get_text(text, language_str, hps, device):
# 在此处实现当前版本的get_text
norm_text, phone, tone, word2ph = clean_text(text, language_str)
phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
if hps.data.add_blank:
phone = commons.intersperse(phone, 0)
tone = commons.intersperse(tone, 0)
language = commons.intersperse(language, 0)
for i in range(len(word2ph)):
word2ph[i] = word2ph[i] * 2
word2ph[0] += 1
bert_ori = get_bert(norm_text, word2ph, language_str, device)
del word2ph
assert bert_ori.shape[-1] == len(phone), phone
if language_str == "ZH":
bert = bert_ori
ja_bert = torch.zeros(1024, len(phone))
en_bert = torch.zeros(1024, len(phone))
elif language_str == "JP":
bert = torch.zeros(1024, len(phone))
ja_bert = bert_ori
en_bert = torch.zeros(1024, len(phone))
elif language_str == "EN":
bert = torch.zeros(1024, len(phone))
ja_bert = torch.zeros(1024, len(phone))
en_bert = bert_ori
else:
raise ValueError("language_str should be ZH, JP or EN")
assert bert.shape[-1] == len(
phone
), f"Bert seq len {bert.shape[-1]} != {len(phone)}"
phone = torch.LongTensor(phone)
tone = torch.LongTensor(tone)
language = torch.LongTensor(language)
return bert, ja_bert, en_bert, phone, tone, language
def get_emo_(reference_audio, emotion):
emo = (
torch.from_numpy(get_emo(reference_audio))
if reference_audio
else torch.Tensor([emotion])
)
return emo
def infer(
text,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
language,
hps,
net_g,
device,
reference_audio=None,
emotion=None,
skip_start=False,
skip_end=False,
):
# 支持中日英三语版本
inferMap_V2 = {
"2.0.2-fix": V200.infer,
"2.0.1": V200.infer,
"2.0": V200.infer,
"1.1.1-fix": V111.infer_fix,
"1.1.1": V111.infer,
"1.1": V110.infer,
"1.1.0": V110.infer,
}
# 仅支持中文版本
# 在测试中，并未发现两个版本的模型不能互相通用
inferMap_V1 = {
"1.0.1": V101.infer,
"1.0": V101.infer,
"1.0.0": V101.infer,
}
version = hps.version if hasattr(hps, "version") else latest_version
# 非当前版本，根据版本号选择合适的infer
if version != latest_version:
if version in inferMap_V2.keys():
return inferMap_V2[version](
text,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
language,
hps,
net_g,
device,
)
if version in inferMap_V1.keys():
return inferMap_V1[version](
text,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
hps,
net_g,
device,
)
# 在此处实现当前版本的推理
bert, ja_bert, en_bert, phones, tones, lang_ids = get_text(
text, language, hps, device
)
emo = get_emo_(reference_audio, emotion)
if skip_start:
phones = phones[3:]
tones = tones[3:]
lang_ids = lang_ids[3:]
bert = bert[:, 3:]
ja_bert = ja_bert[:, 3:]
en_bert = en_bert[:, 3:]
if skip_end:
phones = phones[:-2]
tones = tones[:-2]
lang_ids = lang_ids[:-2]
bert = bert[:, :-2]
ja_bert = ja_bert[:, :-2]
en_bert = en_bert[:, :-2]
with torch.no_grad():
x_tst = phones.to(device).unsqueeze(0)
tones = tones.to(device).unsqueeze(0)
lang_ids = lang_ids.to(device).unsqueeze(0)
bert = bert.to(device).unsqueeze(0)
ja_bert = ja_bert.to(device).unsqueeze(0)
en_bert = en_bert.to(device).unsqueeze(0)
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
emo = emo.to(device).unsqueeze(0)
del phones
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
audio = (
net_g.infer(
x_tst,
x_tst_lengths,
speakers,
tones,
lang_ids,
bert,
ja_bert,
en_bert,
emo,
sdp_ratio=sdp_ratio,
noise_scale=noise_scale,
noise_scale_w=noise_scale_w,
length_scale=length_scale,
)[0][0, 0]
.data.cpu()
.float()
.numpy()
)
del x_tst, tones, lang_ids, bert, x_tst_lengths, speakers, ja_bert, en_bert, emo
if torch.cuda.is_available():
torch.cuda.empty_cache()
return audio
def infer_multilang(
text,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
language,
hps,
net_g,
device,
reference_audio=None,
emotion=None,
skip_start=False,
skip_end=False,
):
bert, ja_bert, en_bert, phones, tones, lang_ids = [], [], [], [], [], []
emo = get_emo_(reference_audio, emotion)
for idx, (txt, lang) in enumerate(zip(text, language)):
skip_start = (idx != 0) or (skip_start and idx == 0)
skip_end = (idx != len(text) - 1) or (skip_end and idx == len(text) - 1)
(
temp_bert,
temp_ja_bert,
temp_en_bert,
temp_phones,
temp_tones,
temp_lang_ids,
) = get_text(txt, lang, hps, device)
if skip_start:
temp_bert = temp_bert[:, 3:]
temp_ja_bert = temp_ja_bert[:, 3:]
temp_en_bert = temp_en_bert[:, 3:]
temp_phones = temp_phones[3:]
temp_tones = temp_tones[3:]
temp_lang_ids = temp_lang_ids[3:]
if skip_end:
temp_bert = temp_bert[:, :-2]
temp_ja_bert = temp_ja_bert[:, :-2]
temp_en_bert = temp_en_bert[:, :-2]
temp_phones = temp_phones[:-2]
temp_tones = temp_tones[:-2]
temp_lang_ids = temp_lang_ids[:-2]
bert.append(temp_bert)
ja_bert.append(temp_ja_bert)
en_bert.append(temp_en_bert)
phones.append(temp_phones)
tones.append(temp_tones)
lang_ids.append(temp_lang_ids)
bert = torch.concatenate(bert, dim=1)
ja_bert = torch.concatenate(ja_bert, dim=1)
en_bert = torch.concatenate(en_bert, dim=1)
phones = torch.concatenate(phones, dim=0)
tones = torch.concatenate(tones, dim=0)
lang_ids = torch.concatenate(lang_ids, dim=0)
with torch.no_grad():
x_tst = phones.to(device).unsqueeze(0)
tones = tones.to(device).unsqueeze(0)
lang_ids = lang_ids.to(device).unsqueeze(0)
bert = bert.to(device).unsqueeze(0)
ja_bert = ja_bert.to(device).unsqueeze(0)
en_bert = en_bert.to(device).unsqueeze(0)
emo = emo.to(device).unsqueeze(0)
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
del phones
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
audio = (
net_g.infer(
x_tst,
x_tst_lengths,
speakers,
tones,
lang_ids,
bert,
ja_bert,
en_bert,
emo,
sdp_ratio=sdp_ratio,
noise_scale=noise_scale,
noise_scale_w=noise_scale_w,
length_scale=length_scale,
)[0][0, 0]
.data.cpu()
.float()
.numpy()
)
del x_tst, tones, lang_ids, bert, x_tst_lengths, speakers, ja_bert, en_bert, emo
if torch.cuda.is_available():
torch.cuda.empty_cache()
return audio
"""
版本管理、兼容推理及模型加载实现。
版本说明：
1. 版本号与github的release版本号对应，使用哪个release版本训练的模型即对应其版本号
2. 请在模型的config.json中显示声明版本号，添加一个字段"version" : "你的版本号"
特殊版本说明：
1.1.1-fix： 1.1.1版本训练的模型，但是在推理时使用dev的日语修复
2.2：当前版本
"""
import torch
import commons
from text import cleaned_text_to_sequence, get_bert
from clap_wrapper import get_clap_audio_feature, get_clap_text_feature
from text.cleaner import clean_text
import utils
import numpy as np
from models import SynthesizerTrn
from text.symbols import symbols
from oldVersion.V210.models import SynthesizerTrn as V210SynthesizerTrn
from oldVersion.V210.text import symbols as V210symbols
from oldVersion.V200.models import SynthesizerTrn as V200SynthesizerTrn
from oldVersion.V200.text import symbols as V200symbols
from oldVersion.V111.models import SynthesizerTrn as V111SynthesizerTrn
from oldVersion.V111.text import symbols as V111symbols
from oldVersion.V110.models import SynthesizerTrn as V110SynthesizerTrn
from oldVersion.V110.text import symbols as V110symbols
from oldVersion.V101.models import SynthesizerTrn as V101SynthesizerTrn
from oldVersion.V101.text import symbols as V101symbols
from oldVersion import V111, V110, V101, V200
# 当前版本信息
latest_version = "2.2"
# 版本兼容
SynthesizerTrnMap = {
"2.1": V210SynthesizerTrn,
"2.0.2-fix": V200SynthesizerTrn,
"2.0.1": V200SynthesizerTrn,
"2.0": V200SynthesizerTrn,
"1.1.1-fix": V111SynthesizerTrn,
"1.1.1": V111SynthesizerTrn,
"1.1": V110SynthesizerTrn,
"1.1.0": V110SynthesizerTrn,
"1.0.1": V101SynthesizerTrn,
"1.0": V101SynthesizerTrn,
"1.0.0": V101SynthesizerTrn,
}
symbolsMap = {
"2.1": V210symbols,
"2.0.2-fix": V200symbols,
"2.0.1": V200symbols,
"2.0": V200symbols,
"1.1.1-fix": V111symbols,
"1.1.1": V111symbols,
"1.1": V110symbols,
"1.1.0": V110symbols,
"1.0.1": V101symbols,
"1.0": V101symbols,
"1.0.0": V101symbols,
}
# def get_emo_(reference_audio, emotion, sid):
# emo = (
# torch.from_numpy(get_emo(reference_audio))
# if reference_audio and emotion == -1
# else torch.FloatTensor(
# np.load(f"emo_clustering/{sid}/cluster_center_{emotion}.npy")
# )
# )
# return emo
def get_net_g(model_path: str, version: str, device: str, hps):
if version != latest_version:
net_g = SynthesizerTrnMap[version](
len(symbolsMap[version]),
hps.data.filter_length // 2 + 1,
hps.train.segment_size // hps.data.hop_length,
n_speakers=hps.data.n_speakers,
**hps.model,
).to(device)
else:
# 当前版本模型 net_g
net_g = SynthesizerTrn(
len(symbols),
hps.data.filter_length // 2 + 1,
hps.train.segment_size // hps.data.hop_length,
n_speakers=hps.data.n_speakers,
**hps.model,
).to(device)
_ = net_g.eval()
_ = utils.load_checkpoint(model_path, net_g, None, skip_optimizer=True)
return net_g
def get_text(text, language_str, hps, device):
# 在此处实现当前版本的get_text
norm_text, phone, tone, word2ph = clean_text(text, language_str)
phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
if hps.data.add_blank:
phone = commons.intersperse(phone, 0)
tone = commons.intersperse(tone, 0)
language = commons.intersperse(language, 0)
for i in range(len(word2ph)):
word2ph[i] = word2ph[i] * 2
word2ph[0] += 1
bert_ori = get_bert(norm_text, word2ph, language_str, device)
del word2ph
assert bert_ori.shape[-1] == len(phone), phone
if language_str == "ZH":
bert = bert_ori
ja_bert = torch.rand(1024, len(phone))
en_bert = torch.rand(1024, len(phone))
elif language_str == "JP":
bert = torch.rand(1024, len(phone))
ja_bert = bert_ori
en_bert = torch.rand(1024, len(phone))
elif language_str == "EN":
bert = torch.rand(1024, len(phone))
ja_bert = torch.rand(1024, len(phone))
en_bert = bert_ori
else:
raise ValueError("language_str should be ZH, JP or EN")
assert bert.shape[-1] == len(
phone
), f"Bert seq len {bert.shape[-1]} != {len(phone)}"
phone = torch.LongTensor(phone)
tone = torch.LongTensor(tone)
language = torch.LongTensor(language)
return bert, ja_bert, en_bert, phone, tone, language
def infer(
text,
emotion,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
language,
hps,
net_g,
device,
reference_audio=None,
skip_start=False,
skip_end=False,
):
# 2.2版本参数位置变了
# 2.1 参数新增 emotion reference_audio skip_start skip_end
# inferMap_V3 = {
# "2.1": V210.infer,
# }
# 支持中日英三语版本
inferMap_V2 = {
"2.0.2-fix": V200.infer,
"2.0.1": V200.infer,
"2.0": V200.infer,
"1.1.1-fix": V111.infer_fix,
"1.1.1": V111.infer,
"1.1": V110.infer,
"1.1.0": V110.infer,
}
# 仅支持中文版本
# 在测试中，并未发现两个版本的模型不能互相通用
inferMap_V1 = {
"1.0.1": V101.infer,
"1.0": V101.infer,
"1.0.0": V101.infer,
}
version = hps.version if hasattr(hps, "version") else latest_version
# 非当前版本，根据版本号选择合适的infer
if version != latest_version:
# if version in inferMap_V3.keys():
# return inferMap_V3[version](
# text,
# sdp_ratio,
# noise_scale,
# noise_scale_w,
# length_scale,
# sid,
# language,
# hps,
# net_g,
# device,
# reference_audio,
# emotion,
# skip_start,
# skip_end,
# )
if version in inferMap_V2.keys():
return inferMap_V2[version](
text,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
language,
hps,
net_g,
device,
)
if version in inferMap_V1.keys():
return inferMap_V1[version](
text,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
hps,
net_g,
device,
)
# 在此处实现当前版本的推理
# emo = get_emo_(reference_audio, emotion, sid)
if isinstance(reference_audio, np.ndarray):
emo = get_clap_audio_feature(reference_audio, device)
else:
emo = get_clap_text_feature(emotion, device)
emo = torch.squeeze(emo, dim=1)
bert, ja_bert, en_bert, phones, tones, lang_ids = get_text(
text, language, hps, device
)
if skip_start:
phones = phones[3:]
tones = tones[3:]
lang_ids = lang_ids[3:]
bert = bert[:, 3:]
ja_bert = ja_bert[:, 3:]
en_bert = en_bert[:, 3:]
if skip_end:
phones = phones[:-2]
tones = tones[:-2]
lang_ids = lang_ids[:-2]
bert = bert[:, :-2]
ja_bert = ja_bert[:, :-2]
en_bert = en_bert[:, :-2]
with torch.no_grad():
x_tst = phones.to(device).unsqueeze(0)
tones = tones.to(device).unsqueeze(0)
lang_ids = lang_ids.to(device).unsqueeze(0)
bert = bert.to(device).unsqueeze(0)
ja_bert = ja_bert.to(device).unsqueeze(0)
en_bert = en_bert.to(device).unsqueeze(0)
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
emo = emo.to(device).unsqueeze(0)
del phones
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
audio = (
net_g.infer(
x_tst,
x_tst_lengths,
speakers,
tones,
lang_ids,
bert,
ja_bert,
en_bert,
emo,
sdp_ratio=sdp_ratio,
noise_scale=noise_scale,
noise_scale_w=noise_scale_w,
length_scale=length_scale,
)[0][0, 0]
.data.cpu()
.float()
.numpy()
)
del x_tst, tones, lang_ids, bert, x_tst_lengths, speakers, ja_bert, en_bert, emo
if torch.cuda.is_available():
torch.cuda.empty_cache()
return audio
def infer_multilang(
text,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
language,
hps,
net_g,
device,
reference_audio=None,
emotion=None,
skip_start=False,
skip_end=False,
):
bert, ja_bert, en_bert, phones, tones, lang_ids = [], [], [], [], [], []
# emo = get_emo_(reference_audio, emotion, sid)
if isinstance(reference_audio, np.ndarray):
emo = get_clap_audio_feature(reference_audio, device)
else:
emo = get_clap_text_feature(emotion, device)
emo = torch.squeeze(emo, dim=1)
for idx, (txt, lang) in enumerate(zip(text, language)):
skip_start = (idx != 0) or (skip_start and idx == 0)
skip_end = (idx != len(text) - 1) or (skip_end and idx == len(text) - 1)
(
temp_bert,
temp_ja_bert,
temp_en_bert,
temp_phones,
temp_tones,
temp_lang_ids,
) = get_text(txt, lang, hps, device)
if skip_start:
temp_bert = temp_bert[:, 3:]
temp_ja_bert = temp_ja_bert[:, 3:]
temp_en_bert = temp_en_bert[:, 3:]
temp_phones = temp_phones[3:]
temp_tones = temp_tones[3:]
temp_lang_ids = temp_lang_ids[3:]
if skip_end:
temp_bert = temp_bert[:, :-2]
temp_ja_bert = temp_ja_bert[:, :-2]
temp_en_bert = temp_en_bert[:, :-2]
temp_phones = temp_phones[:-2]
temp_tones = temp_tones[:-2]
temp_lang_ids = temp_lang_ids[:-2]
bert.append(temp_bert)
ja_bert.append(temp_ja_bert)
en_bert.append(temp_en_bert)
phones.append(temp_phones)
tones.append(temp_tones)
lang_ids.append(temp_lang_ids)
bert = torch.concatenate(bert, dim=1)
ja_bert = torch.concatenate(ja_bert, dim=1)
en_bert = torch.concatenate(en_bert, dim=1)
phones = torch.concatenate(phones, dim=0)
tones = torch.concatenate(tones, dim=0)
lang_ids = torch.concatenate(lang_ids, dim=0)
with torch.no_grad():
x_tst = phones.to(device).unsqueeze(0)
tones = tones.to(device).unsqueeze(0)
lang_ids = lang_ids.to(device).unsqueeze(0)
bert = bert.to(device).unsqueeze(0)
ja_bert = ja_bert.to(device).unsqueeze(0)
en_bert = en_bert.to(device).unsqueeze(0)
emo = emo.to(device).unsqueeze(0)
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
del phones
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
audio = (
net_g.infer(
x_tst,
x_tst_lengths,
speakers,
tones,
lang_ids,
bert,
ja_bert,
en_bert,
emo,
sdp_ratio=sdp_ratio,
noise_scale=noise_scale,
noise_scale_w=noise_scale_w,
length_scale=length_scale,
)[0][0, 0]
.data.cpu()
.float()
.numpy()
)
del x_tst, tones, lang_ids, bert, x_tst_lengths, speakers, ja_bert, en_bert, emo
if torch.cuda.is_available():
torch.cuda.empty_cache()
return audio

2115
models.py
View File

File diff suppressed because it is too large Load Diff

221
oldVersion/V210/__init__.py Normal file
View File

@@ -0,0 +1,221 @@
"""
@Desc: 2.1版本兼容 对应版本 v2.1 Emo and muti-lang optimize
"""
import torch
import commons
from .text import cleaned_text_to_sequence, get_bert
from .text.cleaner import clean_text
from .emo_gen import get_emo
def get_text(text, language_str, hps, device):
# 在此处实现当前版本的get_text
norm_text, phone, tone, word2ph = clean_text(text, language_str)
phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
if hps.data.add_blank:
phone = commons.intersperse(phone, 0)
tone = commons.intersperse(tone, 0)
language = commons.intersperse(language, 0)
for i in range(len(word2ph)):
word2ph[i] = word2ph[i] * 2
word2ph[0] += 1
bert_ori = get_bert(norm_text, word2ph, language_str, device)
del word2ph
assert bert_ori.shape[-1] == len(phone), phone
if language_str == "ZH":
bert = bert_ori
ja_bert = torch.zeros(1024, len(phone))
en_bert = torch.zeros(1024, len(phone))
elif language_str == "JP":
bert = torch.zeros(1024, len(phone))
ja_bert = bert_ori
en_bert = torch.zeros(1024, len(phone))
elif language_str == "EN":
bert = torch.zeros(1024, len(phone))
ja_bert = torch.zeros(1024, len(phone))
en_bert = bert_ori
else:
raise ValueError("language_str should be ZH, JP or EN")
assert bert.shape[-1] == len(
phone
), f"Bert seq len {bert.shape[-1]} != {len(phone)}"
phone = torch.LongTensor(phone)
tone = torch.LongTensor(tone)
language = torch.LongTensor(language)
return bert, ja_bert, en_bert, phone, tone, language
def get_emo_(reference_audio, emotion):
emo = (
torch.from_numpy(get_emo(reference_audio))
if reference_audio
else torch.Tensor([emotion])
)
return emo
def infer(
text,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
language,
hps,
net_g,
device,
reference_audio=None,
emotion=None,
skip_start=False,
skip_end=False,
):
bert, ja_bert, en_bert, phones, tones, lang_ids = get_text(
text, language, hps, device
)
emo = get_emo_(reference_audio, emotion)
if skip_start:
phones = phones[1:]
tones = tones[1:]
lang_ids = lang_ids[1:]
bert = bert[:, 1:]
ja_bert = ja_bert[:, 1:]
en_bert = en_bert[:, 1:]
if skip_end:
phones = phones[:-1]
tones = tones[:-1]
lang_ids = lang_ids[:-1]
bert = bert[:, :-1]
ja_bert = ja_bert[:, :-1]
en_bert = en_bert[:, :-1]
with torch.no_grad():
x_tst = phones.to(device).unsqueeze(0)
tones = tones.to(device).unsqueeze(0)
lang_ids = lang_ids.to(device).unsqueeze(0)
bert = bert.to(device).unsqueeze(0)
ja_bert = ja_bert.to(device).unsqueeze(0)
en_bert = en_bert.to(device).unsqueeze(0)
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
emo = emo.to(device).unsqueeze(0)
del phones
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
audio = (
net_g.infer(
x_tst,
x_tst_lengths,
speakers,
tones,
lang_ids,
bert,
ja_bert,
en_bert,
emo,
sdp_ratio=sdp_ratio,
noise_scale=noise_scale,
noise_scale_w=noise_scale_w,
length_scale=length_scale,
)[0][0, 0]
.data.cpu()
.float()
.numpy()
)
del x_tst, tones, lang_ids, bert, x_tst_lengths, speakers, ja_bert, en_bert, emo
if torch.cuda.is_available():
torch.cuda.empty_cache()
return audio
def infer_multilang(
text,
sdp_ratio,
noise_scale,
noise_scale_w,
length_scale,
sid,
language,
hps,
net_g,
device,
reference_audio=None,
emotion=None,
skip_start=False,
skip_end=False,
):
bert, ja_bert, en_bert, phones, tones, lang_ids = [], [], [], [], [], []
emo = get_emo_(reference_audio, emotion)
for idx, (txt, lang) in enumerate(zip(text, language)):
skip_start = (idx != 0) or (skip_start and idx == 0)
skip_end = (idx != len(text) - 1) or (skip_end and idx == len(text) - 1)
(
temp_bert,
temp_ja_bert,
temp_en_bert,
temp_phones,
temp_tones,
temp_lang_ids,
) = get_text(txt, lang, hps, device)
if skip_start:
temp_bert = temp_bert[:, 1:]
temp_ja_bert = temp_ja_bert[:, 1:]
temp_en_bert = temp_en_bert[:, 1:]
temp_phones = temp_phones[1:]
temp_tones = temp_tones[1:]
temp_lang_ids = temp_lang_ids[1:]
if skip_end:
temp_bert = temp_bert[:, :-1]
temp_ja_bert = temp_ja_bert[:, :-1]
temp_en_bert = temp_en_bert[:, :-1]
temp_phones = temp_phones[:-1]
temp_tones = temp_tones[:-1]
temp_lang_ids = temp_lang_ids[:-1]
bert.append(temp_bert)
ja_bert.append(temp_ja_bert)
en_bert.append(temp_en_bert)
phones.append(temp_phones)
tones.append(temp_tones)
lang_ids.append(temp_lang_ids)
bert = torch.concatenate(bert, dim=1)
ja_bert = torch.concatenate(ja_bert, dim=1)
en_bert = torch.concatenate(en_bert, dim=1)
phones = torch.concatenate(phones, dim=0)
tones = torch.concatenate(tones, dim=0)
lang_ids = torch.concatenate(lang_ids, dim=0)
with torch.no_grad():
x_tst = phones.to(device).unsqueeze(0)
tones = tones.to(device).unsqueeze(0)
lang_ids = lang_ids.to(device).unsqueeze(0)
bert = bert.to(device).unsqueeze(0)
ja_bert = ja_bert.to(device).unsqueeze(0)
en_bert = en_bert.to(device).unsqueeze(0)
emo = emo.to(device).unsqueeze(0)
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
del phones
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
audio = (
net_g.infer(
x_tst,
x_tst_lengths,
speakers,
tones,
lang_ids,
bert,
ja_bert,
en_bert,
emo,
sdp_ratio=sdp_ratio,
noise_scale=noise_scale,
noise_scale_w=noise_scale_w,
length_scale=length_scale,
)[0][0, 0]
.data.cpu()
.float()
.numpy()
)
del x_tst, tones, lang_ids, bert, x_tst_lengths, speakers, ja_bert, en_bert, emo
if torch.cuda.is_available():
torch.cuda.empty_cache()
return audio

272
emo_gen.py → oldVersion/V210/emo_gen.py
View File

@@ -1,155 +1,117 @@
import argparse
import os
from pathlib import Path
import librosa
import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import Dataset
from torch.utils.data import DataLoader, Dataset
from tqdm import tqdm
from transformers import Wav2Vec2Processor
from transformers.models.wav2vec2.modeling_wav2vec2 import (
Wav2Vec2Model,
Wav2Vec2PreTrainedModel,
)
import utils
from config import config
class RegressionHead(nn.Module):
r"""Classification head."""
def __init__(self, config):
super().__init__()
self.dense = nn.Linear(config.hidden_size, config.hidden_size)
self.dropout = nn.Dropout(config.final_dropout)
self.out_proj = nn.Linear(config.hidden_size, config.num_labels)
def forward(self, features, **kwargs):
x = features
x = self.dropout(x)
x = self.dense(x)
x = torch.tanh(x)
x = self.dropout(x)
x = self.out_proj(x)
return x
class EmotionModel(Wav2Vec2PreTrainedModel):
r"""Speech emotion classifier."""
def __init__(self, config):
super().__init__(config)
self.config = config
self.wav2vec2 = Wav2Vec2Model(config)
self.classifier = RegressionHead(config)
self.init_weights()
def forward(
self,
input_values,
):
outputs = self.wav2vec2(input_values)
hidden_states = outputs[0]
hidden_states = torch.mean(hidden_states, dim=1)
logits = self.classifier(hidden_states)
return hidden_states, logits
class AudioDataset(Dataset):
def __init__(self, list_of_wav_files, sr, processor):
self.list_of_wav_files = list_of_wav_files
self.processor = processor
self.sr = sr
def __len__(self):
return len(self.list_of_wav_files)
def __getitem__(self, idx):
wav_file = self.list_of_wav_files[idx]
audio_data, _ = librosa.load(wav_file, sr=self.sr)
processed_data = self.processor(audio_data, sampling_rate=self.sr)[
"input_values"
][0]
return torch.from_numpy(processed_data)
def process_func(
x: np.ndarray,
sampling_rate: int,
model: EmotionModel,
processor: Wav2Vec2Processor,
device: str,
embeddings: bool = False,
) -> np.ndarray:
r"""Predict emotions or extract embeddings from raw audio signal."""
model = model.to(device)
y = processor(x, sampling_rate=sampling_rate)
y = y["input_values"][0]
y = torch.from_numpy(y).unsqueeze(0).to(device)
# run through model
with torch.no_grad():
y = model(y)[0 if embeddings else 1]
# convert to numpy
y = y.detach().cpu().numpy()
return y
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument(
"-c", "--config", type=str, default=config.bert_gen_config.config_path
)
parser.add_argument(
"--num_processes", type=int, default=config.bert_gen_config.num_processes
)
args, _ = parser.parse_known_args()
config_path = args.config
hps = utils.get_hparams_from_file(config_path)
device = config.bert_gen_config.device
model_name = "./emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim"
REPO_ID = "audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim"
if not Path(model_name).joinpath("pytorch_model.bin").exists():
utils.download_emo_models(config.mirror, REPO_ID, model_name)
processor = Wav2Vec2Processor.from_pretrained(model_name)
model = EmotionModel.from_pretrained(model_name).to(device)
lines = []
with open(hps.data.training_files, encoding="utf-8") as f:
lines.extend(f.readlines())
with open(hps.data.validation_files, encoding="utf-8") as f:
lines.extend(f.readlines())
wavnames = [line.split("|")[0] for line in lines]
dataset = AudioDataset(wavnames, 16000, processor)
data_loader = DataLoader(
dataset,
batch_size=1,
shuffle=False,
num_workers=min(args.num_processes, os.cpu_count() - 1),
)
with torch.no_grad():
for i, data in tqdm(enumerate(data_loader), total=len(data_loader)):
wavname = wavnames[i]
emo_path = wavname.replace(".wav", ".emo.npy")
if os.path.exists(emo_path):
continue
emb = model(data.to(device))[0].detach().cpu().numpy()
np.save(emo_path, emb)
print("Emo vec 生成完毕!")
import librosa
import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import Dataset
from torch.utils.data import Dataset
from transformers import Wav2Vec2Processor
from transformers.models.wav2vec2.modeling_wav2vec2 import (
Wav2Vec2Model,
Wav2Vec2PreTrainedModel,
)
from config import config
class RegressionHead(nn.Module):
r"""Classification head."""
def __init__(self, config):
super().__init__()
self.dense = nn.Linear(config.hidden_size, config.hidden_size)
self.dropout = nn.Dropout(config.final_dropout)
self.out_proj = nn.Linear(config.hidden_size, config.num_labels)
def forward(self, features, **kwargs):
x = features
x = self.dropout(x)
x = self.dense(x)
x = torch.tanh(x)
x = self.dropout(x)
x = self.out_proj(x)
return x
class EmotionModel(Wav2Vec2PreTrainedModel):
r"""Speech emotion classifier."""
def __init__(self, config):
super().__init__(config)
self.config = config
self.wav2vec2 = Wav2Vec2Model(config)
self.classifier = RegressionHead(config)
self.init_weights()
def forward(
self,
input_values,
):
outputs = self.wav2vec2(input_values)
hidden_states = outputs[0]
hidden_states = torch.mean(hidden_states, dim=1)
logits = self.classifier(hidden_states)
return hidden_states, logits
class AudioDataset(Dataset):
def __init__(self, list_of_wav_files, sr, processor):
self.list_of_wav_files = list_of_wav_files
self.processor = processor
self.sr = sr
def __len__(self):
return len(self.list_of_wav_files)
def __getitem__(self, idx):
wav_file = self.list_of_wav_files[idx]
audio_data, _ = librosa.load(wav_file, sr=self.sr)
processed_data = self.processor(audio_data, sampling_rate=self.sr)[
"input_values"
][0]
return torch.from_numpy(processed_data)
device = config.emo_gen_config.device
model_name = "./emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim"
processor = Wav2Vec2Processor.from_pretrained(model_name)
model = EmotionModel.from_pretrained(model_name).to(device)
def process_func(
x: np.ndarray,
sampling_rate: int,
model: EmotionModel,
processor: Wav2Vec2Processor,
device: str,
embeddings: bool = False,
) -> np.ndarray:
r"""Predict emotions or extract embeddings from raw audio signal."""
model = model.to(device)
y = processor(x, sampling_rate=sampling_rate)
y = y["input_values"][0]
y = torch.from_numpy(y).unsqueeze(0).to(device)
# run through model
with torch.no_grad():
y = model(y)[0 if embeddings else 1]
# convert to numpy
y = y.detach().cpu().numpy()
return y
def get_emo(path):
wav, sr = librosa.load(path, 16000)
return process_func(
np.expand_dims(wav, 0).astype(np.float64),
sr,
model,
processor,
device,
embeddings=True,
).squeeze(0)

1040
oldVersion/V210/models.py Normal file
View File

File diff suppressed because it is too large Load Diff

51
oldVersion/V210/text/__init__.py Normal file
View File

@@ -0,0 +1,51 @@
from .symbols import *
_symbol_to_id = {s: i for i, s in enumerate(symbols)}
def cleaned_text_to_sequence(cleaned_text, tones, language):
"""Converts a string of text to a sequence of IDs corresponding to the symbols in the text.
Args:
text: string to convert to a sequence
Returns:
List of integers corresponding to the symbols in the text
"""
phones = [_symbol_to_id[symbol] for symbol in cleaned_text]
tone_start = language_tone_start_map[language]
tones = [i + tone_start for i in tones]
lang_id = language_id_map[language]
lang_ids = [lang_id for i in phones]
return phones, tones, lang_ids
def get_bert(norm_text, word2ph, language, device):
from .chinese_bert import get_bert_feature as zh_bert
from .english_bert_mock import get_bert_feature as en_bert
from .japanese_bert import get_bert_feature as jp_bert
lang_bert_func_map = {"ZH": zh_bert, "EN": en_bert, "JP": jp_bert}
bert = lang_bert_func_map[language](norm_text, word2ph, device)
return bert
def check_bert_models():
import json
from pathlib import Path
from config import config
from .bert_utils import _check_bert
if config.mirror.lower() == "openi":
import openi
kwargs = {"token": config.openi_token} if config.openi_token else {}
openi.login(**kwargs)
with open("./bert/bert_models.json", "r") as fp:
models = json.load(fp)
for k, v in models.items():
local_path = Path("./bert").joinpath(k)
_check_bert(v["repo_id"], v["files"], local_path)
check_bert_models()

23
oldVersion/V210/text/bert_utils.py Normal file
View File

@@ -0,0 +1,23 @@
from pathlib import Path
from huggingface_hub import hf_hub_download
from config import config
MIRROR: str = config.mirror
def _check_bert(repo_id, files, local_path):
for file in files:
if not Path(local_path).joinpath(file).exists():
if MIRROR.lower() == "openi":
import openi
openi.model.download_model(
"Stardust_minus/Bert-VITS2", repo_id.split("/")[-1], "./bert"
)
else:
hf_hub_download(
repo_id, file, local_dir=local_path, local_dir_use_symlinks=False
)

199
oldVersion/V210/text/chinese.py Normal file
View File

@@ -0,0 +1,199 @@
import os
import re
import cn2an
from pypinyin import lazy_pinyin, Style
from .symbols import punctuation
from .tone_sandhi import ToneSandhi
current_file_path = os.path.dirname(__file__)
pinyin_to_symbol_map = {
line.split("\t")[0]: line.strip().split("\t")[1]
for line in open(os.path.join(current_file_path, "opencpop-strict.txt")).readlines()
}
import jieba.posseg as psg
rep_map = {
"：": ",",
"；": ",",
"，": ",",
"。": ".",
"！": "!",
"？": "?",
"\n": ".",
"·": ",",
"、": ",",
"...": "…",
"$": ".",
"“": "'",
"”": "'",
'"': "'",
"‘": "'",
"’": "'",
"（": "'",
"）": "'",
"(": "'",
")": "'",
"《": "'",
"》": "'",
"【": "'",
"】": "'",
"[": "'",
"]": "'",
"—": "-",
"～": "-",
"~": "-",
"「": "'",
"」": "'",
}
tone_modifier = ToneSandhi()
def replace_punctuation(text):
text = text.replace("嗯", "恩").replace("呣", "母")
pattern = re.compile("|".join(re.escape(p) for p in rep_map.keys()))
replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
replaced_text = re.sub(
r"[^\u4e00-\u9fa5" + "".join(punctuation) + r"]+", "", replaced_text
)
return replaced_text
def g2p(text):
pattern = r"(?<=[{0}])\s*".format("".join(punctuation))
sentences = [i for i in re.split(pattern, text) if i.strip() != ""]
phones, tones, word2ph = _g2p(sentences)
assert sum(word2ph) == len(phones)
assert len(word2ph) == len(text) # Sometimes it will crash,you can add a try-catch.
phones = ["_"] + phones + ["_"]
tones = [0] + tones + [0]
word2ph = [1] + word2ph + [1]
return phones, tones, word2ph
def _get_initials_finals(word):
initials = []
finals = []
orig_initials = lazy_pinyin(word, neutral_tone_with_five=True, style=Style.INITIALS)
orig_finals = lazy_pinyin(
word, neutral_tone_with_five=True, style=Style.FINALS_TONE3
)
for c, v in zip(orig_initials, orig_finals):
initials.append(c)
finals.append(v)
return initials, finals
def _g2p(segments):
phones_list = []
tones_list = []
word2ph = []
for seg in segments:
# Replace all English words in the sentence
seg = re.sub("[a-zA-Z]+", "", seg)
seg_cut = psg.lcut(seg)
initials = []
finals = []
seg_cut = tone_modifier.pre_merge_for_modify(seg_cut)
for word, pos in seg_cut:
if pos == "eng":
continue
sub_initials, sub_finals = _get_initials_finals(word)
sub_finals = tone_modifier.modified_tone(word, pos, sub_finals)
initials.append(sub_initials)
finals.append(sub_finals)
# assert len(sub_initials) == len(sub_finals) == len(word)
initials = sum(initials, [])
finals = sum(finals, [])
#
for c, v in zip(initials, finals):
raw_pinyin = c + v
# NOTE: post process for pypinyin outputs
# we discriminate i, ii and iii
if c == v:
assert c in punctuation
phone = [c]
tone = "0"
word2ph.append(1)
else:
v_without_tone = v[:-1]
tone = v[-1]
pinyin = c + v_without_tone
assert tone in "12345"
if c:
# 多音节
v_rep_map = {
"uei": "ui",
"iou": "iu",
"uen": "un",
}
if v_without_tone in v_rep_map.keys():
pinyin = c + v_rep_map[v_without_tone]
else:
# 单音节
pinyin_rep_map = {
"ing": "ying",
"i": "yi",
"in": "yin",
"u": "wu",
}
if pinyin in pinyin_rep_map.keys():
pinyin = pinyin_rep_map[pinyin]
else:
single_rep_map = {
"v": "yu",
"e": "e",
"i": "y",
"u": "w",
}
if pinyin[0] in single_rep_map.keys():
pinyin = single_rep_map[pinyin[0]] + pinyin[1:]
assert pinyin in pinyin_to_symbol_map.keys(), (pinyin, seg, raw_pinyin)
phone = pinyin_to_symbol_map[pinyin].split(" ")
word2ph.append(len(phone))
phones_list += phone
tones_list += [int(tone)] * len(phone)
return phones_list, tones_list, word2ph
def text_normalize(text):
numbers = re.findall(r"\d+(?:\.?\d+)?", text)
for number in numbers:
text = text.replace(number, cn2an.an2cn(number), 1)
text = replace_punctuation(text)
return text
def get_bert_feature(text, word2ph):
from text import chinese_bert
return chinese_bert.get_bert_feature(text, word2ph)
if __name__ == "__main__":
from text.chinese_bert import get_bert_feature
text = "啊！但是《原神》是由,米哈\游自主， [研发]的一款全.新开放世界.冒险游戏"
text = text_normalize(text)
print(text)
phones, tones, word2ph = g2p(text)
bert = get_bert_feature(text, word2ph)
print(phones, tones, word2ph, bert.shape)
# # 示例用法
# text = "这是一个示例文本：,你好！这是一个测试...."
# print(g2p_paddle(text)) # 输出: 这是一个示例文本你好这是一个测试

101
oldVersion/V210/text/chinese_bert.py Normal file
View File

@@ -0,0 +1,101 @@
import sys
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer
from config import config
LOCAL_PATH = "./bert/chinese-roberta-wwm-ext-large"
tokenizer = AutoTokenizer.from_pretrained(LOCAL_PATH)
models = dict()
def get_bert_feature(text, word2ph, device=config.bert_gen_config.device):
if (
sys.platform == "darwin"
and torch.backends.mps.is_available()
and device == "cpu"
):
device = "mps"
if not device:
device = "cuda"
if device not in models.keys():
models[device] = AutoModelForMaskedLM.from_pretrained(LOCAL_PATH).to(device)
with torch.no_grad():
inputs = tokenizer(text, return_tensors="pt")
for i in inputs:
inputs[i] = inputs[i].to(device)
res = models[device](**inputs, output_hidden_states=True)
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
assert len(word2ph) == len(text) + 2
word2phone = word2ph
phone_level_feature = []
for i in range(len(word2phone)):
repeat_feature = res[i].repeat(word2phone[i], 1)
phone_level_feature.append(repeat_feature)
phone_level_feature = torch.cat(phone_level_feature, dim=0)
return phone_level_feature.T
if __name__ == "__main__":
word_level_feature = torch.rand(38, 1024) # 12个词,每个词1024维特征
word2phone = [
1,
2,
1,
2,
2,
1,
2,
2,
1,
2,
2,
1,
2,
2,
2,
2,
2,
1,
1,
2,
2,
1,
2,
2,
2,
2,
1,
2,
2,
2,
2,
2,
1,
2,
2,
2,
2,
1,
]
# 计算总帧数
total_frames = sum(word2phone)
print(word_level_feature.shape)
print(word2phone)
phone_level_feature = []
for i in range(len(word2phone)):
print(word_level_feature[i].shape)
# 对每个词重复word2phone[i]次
repeat_feature = word_level_feature[i].repeat(word2phone[i], 1)
phone_level_feature.append(repeat_feature)
phone_level_feature = torch.cat(phone_level_feature, dim=0)
print(phone_level_feature.shape) # torch.Size([36, 1024])

28
oldVersion/V210/text/cleaner.py Normal file
View File

@@ -0,0 +1,28 @@
from . import chinese, japanese, english, cleaned_text_to_sequence
language_module_map = {"ZH": chinese, "JP": japanese, "EN": english}
def clean_text(text, language):
language_module = language_module_map[language]
norm_text = language_module.text_normalize(text)
phones, tones, word2ph = language_module.g2p(norm_text)
return norm_text, phones, tones, word2ph
def clean_text_bert(text, language):
language_module = language_module_map[language]
norm_text = language_module.text_normalize(text)
phones, tones, word2ph = language_module.g2p(norm_text)
bert = language_module.get_bert_feature(norm_text, word2ph)
return phones, tones, bert
def text_to_sequence(text, language):
norm_text, phones, tones, word2ph = clean_text(text, language)
return cleaned_text_to_sequence(phones, tones, language)
if __name__ == "__main__":
pass

129530
oldVersion/V210/text/cmudict.rep Normal file
View File

File diff suppressed because it is too large Load Diff

BIN
oldVersion/V210/text/cmudict_cache.pickle Normal file
View File

Binary file not shown.

453
oldVersion/V210/text/english.py Normal file
View File

@@ -0,0 +1,453 @@
import pickle
import os
import re
from g2p_en import G2p
from transformers import DebertaV2Tokenizer
from . import symbols
current_file_path = os.path.dirname(__file__)
CMU_DICT_PATH = os.path.join(current_file_path, "cmudict.rep")
CACHE_PATH = os.path.join(current_file_path, "cmudict_cache.pickle")
_g2p = G2p()
LOCAL_PATH = "./bert/deberta-v3-large"
tokenizer = DebertaV2Tokenizer.from_pretrained(LOCAL_PATH)
arpa = {
"AH0",
"S",
"AH1",
"EY2",
"AE2",
"EH0",
"OW2",
"UH0",
"NG",
"B",
"G",
"AY0",
"M",
"AA0",
"F",
"AO0",
"ER2",
"UH1",
"IY1",
"AH2",
"DH",
"IY0",
"EY1",
"IH0",
"K",
"N",
"W",
"IY2",
"T",
"AA1",
"ER1",
"EH2",
"OY0",
"UH2",
"UW1",
"Z",
"AW2",
"AW1",
"V",
"UW2",
"AA2",
"ER",
"AW0",
"UW0",
"R",
"OW1",
"EH1",
"ZH",
"AE0",
"IH2",
"IH",
"Y",
"JH",
"P",
"AY1",
"EY0",
"OY2",
"TH",
"HH",
"D",
"ER0",
"CH",
"AO1",
"AE1",
"AO2",
"OY1",
"AY2",
"IH1",
"OW0",
"L",
"SH",
}
def post_replace_ph(ph):
rep_map = {
"：": ",",
"；": ",",
"，": ",",
"。": ".",
"！": "!",
"？": "?",
"\n": ".",
"·": ",",
"、": ",",
"…": "...",
"···": "...",
"・・・": "...",
"v": "V",
}
if ph in rep_map.keys():
ph = rep_map[ph]
if ph in symbols:
return ph
if ph not in symbols:
ph = "UNK"
return ph
rep_map = {
"：": ",",
"；": ",",
"，": ",",
"。": ".",
"！": "!",
"？": "?",
"\n": ".",
"．": ".",
"…": "...",
"···": "...",
"・・・": "...",
"·": ",",
"・": ",",
"、": ",",
"$": ".",
"“": "'",
"”": "'",
'"': "'",
"‘": "'",
"’": "'",
"（": "'",
"）": "'",
"(": "'",
")": "'",
"《": "'",
"》": "'",
"【": "'",
"】": "'",
"[": "'",
"]": "'",
"—": "-",
"−": "-",
"～": "-",
"~": "-",
"「": "'",
"」": "'",
}
def replace_punctuation(text):
pattern = re.compile("|".join(re.escape(p) for p in rep_map.keys()))
replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
# replaced_text = re.sub(
# r"[^\u3040-\u309F\u30A0-\u30FF\u4E00-\u9FFF\u3400-\u4DBF\u3005"
# + "".join(punctuation)
# + r"]+",
# "",
# replaced_text,
# )
return replaced_text
def read_dict():
g2p_dict = {}
start_line = 49
with open(CMU_DICT_PATH) as f:
line = f.readline()
line_index = 1
while line:
if line_index >= start_line:
line = line.strip()
word_split = line.split(" ")
word = word_split[0]
syllable_split = word_split[1].split(" - ")
g2p_dict[word] = []
for syllable in syllable_split:
phone_split = syllable.split(" ")
g2p_dict[word].append(phone_split)
line_index = line_index + 1
line = f.readline()
return g2p_dict
def cache_dict(g2p_dict, file_path):
with open(file_path, "wb") as pickle_file:
pickle.dump(g2p_dict, pickle_file)
def get_dict():
if os.path.exists(CACHE_PATH):
with open(CACHE_PATH, "rb") as pickle_file:
g2p_dict = pickle.load(pickle_file)
else:
g2p_dict = read_dict()
cache_dict(g2p_dict, CACHE_PATH)
return g2p_dict
eng_dict = get_dict()
def refine_ph(phn):
tone = 0
if re.search(r"\d$", phn):
tone = int(phn[-1]) + 1
phn = phn[:-1]
return phn.lower(), tone
def refine_syllables(syllables):
tones = []
phonemes = []
for phn_list in syllables:
for i in range(len(phn_list)):
phn = phn_list[i]
phn, tone = refine_ph(phn)
phonemes.append(phn)
tones.append(tone)
return phonemes, tones
import re
import inflect
_inflect = inflect.engine()
_comma_number_re = re.compile(r"([0-9][0-9\,]+[0-9])")
_decimal_number_re = re.compile(r"([0-9]+\.[0-9]+)")
_pounds_re = re.compile(r"£([0-9\,]*[0-9]+)")
_dollars_re = re.compile(r"\$([0-9\.\,]*[0-9]+)")
_ordinal_re = re.compile(r"[0-9]+(st|nd|rd|th)")
_number_re = re.compile(r"[0-9]+")
# List of (regular expression, replacement) pairs for abbreviations:
_abbreviations = [
(re.compile("\\b%s\\." % x[0], re.IGNORECASE), x[1])
for x in [
("mrs", "misess"),
("mr", "mister"),
("dr", "doctor"),
("st", "saint"),
("co", "company"),
("jr", "junior"),
("maj", "major"),
("gen", "general"),
("drs", "doctors"),
("rev", "reverend"),
("lt", "lieutenant"),
("hon", "honorable"),
("sgt", "sergeant"),
("capt", "captain"),
("esq", "esquire"),
("ltd", "limited"),
("col", "colonel"),
("ft", "fort"),
]
]
# List of (ipa, lazy ipa) pairs:
_lazy_ipa = [
(re.compile("%s" % x[0]), x[1])
for x in [
("r", "ɹ"),
("æ", "e"),
("ɑ", "a"),
("ɔ", "o"),
("ð", "z"),
("θ", "s"),
("ɛ", "e"),
("ɪ", "i"),
("ʊ", "u"),
("ʒ", "ʥ"),
("ʤ", "ʥ"),
("ˈ", "↓"),
]
]
# List of (ipa, lazy ipa2) pairs:
_lazy_ipa2 = [
(re.compile("%s" % x[0]), x[1])
for x in [
("r", "ɹ"),
("ð", "z"),
("θ", "s"),
("ʒ", "ʑ"),
("ʤ", "dʑ"),
("ˈ", "↓"),
]
]
# List of (ipa, ipa2) pairs
_ipa_to_ipa2 = [
(re.compile("%s" % x[0]), x[1]) for x in [("r", "ɹ"), ("ʤ", "dʒ"), ("ʧ", "tʃ")]
]
def _expand_dollars(m):
match = m.group(1)
parts = match.split(".")
if len(parts) > 2:
return match + " dollars" # Unexpected format
dollars = int(parts[0]) if parts[0] else 0
cents = int(parts[1]) if len(parts) > 1 and parts[1] else 0
if dollars and cents:
dollar_unit = "dollar" if dollars == 1 else "dollars"
cent_unit = "cent" if cents == 1 else "cents"
return "%s %s, %s %s" % (dollars, dollar_unit, cents, cent_unit)
elif dollars:
dollar_unit = "dollar" if dollars == 1 else "dollars"
return "%s %s" % (dollars, dollar_unit)
elif cents:
cent_unit = "cent" if cents == 1 else "cents"
return "%s %s" % (cents, cent_unit)
else:
return "zero dollars"
def _remove_commas(m):
return m.group(1).replace(",", "")
def _expand_ordinal(m):
return _inflect.number_to_words(m.group(0))
def _expand_number(m):
num = int(m.group(0))
if num > 1000 and num < 3000:
if num == 2000:
return "two thousand"
elif num > 2000 and num < 2010:
return "two thousand " + _inflect.number_to_words(num % 100)
elif num % 100 == 0:
return _inflect.number_to_words(num // 100) + " hundred"
else:
return _inflect.number_to_words(
num, andword="", zero="oh", group=2
).replace(", ", " ")
else:
return _inflect.number_to_words(num, andword="")
def _expand_decimal_point(m):
return m.group(1).replace(".", " point ")
def normalize_numbers(text):
text = re.sub(_comma_number_re, _remove_commas, text)
text = re.sub(_pounds_re, r"\1 pounds", text)
text = re.sub(_dollars_re, _expand_dollars, text)
text = re.sub(_decimal_number_re, _expand_decimal_point, text)
text = re.sub(_ordinal_re, _expand_ordinal, text)
text = re.sub(_number_re, _expand_number, text)
return text
def text_normalize(text):
text = normalize_numbers(text)
text = replace_punctuation(text)
text = re.sub(r"([,;.\?\!])([\w])", r"\1 \2", text)
return text
def distribute_phone(n_phone, n_word):
phones_per_word = [0] * n_word
for task in range(n_phone):
min_tasks = min(phones_per_word)
min_index = phones_per_word.index(min_tasks)
phones_per_word[min_index] += 1
return phones_per_word
def sep_text(text):
words = re.split(r"([,;.\?\!\s+])", text)
words = [word for word in words if word.strip() != ""]
return words
def g2p(text):
phones = []
tones = []
# word2ph = []
words = sep_text(text)
tokens = [tokenizer.tokenize(i) for i in words]
for word in words:
if word.upper() in eng_dict:
phns, tns = refine_syllables(eng_dict[word.upper()])
phones.append([post_replace_ph(i) for i in phns])
tones.append(tns)
# word2ph.append(len(phns))
else:
phone_list = list(filter(lambda p: p != " ", _g2p(word)))
phns = []
tns = []
for ph in phone_list:
if ph in arpa:
ph, tn = refine_ph(ph)
phns.append(ph)
tns.append(tn)
else:
phns.append(ph)
tns.append(0)
phones.append([post_replace_ph(i) for i in phns])
tones.append(tns)
# word2ph.append(len(phns))
# phones = [post_replace_ph(i) for i in phones]
word2ph = []
for token, phoneme in zip(tokens, phones):
phone_len = len(phoneme)
word_len = len(token)
aaa = distribute_phone(phone_len, word_len)
word2ph += aaa
phones = ["_"] + [j for i in phones for j in i] + ["_"]
tones = [0] + [j for i in tones for j in i] + [0]
word2ph = [1] + word2ph + [1]
assert len(phones) == len(tones), text
assert len(phones) == sum(word2ph), text
return phones, tones, word2ph
def get_bert_feature(text, word2ph):
from text import english_bert_mock
return english_bert_mock.get_bert_feature(text, word2ph)
if __name__ == "__main__":
# print(get_dict())
# print(eng_word_to_phoneme("hello"))
print(g2p("In this paper, we propose 1 DSPGAN, a GAN-based universal vocoder."))
# all_phones = set()
# for k, syllables in eng_dict.items():
# for group in syllables:
# for ph in group:
# all_phones.add(ph)
# print(all_phones)

42
oldVersion/V210/text/english_bert_mock.py Normal file
View File

@@ -0,0 +1,42 @@
import sys
import torch
from transformers import DebertaV2Model, DebertaV2Tokenizer
from config import config
LOCAL_PATH = "./bert/deberta-v3-large"
tokenizer = DebertaV2Tokenizer.from_pretrained(LOCAL_PATH)
models = dict()
def get_bert_feature(text, word2ph, device=config.bert_gen_config.device):
if (
sys.platform == "darwin"
and torch.backends.mps.is_available()
and device == "cpu"
):
device = "mps"
if not device:
device = "cuda"
if device not in models.keys():
models[device] = DebertaV2Model.from_pretrained(LOCAL_PATH).to(device)
with torch.no_grad():
inputs = tokenizer(text, return_tensors="pt")
for i in inputs:
inputs[i] = inputs[i].to(device)
res = models[device](**inputs, output_hidden_states=True)
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
assert len(word2ph) == res.shape[0], (text, res.shape[0], len(word2ph))
word2phone = word2ph
phone_level_feature = []
for i in range(len(word2phone)):
repeat_feature = res[i].repeat(word2phone[i], 1)
phone_level_feature.append(repeat_feature)
phone_level_feature = torch.cat(phone_level_feature, dim=0)
return phone_level_feature.T

432
oldVersion/V210/text/japanese.py Normal file
View File

@@ -0,0 +1,432 @@
# Convert Japanese text to phonemes which is
# compatible with Julius https://github.com/julius-speech/segmentation-kit
import re
import unicodedata
from transformers import AutoTokenizer
from . import punctuation, symbols
from num2words import num2words
import pyopenjtalk
import jaconv
def kata2phoneme(text: str) -> str:
"""Convert katakana text to phonemes."""
text = text.strip()
if text == "ー":
return ["ー"]
elif text.startswith("ー"):
return ["ー"] + kata2phoneme(text[1:])
res = []
prev = None
while text:
if re.match(_MARKS, text):
res.append(text)
text = text[1:]
continue
if text.startswith("ー"):
if prev:
res.append(prev[-1])
text = text[1:]
continue
res += pyopenjtalk.g2p(text).lower().replace("cl", "q").split(" ")
break
# res = _COLON_RX.sub(":", res)
return res
def hira2kata(text: str) -> str:
return jaconv.hira2kata(text)
_SYMBOL_TOKENS = set(list("・、。？！"))
_NO_YOMI_TOKENS = set(list("「」『』―（）［］[]"))
_MARKS = re.compile(
r"[^A-Za-z\d\u3005\u3040-\u30ff\u4e00-\u9fff\uff11-\uff19\uff21-\uff3a\uff41-\uff5a\uff66-\uff9d]"
)
def text2kata(text: str) -> str:
parsed = pyopenjtalk.run_frontend(text)
res = []
for parts in parsed:
word, yomi = replace_punctuation(parts["string"]), parts["pron"].replace(
"’", ""
)
if yomi:
if re.match(_MARKS, yomi):
if len(word) > 1:
word = [replace_punctuation(i) for i in list(word)]
yomi = word
res += yomi
sep += word
continue
elif word not in rep_map.keys() and word not in rep_map.values():
word = ","
yomi = word
res.append(yomi)
else:
if word in _SYMBOL_TOKENS:
res.append(word)
elif word in ("っ", "ッ"):
res.append("ッ")
elif word in _NO_YOMI_TOKENS:
pass
else:
res.append(word)
return hira2kata("".join(res))
def text2sep_kata(text: str) -> (list, list):
parsed = pyopenjtalk.run_frontend(text)
res = []
sep = []
for parts in parsed:
word, yomi = replace_punctuation(parts["string"]), parts["pron"].replace(
"’", ""
)
if yomi:
if re.match(_MARKS, yomi):
if len(word) > 1:
word = [replace_punctuation(i) for i in list(word)]
yomi = word
res += yomi
sep += word
continue
elif word not in rep_map.keys() and word not in rep_map.values():
word = ","
yomi = word
res.append(yomi)
else:
if word in _SYMBOL_TOKENS:
res.append(word)
elif word in ("っ", "ッ"):
res.append("ッ")
elif word in _NO_YOMI_TOKENS:
pass
else:
res.append(word)
sep.append(word)
return sep, [hira2kata(i) for i in res], get_accent(parsed)
def get_accent(parsed):
labels = pyopenjtalk.make_label(parsed)
phonemes = []
accents = []
for n, label in enumerate(labels):
phoneme = re.search(r"\-([^\+]*)\+", label).group(1)
if phoneme not in ["sil", "pau"]:
phonemes.append(phoneme.replace("cl", "q").lower())
else:
continue
a1 = int(re.search(r"/A:(\-?[0-9]+)\+", label).group(1))
a2 = int(re.search(r"\+(\d+)\+", label).group(1))
if re.search(r"\-([^\+]*)\+", labels[n + 1]).group(1) in ["sil", "pau"]:
a2_next = -1
else:
a2_next = int(re.search(r"\+(\d+)\+", labels[n + 1]).group(1))
# Falling
if a1 == 0 and a2_next == a2 + 1:
accents.append(-1)
# Rising
elif a2 == 1 and a2_next == 2:
accents.append(1)
else:
accents.append(0)
return list(zip(phonemes, accents))
_ALPHASYMBOL_YOMI = {
"#": "シャープ",
"%": "パーセント",
"&": "アンド",
"+": "プラス",
"-": "マイナス",
":": "コロン",
";": "セミコロン",
"<": "小なり",
"=": "イコール",
">": "大なり",
"@": "アット",
"a": "エー",
"b": "ビー",
"c": "シー",
"d": "ディー",
"e": "イー",
"f": "エフ",
"g": "ジー",
"h": "エイチ",
"i": "アイ",
"j": "ジェー",
"k": "ケー",
"l": "エル",
"m": "エム",
"n": "エヌ",
"o": "オー",
"p": "ピー",
"q": "キュー",
"r": "アール",
"s": "エス",
"t": "ティー",
"u": "ユー",
"v": "ブイ",
"w": "ダブリュー",
"x": "エックス",
"y": "ワイ",
"z": "ゼット",
"α": "アルファ",
"β": "ベータ",
"γ": "ガンマ",
"δ": "デルタ",
"ε": "イプシロン",
"ζ": "ゼータ",
"η": "イータ",
"θ": "シータ",
"ι": "イオタ",
"κ": "カッパ",
"λ": "ラムダ",
"μ": "ミュー",
"ν": "ニュー",
"ξ": "クサイ",
"ο": "オミクロン",
"π": "パイ",
"ρ": "ロー",
"σ": "シグマ",
"τ": "タウ",
"υ": "ウプシロン",
"φ": "ファイ",
"χ": "カイ",
"ψ": "プサイ",
"ω": "オメガ",
}
_NUMBER_WITH_SEPARATOR_RX = re.compile("[0-9]{1,3}(,[0-9]{3})+")
_CURRENCY_MAP = {"$": "ドル", "¥": "円", "£": "ポンド", "€": "ユーロ"}
_CURRENCY_RX = re.compile(r"([$¥£€])([0-9.]*[0-9])")
_NUMBER_RX = re.compile(r"[0-9]+(\.[0-9]+)?")
def japanese_convert_numbers_to_words(text: str) -> str:
res = _NUMBER_WITH_SEPARATOR_RX.sub(lambda m: m[0].replace(",", ""), text)
res = _CURRENCY_RX.sub(lambda m: m[2] + _CURRENCY_MAP.get(m[1], m[1]), res)
res = _NUMBER_RX.sub(lambda m: num2words(m[0], lang="ja"), res)
return res
def japanese_convert_alpha_symbols_to_words(text: str) -> str:
return "".join([_ALPHASYMBOL_YOMI.get(ch, ch) for ch in text.lower()])
def japanese_text_to_phonemes(text: str) -> str:
"""Convert Japanese text to phonemes."""
res = unicodedata.normalize("NFKC", text)
res = japanese_convert_numbers_to_words(res)
# res = japanese_convert_alpha_symbols_to_words(res)
res = text2kata(res)
res = kata2phoneme(res)
return res
def is_japanese_character(char):
# 定义日语文字系统的 Unicode 范围
japanese_ranges = [
(0x3040, 0x309F), # 平假名
(0x30A0, 0x30FF), # 片假名
(0x4E00, 0x9FFF), # 汉字 (CJK Unified Ideographs)
(0x3400, 0x4DBF), # 汉字扩展 A
(0x20000, 0x2A6DF), # 汉字扩展 B
# 可以根据需要添加其他汉字扩展范围
]
# 将字符的 Unicode 编码转换为整数
char_code = ord(char)
# 检查字符是否在任何一个日语范围内
for start, end in japanese_ranges:
if start <= char_code <= end:
return True
return False
rep_map = {
"：": ",",
"；": ",",
"，": ",",
"。": ".",
"！": "!",
"？": "?",
"\n": ".",
"．": ".",
"…": "...",
"···": "...",
"・・・": "...",
"·": ",",
"・": ",",
"、": ",",
"$": ".",
"“": "'",
"”": "'",
'"': "'",
"‘": "'",
"’": "'",
"（": "'",
"）": "'",
"(": "'",
")": "'",
"《": "'",
"》": "'",
"【": "'",
"】": "'",
"[": "'",
"]": "'",
"—": "-",
"−": "-",
"～": "-",
"~": "-",
"「": "'",
"」": "'",
}
def replace_punctuation(text):
pattern = re.compile("|".join(re.escape(p) for p in rep_map.keys()))
replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
replaced_text = re.sub(
r"[^\u3040-\u309F\u30A0-\u30FF\u4E00-\u9FFF\u3400-\u4DBF\u3005"
+ "".join(punctuation)
+ r"]+",
"",
replaced_text,
)
return replaced_text
def text_normalize(text):
res = unicodedata.normalize("NFKC", text)
res = japanese_convert_numbers_to_words(res)
# res = "".join([i for i in res if is_japanese_character(i)])
res = replace_punctuation(res)
res = res.replace("゙", "")
return res
def distribute_phone(n_phone, n_word):
phones_per_word = [0] * n_word
for task in range(n_phone):
min_tasks = min(phones_per_word)
min_index = phones_per_word.index(min_tasks)
phones_per_word[min_index] += 1
return phones_per_word
def handle_long(sep_phonemes):
for i in range(len(sep_phonemes)):
if sep_phonemes[i][0] == "ー":
sep_phonemes[i][0] = sep_phonemes[i - 1][-1]
if "ー" in sep_phonemes[i]:
for j in range(len(sep_phonemes[i])):
if sep_phonemes[i][j] == "ー":
sep_phonemes[i][j] = sep_phonemes[i][j - 1][-1]
return sep_phonemes
tokenizer = AutoTokenizer.from_pretrained("./bert/deberta-v2-large-japanese-char-wwm")
def align_tones(phones, tones):
res = []
for pho in phones:
temp = [0] * len(pho)
for idx, p in enumerate(pho):
if len(tones) == 0:
break
if p == tones[0][0]:
temp[idx] = tones[0][1]
if idx > 0:
temp[idx] += temp[idx - 1]
tones.pop(0)
temp = [0] + temp
temp = temp[:-1]
if -1 in temp:
temp = [i + 1 for i in temp]
res.append(temp)
res = [i for j in res for i in j]
assert not any([i < 0 for i in res]) and not any([i > 1 for i in res])
return res
def rearrange_tones(tones, phones):
res = [0] * len(tones)
for i in range(len(tones)):
if i == 0:
if tones[i] not in punctuation:
res[i] = 1
elif tones[i] == prev:
if phones[i] in punctuation:
res[i] = 0
else:
res[i] = 1
elif tones[i] > prev:
res[i] = 2
elif tones[i] < prev:
res[i - 1] = 3
res[i] = 1
prev = tones[i]
return res
def g2p(norm_text):
sep_text, sep_kata, acc = text2sep_kata(norm_text)
sep_tokenized = []
for i in sep_text:
if i not in punctuation:
sep_tokenized.append(tokenizer.tokenize(i))
else:
sep_tokenized.append([i])
sep_phonemes = handle_long([kata2phoneme(i) for i in sep_kata])
# 异常处理，MeCab不认识的词的话会一路传到这里来，然后炸掉。目前来看只有那些超级稀有的生僻词会出现这种情况
for i in sep_phonemes:
for j in i:
assert j in symbols, (sep_text, sep_kata, sep_phonemes)
tones = align_tones(sep_phonemes, acc)
word2ph = []
for token, phoneme in zip(sep_tokenized, sep_phonemes):
phone_len = len(phoneme)
word_len = len(token)
aaa = distribute_phone(phone_len, word_len)
word2ph += aaa
phones = ["_"] + [j for i in sep_phonemes for j in i] + ["_"]
# tones = [0] + rearrange_tones(tones, phones[1:-1]) + [0]
tones = [0] + tones + [0]
word2ph = [1] + word2ph + [1]
assert len(phones) == len(tones)
return phones, tones, word2ph
if __name__ == "__main__":
tokenizer = AutoTokenizer.from_pretrained("./bert/deberta-v2-large-japanese")
text = "hello,こんにちは、世界ー！……"
from text.japanese_bert import get_bert_feature
text = text_normalize(text)
print(text)
phones, tones, word2ph = g2p(text)
bert = get_bert_feature(text, word2ph)
print(phones, tones, word2ph, bert.shape)

44
oldVersion/V210/text/japanese_bert.py Normal file
View File

@@ -0,0 +1,44 @@
import sys
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer
from config import config
from .japanese import text2sep_kata
LOCAL_PATH = "./bert/deberta-v2-large-japanese-char-wwm"
tokenizer = AutoTokenizer.from_pretrained(LOCAL_PATH)
models = dict()
def get_bert_feature(text, word2ph, device=config.bert_gen_config.device):
text = "".join(text2sep_kata(text)[0])
if (
sys.platform == "darwin"
and torch.backends.mps.is_available()
and device == "cpu"
):
device = "mps"
if not device:
device = "cuda"
if device not in models.keys():
models[device] = AutoModelForMaskedLM.from_pretrained(LOCAL_PATH).to(device)
with torch.no_grad():
inputs = tokenizer(text, return_tensors="pt")
for i in inputs:
inputs[i] = inputs[i].to(device)
res = models[device](**inputs, output_hidden_states=True)
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
assert len(word2ph) == len(text) + 2
word2phone = word2ph
phone_level_feature = []
for i in range(len(word2phone)):
repeat_feature = res[i].repeat(word2phone[i], 1)
phone_level_feature.append(repeat_feature)
phone_level_feature = torch.cat(phone_level_feature, dim=0)
return phone_level_feature.T

429
oldVersion/V210/text/opencpop-strict.txt Normal file
View File

@@ -0,0 +1,429 @@
a AA a
ai AA ai
an AA an
ang AA ang
ao AA ao
ba b a
bai b ai
ban b an
bang b ang
bao b ao
bei b ei
ben b en
beng b eng
bi b i
bian b ian
biao b iao
bie b ie
bin b in
bing b ing
bo b o
bu b u
ca c a
cai c ai
can c an
cang c ang
cao c ao
ce c e
cei c ei
cen c en
ceng c eng
cha ch a
chai ch ai
chan ch an
chang ch ang
chao ch ao
che ch e
chen ch en
cheng ch eng
chi ch ir
chong ch ong
chou ch ou
chu ch u
chua ch ua
chuai ch uai
chuan ch uan
chuang ch uang
chui ch ui
chun ch un
chuo ch uo
ci c i0
cong c ong
cou c ou
cu c u
cuan c uan
cui c ui
cun c un
cuo c uo
da d a
dai d ai
dan d an
dang d ang
dao d ao
de d e
dei d ei
den d en
deng d eng
di d i
dia d ia
dian d ian
diao d iao
die d ie
ding d ing
diu d iu
dong d ong
dou d ou
du d u
duan d uan
dui d ui
dun d un
duo d uo
e EE e
ei EE ei
en EE en
eng EE eng
er EE er
fa f a
fan f an
fang f ang
fei f ei
fen f en
feng f eng
fo f o
fou f ou
fu f u
ga g a
gai g ai
gan g an
gang g ang
gao g ao
ge g e
gei g ei
gen g en
geng g eng
gong g ong
gou g ou
gu g u
gua g ua
guai g uai
guan g uan
guang g uang
gui g ui
gun g un
guo g uo
ha h a
hai h ai
han h an
hang h ang
hao h ao
he h e
hei h ei
hen h en
heng h eng
hong h ong
hou h ou
hu h u
hua h ua
huai h uai
huan h uan
huang h uang
hui h ui
hun h un
huo h uo
ji j i
jia j ia
jian j ian
jiang j iang
jiao j iao
jie j ie
jin j in
jing j ing
jiong j iong
jiu j iu
ju j v
jv j v
juan j van
jvan j van
jue j ve
jve j ve
jun j vn
jvn j vn
ka k a
kai k ai
kan k an
kang k ang
kao k ao
ke k e
kei k ei
ken k en
keng k eng
kong k ong
kou k ou
ku k u
kua k ua
kuai k uai
kuan k uan
kuang k uang
kui k ui
kun k un
kuo k uo
la l a
lai l ai
lan l an
lang l ang
lao l ao
le l e
lei l ei
leng l eng
li l i
lia l ia
lian l ian
liang l iang
liao l iao
lie l ie
lin l in
ling l ing
liu l iu
lo l o
long l ong
lou l ou
lu l u
luan l uan
lun l un
luo l uo
lv l v
lve l ve
ma m a
mai m ai
man m an
mang m ang
mao m ao
me m e
mei m ei
men m en
meng m eng
mi m i
mian m ian
miao m iao
mie m ie
min m in
ming m ing
miu m iu
mo m o
mou m ou
mu m u
na n a
nai n ai
nan n an
nang n ang
nao n ao
ne n e
nei n ei
nen n en
neng n eng
ni n i
nian n ian
niang n iang
niao n iao
nie n ie
nin n in
ning n ing
niu n iu
nong n ong
nou n ou
nu n u
nuan n uan
nun n un
nuo n uo
nv n v
nve n ve
o OO o
ou OO ou
pa p a
pai p ai
pan p an
pang p ang
pao p ao
pei p ei
pen p en
peng p eng
pi p i
pian p ian
piao p iao
pie p ie
pin p in
ping p ing
po p o
pou p ou
pu p u
qi q i
qia q ia
qian q ian
qiang q iang
qiao q iao
qie q ie
qin q in
qing q ing
qiong q iong
qiu q iu
qu q v
qv q v
quan q van
qvan q van
que q ve
qve q ve
qun q vn
qvn q vn
ran r an
rang r ang
rao r ao
re r e
ren r en
reng r eng
ri r ir
rong r ong
rou r ou
ru r u
rua r ua
ruan r uan
rui r ui
run r un
ruo r uo
sa s a
sai s ai
san s an
sang s ang
sao s ao
se s e
sen s en
seng s eng
sha sh a
shai sh ai
shan sh an
shang sh ang
shao sh ao
she sh e
shei sh ei
shen sh en
sheng sh eng
shi sh ir
shou sh ou
shu sh u
shua sh ua
shuai sh uai
shuan sh uan
shuang sh uang
shui sh ui
shun sh un
shuo sh uo
si s i0
song s ong
sou s ou
su s u
suan s uan
sui s ui
sun s un
suo s uo
ta t a
tai t ai
tan t an
tang t ang
tao t ao
te t e
tei t ei
teng t eng
ti t i
tian t ian
tiao t iao
tie t ie
ting t ing
tong t ong
tou t ou
tu t u
tuan t uan
tui t ui
tun t un
tuo t uo
wa w a
wai w ai
wan w an
wang w ang
wei w ei
wen w en
weng w eng
wo w o
wu w u
xi x i
xia x ia
xian x ian
xiang x iang
xiao x iao
xie x ie
xin x in
xing x ing
xiong x iong
xiu x iu
xu x v
xv x v
xuan x van
xvan x van
xue x ve
xve x ve
xun x vn
xvn x vn
ya y a
yan y En
yang y ang
yao y ao
ye y E
yi y i
yin y in
ying y ing
yo y o
yong y ong
you y ou
yu y v
yv y v
yuan y van
yvan y van
yue y ve
yve y ve
yun y vn
yvn y vn
za z a
zai z ai
zan z an
zang z ang
zao z ao
ze z e
zei z ei
zen z en
zeng z eng
zha zh a
zhai zh ai
zhan zh an
zhang zh ang
zhao zh ao
zhe zh e
zhei zh ei
zhen zh en
zheng zh eng
zhi zh ir
zhong zh ong
zhou zh ou
zhu zh u
zhua zh ua
zhuai zh uai
zhuan zh uan
zhuang zh uang
zhui zh ui
zhun zh un
zhuo zh uo
zi z i0
zong z ong
zou z ou
zu z u
zuan z uan
zui z ui
zun z un
zuo z uo

187
oldVersion/V210/text/symbols.py Normal file
View File

@@ -0,0 +1,187 @@
punctuation = ["!", "?", "…", ",", ".", "'", "-"]
pu_symbols = punctuation + ["SP", "UNK"]
pad = "_"
# chinese
zh_symbols = [
"E",
"En",
"a",
"ai",
"an",
"ang",
"ao",
"b",
"c",
"ch",
"d",
"e",
"ei",
"en",
"eng",
"er",
"f",
"g",
"h",
"i",
"i0",
"ia",
"ian",
"iang",
"iao",
"ie",
"in",
"ing",
"iong",
"ir",
"iu",
"j",
"k",
"l",
"m",
"n",
"o",
"ong",
"ou",
"p",
"q",
"r",
"s",
"sh",
"t",
"u",
"ua",
"uai",
"uan",
"uang",
"ui",
"un",
"uo",
"v",
"van",
"ve",
"vn",
"w",
"x",
"y",
"z",
"zh",
"AA",
"EE",
"OO",
]
num_zh_tones = 6
# japanese
ja_symbols = [
"N",
"a",
"a:",
"b",
"by",
"ch",
"d",
"dy",
"e",
"e:",
"f",
"g",
"gy",
"h",
"hy",
"i",
"i:",
"j",
"k",
"ky",
"m",
"my",
"n",
"ny",
"o",
"o:",
"p",
"py",
"q",
"r",
"ry",
"s",
"sh",
"t",
"ts",
"ty",
"u",
"u:",
"w",
"y",
"z",
"zy",
]
num_ja_tones = 2
# English
en_symbols = [
"aa",
"ae",
"ah",
"ao",
"aw",
"ay",
"b",
"ch",
"d",
"dh",
"eh",
"er",
"ey",
"f",
"g",
"hh",
"ih",
"iy",
"jh",
"k",
"l",
"m",
"n",
"ng",
"ow",
"oy",
"p",
"r",
"s",
"sh",
"t",
"th",
"uh",
"uw",
"V",
"w",
"y",
"z",
"zh",
]
num_en_tones = 4
# combine all symbols
normal_symbols = sorted(set(zh_symbols + ja_symbols + en_symbols))
symbols = [pad] + normal_symbols + pu_symbols
sil_phonemes_ids = [symbols.index(i) for i in pu_symbols]
# combine all tones
num_tones = num_zh_tones + num_ja_tones + num_en_tones
# language maps
language_id_map = {"ZH": 0, "JP": 1, "EN": 2}
num_languages = len(language_id_map.keys())
language_tone_start_map = {
"ZH": 0,
"JP": num_zh_tones,
"EN": num_zh_tones + num_ja_tones,
}
if __name__ == "__main__":
a = set(zh_symbols)
b = set(en_symbols)
print(sorted(a & b))

769
oldVersion/V210/text/tone_sandhi.py Normal file
View File

@@ -0,0 +1,769 @@
# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from typing import List
from typing import Tuple
import jieba
from pypinyin import lazy_pinyin
from pypinyin import Style
class ToneSandhi:
def __init__(self):
self.must_neural_tone_words = {
"麻烦",
"麻利",
"鸳鸯",
"高粱",
"骨头",
"骆驼",
"马虎",
"首饰",
"馒头",
"馄饨",
"风筝",
"难为",
"队伍",
"阔气",
"闺女",
"门道",
"锄头",
"铺盖",
"铃铛",
"铁匠",
"钥匙",
"里脊",
"里头",
"部分",
"那么",
"道士",
"造化",
"迷糊",
"连累",
"这么",
"这个",
"运气",
"过去",
"软和",
"转悠",
"踏实",
"跳蚤",
"跟头",
"趔趄",
"财主",
"豆腐",
"讲究",
"记性",
"记号",
"认识",
"规矩",
"见识",
"裁缝",
"补丁",
"衣裳",
"衣服",
"衙门",
"街坊",
"行李",
"行当",
"蛤蟆",
"蘑菇",
"薄荷",
"葫芦",
"葡萄",
"萝卜",
"荸荠",
"苗条",
"苗头",
"苍蝇",
"芝麻",
"舒服",
"舒坦",
"舌头",
"自在",
"膏药",
"脾气",
"脑袋",
"脊梁",
"能耐",
"胳膊",
"胭脂",
"胡萝",
"胡琴",
"胡同",
"聪明",
"耽误",
"耽搁",
"耷拉",
"耳朵",
"老爷",
"老实",
"老婆",
"老头",
"老太",
"翻腾",
"罗嗦",
"罐头",
"编辑",
"结实",
"红火",
"累赘",
"糨糊",
"糊涂",
"精神",
"粮食",
"簸箕",
"篱笆",
"算计",
"算盘",
"答应",
"笤帚",
"笑语",
"笑话",
"窟窿",
"窝囊",
"窗户",
"稳当",
"稀罕",
"称呼",
"秧歌",
"秀气",
"秀才",
"福气",
"祖宗",
"砚台",
"码头",
"石榴",
"石头",
"石匠",
"知识",
"眼睛",
"眯缝",
"眨巴",
"眉毛",
"相声",
"盘算",
"白净",
"痢疾",
"痛快",
"疟疾",
"疙瘩",
"疏忽",
"畜生",
"生意",
"甘蔗",
"琵琶",
"琢磨",
"琉璃",
"玻璃",
"玫瑰",
"玄乎",
"狐狸",
"状元",
"特务",
"牲口",
"牙碜",
"牌楼",
"爽快",
"爱人",
"热闹",
"烧饼",
"烟筒",
"烂糊",
"点心",
"炊帚",
"灯笼",
"火候",
"漂亮",
"滑溜",
"溜达",
"温和",
"清楚",
"消息",
"浪头",
"活泼",
"比方",
"正经",
"欺负",
"模糊",
"槟榔",
"棺材",
"棒槌",
"棉花",
"核桃",
"栅栏",
"柴火",
"架势",
"枕头",
"枇杷",
"机灵",
"本事",
"木头",
"木匠",
"朋友",
"月饼",
"月亮",
"暖和",
"明白",
"时候",
"新鲜",
"故事",
"收拾",
"收成",
"提防",
"挖苦",
"挑剔",
"指甲",
"指头",
"拾掇",
"拳头",
"拨弄",
"招牌",
"招呼",
"抬举",
"护士",
"折腾",
"扫帚",
"打量",
"打算",
"打点",
"打扮",
"打听",
"打发",
"扎实",
"扁担",
"戒指",
"懒得",
"意识",
"意思",
"情形",
"悟性",
"怪物",
"思量",
"怎么",
"念头",
"念叨",
"快活",
"忙活",
"志气",
"心思",
"得罪",
"张罗",
"弟兄",
"开通",
"应酬",
"庄稼",
"干事",
"帮手",
"帐篷",
"希罕",
"师父",
"师傅",
"巴结",
"巴掌",
"差事",
"工夫",
"岁数",
"屁股",
"尾巴",
"少爷",
"小气",
"小伙",
"将就",
"对头",
"对付",
"寡妇",
"家伙",
"客气",
"实在",
"官司",
"学问",
"学生",
"字号",
"嫁妆",
"媳妇",
"媒人",
"婆家",
"娘家",
"委屈",
"姑娘",
"姐夫",
"妯娌",
"妥当",
"妖精",
"奴才",
"女婿",
"头发",
"太阳",
"大爷",
"大方",
"大意",
"大夫",
"多少",
"多么",
"外甥",
"壮实",
"地道",
"地方",
"在乎",
"困难",
"嘴巴",
"嘱咐",
"嘟囔",
"嘀咕",
"喜欢",
"喇嘛",
"喇叭",
"商量",
"唾沫",
"哑巴",
"哈欠",
"哆嗦",
"咳嗽",
"和尚",
"告诉",
"告示",
"含糊",
"吓唬",
"后头",
"名字",
"名堂",
"合同",
"吆喝",
"叫唤",
"口袋",
"厚道",
"厉害",
"千斤",
"包袱",
"包涵",
"匀称",
"勤快",
"动静",
"动弹",
"功夫",
"力气",
"前头",
"刺猬",
"刺激",
"别扭",
"利落",
"利索",
"利害",
"分析",
"出息",
"凑合",
"凉快",
"冷战",
"冤枉",
"冒失",
"养活",
"关系",
"先生",
"兄弟",
"便宜",
"使唤",
"佩服",
"作坊",
"体面",
"位置",
"似的",
"伙计",
"休息",
"什么",
"人家",
"亲戚",
"亲家",
"交情",
"云彩",
"事情",
"买卖",
"主意",
"丫头",
"丧气",
"两口",
"东西",
"东家",
"世故",
"不由",
"不在",
"下水",
"下巴",
"上头",
"上司",
"丈夫",
"丈人",
"一辈",
"那个",
"菩萨",
"父亲",
"母亲",
"咕噜",
"邋遢",
"费用",
"冤家",
"甜头",
"介绍",
"荒唐",
"大人",
"泥鳅",
"幸福",
"熟悉",
"计划",
"扑腾",
"蜡烛",
"姥爷",
"照顾",
"喉咙",
"吉他",
"弄堂",
"蚂蚱",
"凤凰",
"拖沓",
"寒碜",
"糟蹋",
"倒腾",
"报复",
"逻辑",
"盘缠",
"喽啰",
"牢骚",
"咖喱",
"扫把",
"惦记",
}
self.must_not_neural_tone_words = {
"男子",
"女子",
"分子",
"原子",
"量子",
"莲子",
"石子",
"瓜子",
"电子",
"人人",
"虎虎",
}
self.punc = "：，；。？！“”‘’':,;.?!"
# the meaning of jieba pos tag: https://blog.csdn.net/weixin_44174352/article/details/113731041
# e.g.
# word: "家里"
# pos: "s"
# finals: ['ia1', 'i3']
def _neural_sandhi(self, word: str, pos: str, finals: List[str]) -> List[str]:
# reduplication words for n. and v. e.g. 奶奶, 试试, 旺旺
for j, item in enumerate(word):
if (
j - 1 >= 0
and item == word[j - 1]
and pos[0] in {"n", "v", "a"}
and word not in self.must_not_neural_tone_words
):
finals[j] = finals[j][:-1] + "5"
ge_idx = word.find("个")
if len(word) >= 1 and word[-1] in "吧呢啊呐噻嘛吖嗨呐哦哒额滴哩哟喽啰耶喔诶":
finals[-1] = finals[-1][:-1] + "5"
elif len(word) >= 1 and word[-1] in "的地得":
finals[-1] = finals[-1][:-1] + "5"
# e.g. 走了, 看着, 去过
# elif len(word) == 1 and word in "了着过" and pos in {"ul", "uz", "ug"}:
# finals[-1] = finals[-1][:-1] + "5"
elif (
len(word) > 1
and word[-1] in "们子"
and pos in {"r", "n"}
and word not in self.must_not_neural_tone_words
):
finals[-1] = finals[-1][:-1] + "5"
# e.g. 桌上, 地下, 家里
elif len(word) > 1 and word[-1] in "上下里" and pos in {"s", "l", "f"}:
finals[-1] = finals[-1][:-1] + "5"
# e.g. 上来, 下去
elif len(word) > 1 and word[-1] in "来去" and word[-2] in "上下进出回过起开":
finals[-1] = finals[-1][:-1] + "5"
# 个做量词
elif (
ge_idx >= 1
and (word[ge_idx - 1].isnumeric() or word[ge_idx - 1] in "几有两半多各整每做是")
) or word == "个":
finals[ge_idx] = finals[ge_idx][:-1] + "5"
else:
if (
word in self.must_neural_tone_words
or word[-2:] in self.must_neural_tone_words
):
finals[-1] = finals[-1][:-1] + "5"
word_list = self._split_word(word)
finals_list = [finals[: len(word_list[0])], finals[len(word_list[0]) :]]
for i, word in enumerate(word_list):
# conventional neural in Chinese
if (
word in self.must_neural_tone_words
or word[-2:] in self.must_neural_tone_words
):
finals_list[i][-1] = finals_list[i][-1][:-1] + "5"
finals = sum(finals_list, [])
return finals
def _bu_sandhi(self, word: str, finals: List[str]) -> List[str]:
# e.g. 看不懂
if len(word) == 3 and word[1] == "不":
finals[1] = finals[1][:-1] + "5"
else:
for i, char in enumerate(word):
# "不" before tone4 should be bu2, e.g. 不怕
if char == "不" and i + 1 < len(word) and finals[i + 1][-1] == "4":
finals[i] = finals[i][:-1] + "2"
return finals
def _yi_sandhi(self, word: str, finals: List[str]) -> List[str]:
# "一" in number sequences, e.g. 一零零, 二一零
if word.find("一") != -1 and all(
[item.isnumeric() for item in word if item != "一"]
):
return finals
# "一" between reduplication words should be yi5, e.g. 看一看
elif len(word) == 3 and word[1] == "一" and word[0] == word[-1]:
finals[1] = finals[1][:-1] + "5"
# when "一" is ordinal word, it should be yi1
elif word.startswith("第一"):
finals[1] = finals[1][:-1] + "1"
else:
for i, char in enumerate(word):
if char == "一" and i + 1 < len(word):
# "一" before tone4 should be yi2, e.g. 一段
if finals[i + 1][-1] == "4":
finals[i] = finals[i][:-1] + "2"
# "一" before non-tone4 should be yi4, e.g. 一天
else:
# "一" 后面如果是标点，还读一声
if word[i + 1] not in self.punc:
finals[i] = finals[i][:-1] + "4"
return finals
def _split_word(self, word: str) -> List[str]:
word_list = jieba.cut_for_search(word)
word_list = sorted(word_list, key=lambda i: len(i), reverse=False)
first_subword = word_list[0]
first_begin_idx = word.find(first_subword)
if first_begin_idx == 0:
second_subword = word[len(first_subword) :]
new_word_list = [first_subword, second_subword]
else:
second_subword = word[: -len(first_subword)]
new_word_list = [second_subword, first_subword]
return new_word_list
def _three_sandhi(self, word: str, finals: List[str]) -> List[str]:
if len(word) == 2 and self._all_tone_three(finals):
finals[0] = finals[0][:-1] + "2"
elif len(word) == 3:
word_list = self._split_word(word)
if self._all_tone_three(finals):
# disyllabic + monosyllabic, e.g. 蒙古/包
if len(word_list[0]) == 2:
finals[0] = finals[0][:-1] + "2"
finals[1] = finals[1][:-1] + "2"
# monosyllabic + disyllabic, e.g. 纸/老虎
elif len(word_list[0]) == 1:
finals[1] = finals[1][:-1] + "2"
else:
finals_list = [finals[: len(word_list[0])], finals[len(word_list[0]) :]]
if len(finals_list) == 2:
for i, sub in enumerate(finals_list):
# e.g. 所有/人
if self._all_tone_three(sub) and len(sub) == 2:
finals_list[i][0] = finals_list[i][0][:-1] + "2"
# e.g. 好/喜欢
elif (
i == 1
and not self._all_tone_three(sub)
and finals_list[i][0][-1] == "3"
and finals_list[0][-1][-1] == "3"
):
finals_list[0][-1] = finals_list[0][-1][:-1] + "2"
finals = sum(finals_list, [])
# split idiom into two words who's length is 2
elif len(word) == 4:
finals_list = [finals[:2], finals[2:]]
finals = []
for sub in finals_list:
if self._all_tone_three(sub):
sub[0] = sub[0][:-1] + "2"
finals += sub
return finals
def _all_tone_three(self, finals: List[str]) -> bool:
return all(x[-1] == "3" for x in finals)
# merge "不" and the word behind it
# if don't merge, "不" sometimes appears alone according to jieba, which may occur sandhi error
def _merge_bu(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
new_seg = []
last_word = ""
for word, pos in seg:
if last_word == "不":
word = last_word + word
if word != "不":
new_seg.append((word, pos))
last_word = word[:]
if last_word == "不":
new_seg.append((last_word, "d"))
last_word = ""
return new_seg
# function 1: merge "一" and reduplication words in it's left and right, e.g. "听","一","听" ->"听一听"
# function 2: merge single "一" and the word behind it
# if don't merge, "一" sometimes appears alone according to jieba, which may occur sandhi error
# e.g.
# input seg: [('听', 'v'), ('一', 'm'), ('听', 'v')]
# output seg: [['听一听', 'v']]
def _merge_yi(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
new_seg = []
# function 1
for i, (word, pos) in enumerate(seg):
if (
i - 1 >= 0
and word == "一"
and i + 1 < len(seg)
and seg[i - 1][0] == seg[i + 1][0]
and seg[i - 1][1] == "v"
):
new_seg[i - 1][0] = new_seg[i - 1][0] + "一" + new_seg[i - 1][0]
else:
if (
i - 2 >= 0
and seg[i - 1][0] == "一"
and seg[i - 2][0] == word
and pos == "v"
):
continue
else:
new_seg.append([word, pos])
seg = new_seg
new_seg = []
# function 2
for i, (word, pos) in enumerate(seg):
if new_seg and new_seg[-1][0] == "一":
new_seg[-1][0] = new_seg[-1][0] + word
else:
new_seg.append([word, pos])
return new_seg
# the first and the second words are all_tone_three
def _merge_continuous_three_tones(
self, seg: List[Tuple[str, str]]
) -> List[Tuple[str, str]]:
new_seg = []
sub_finals_list = [
lazy_pinyin(word, neutral_tone_with_five=True, style=Style.FINALS_TONE3)
for (word, pos) in seg
]
assert len(sub_finals_list) == len(seg)
merge_last = [False] * len(seg)
for i, (word, pos) in enumerate(seg):
if (
i - 1 >= 0
and self._all_tone_three(sub_finals_list[i - 1])
and self._all_tone_three(sub_finals_list[i])
and not merge_last[i - 1]
):
# if the last word is reduplication, not merge, because reduplication need to be _neural_sandhi
if (
not self._is_reduplication(seg[i - 1][0])
and len(seg[i - 1][0]) + len(seg[i][0]) <= 3
):
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
merge_last[i] = True
else:
new_seg.append([word, pos])
else:
new_seg.append([word, pos])
return new_seg
def _is_reduplication(self, word: str) -> bool:
return len(word) == 2 and word[0] == word[1]
# the last char of first word and the first char of second word is tone_three
def _merge_continuous_three_tones_2(
self, seg: List[Tuple[str, str]]
) -> List[Tuple[str, str]]:
new_seg = []
sub_finals_list = [
lazy_pinyin(word, neutral_tone_with_five=True, style=Style.FINALS_TONE3)
for (word, pos) in seg
]
assert len(sub_finals_list) == len(seg)
merge_last = [False] * len(seg)
for i, (word, pos) in enumerate(seg):
if (
i - 1 >= 0
and sub_finals_list[i - 1][-1][-1] == "3"
and sub_finals_list[i][0][-1] == "3"
and not merge_last[i - 1]
):
# if the last word is reduplication, not merge, because reduplication need to be _neural_sandhi
if (
not self._is_reduplication(seg[i - 1][0])
and len(seg[i - 1][0]) + len(seg[i][0]) <= 3
):
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
merge_last[i] = True
else:
new_seg.append([word, pos])
else:
new_seg.append([word, pos])
return new_seg
def _merge_er(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
new_seg = []
for i, (word, pos) in enumerate(seg):
if i - 1 >= 0 and word == "儿" and seg[i - 1][0] != "#":
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
else:
new_seg.append([word, pos])
return new_seg
def _merge_reduplication(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
new_seg = []
for i, (word, pos) in enumerate(seg):
if new_seg and word == new_seg[-1][0]:
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
else:
new_seg.append([word, pos])
return new_seg
def pre_merge_for_modify(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
seg = self._merge_bu(seg)
try:
seg = self._merge_yi(seg)
except:
print("_merge_yi failed")
seg = self._merge_reduplication(seg)
seg = self._merge_continuous_three_tones(seg)
seg = self._merge_continuous_three_tones_2(seg)
seg = self._merge_er(seg)
return seg
def modified_tone(self, word: str, pos: str, finals: List[str]) -> List[str]:
finals = self._bu_sandhi(word, finals)
finals = self._yi_sandhi(word, finals)
finals = self._neural_sandhi(word, pos, finals)
finals = self._three_sandhi(word, finals)
return finals

13
preprocess_text.py
View File

@@ -27,7 +27,7 @@ preprocess_text_config = config.preprocess_text_config
default=preprocess_text_config.config_path,
type=click.Path(exists=True, file_okay=True, dir_okay=False),
)
@click.option("--val-per-spk", default=preprocess_text_config.val_per_spk)
@click.option("--val-per-lang", default=preprocess_text_config.val_per_lang)
@click.option("--max-val-total", default=preprocess_text_config.max_val_total)
@click.option("--clean/--no-clean", default=preprocess_text_config.clean)
@click.option("-y", "--yml_config")
@@ -37,7 +37,7 @@ def preprocess(
train_path: str,
val_path: str,
config_path: str,
val_per_spk: int,
val_per_lang: int,
max_val_total: int,
clean: bool,
yml_config: str, # 这个不要删
@@ -94,8 +94,7 @@ def preprocess(
countNotFound += 1
continue
audioPaths.add(utt)
spk_utt_map[spk].append(line)
spk_utt_map[language].append(line)
if spk not in spk_id_map.keys():
spk_id_map[spk] = current_sid
current_sid += 1
@@ -106,9 +105,10 @@ def preprocess(
for spk, utts in spk_utt_map.items():
shuffle(utts)
val_list += utts[:val_per_spk]
train_list += utts[val_per_spk:]
val_list += utts[:val_per_lang]
train_list += utts[val_per_lang:]
shuffle(val_list)
if len(val_list) > max_val_total:
train_list += val_list[max_val_total:]
val_list = val_list[:max_val_total]
@@ -123,6 +123,7 @@ def preprocess(
json_config = json.load(open(config_path, encoding="utf-8"))
json_config["data"]["spk2id"] = spk_id_map
json_config["data"]["n_speakers"] = len(spk_id_map)
# 新增写入：写入训练版本、数据集路径
json_config["version"] = latest_version
json_config["data"]["training_files"] = os.path.normpath(train_path).replace(

16
resample.py
View File

@@ -10,11 +10,11 @@ from config import config
def process(item):
spkdir, wav_name, args = item
wav_path = os.path.join(args.in_dir, spkdir, wav_name)
wav_name, args = item
wav_path = os.path.join(args.in_dir, wav_name)
if os.path.exists(wav_path) and wav_path.lower().endswith(".wav"):
wav, sr = librosa.load(wav_path, sr=args.sr)
soundfile.write(os.path.join(args.out_dir, spkdir, wav_name), wav, sr)
soundfile.write(os.path.join(args.out_dir, wav_name), wav, sr)
if __name__ == "__main__":
@@ -54,15 +54,11 @@ if __name__ == "__main__":
tasks = []
for dirpath, _, filenames in os.walk(args.in_dir):
# 子级目录
spk_dir = os.path.relpath(dirpath, args.in_dir)
spk_dir_out = os.path.join(args.out_dir, spk_dir)
if not os.path.isdir(spk_dir_out):
os.makedirs(spk_dir_out, exist_ok=True)
if not os.path.isdir(args.out_dir):
os.makedirs(args.out_dir, exist_ok=True)
for filename in filenames:
if filename.lower().endswith(".wav"):
twople = (spk_dir, filename, args)
tasks.append(twople)
tasks.append((filename, args))
for _ in tqdm(
pool.imap_unordered(process, tasks),

38
train_ms.py
View File

@@ -194,6 +194,21 @@ def run():
**hps.model,
).cuda(local_rank)
if getattr(hps.train, "freeze_ZH_bert", False):
print("Freezing ZH bert encoder !!!")
for param in net_g.enc_p.bert_proj.parameters():
param.requires_grad = False
if getattr(hps.train, "freeze_EN_bert", False):
print("Freezing EN bert encoder !!!")
for param in net_g.enc_p.en_bert_proj.parameters():
param.requires_grad = False
if getattr(hps.train, "freeze_JP_bert", False):
print("Freezing JP bert encoder !!!")
for param in net_g.enc_p.ja_bert_proj.parameters():
param.requires_grad = False
net_d = MultiPeriodDiscriminator(hps.model.use_spectral_norm).cuda(local_rank)
optim_g = torch.optim.AdamW(
filter(lambda p: p.requires_grad, net_g.parameters()),
@@ -216,12 +231,15 @@ def run():
)
else:
optim_dur_disc = None
net_g = DDP(net_g, device_ids=[local_rank])
net_d = DDP(net_d, device_ids=[local_rank])
net_g = DDP(net_g, device_ids=[local_rank], bucket_cap_mb=512)
net_d = DDP(net_d, device_ids=[local_rank], bucket_cap_mb=512)
dur_resume_lr = None
if net_dur_disc is not None:
net_dur_disc = DDP(
net_dur_disc, device_ids=[local_rank], find_unused_parameters=True
net_dur_disc,
device_ids=[local_rank],
find_unused_parameters=True,
bucket_cap_mb=512,
)
# 下载底模
@@ -371,7 +389,7 @@ def train_and_evaluate(
ja_bert,
en_bert,
emo,
) in tqdm(enumerate(train_loader)):
) in enumerate(tqdm(train_loader)):
if net_g.module.use_noise_scaled_mas:
current_mas_noise_scale = (
net_g.module.mas_noise_scale_initial
@@ -405,6 +423,7 @@ def train_and_evaluate(
z_mask,
(z, z_p, m_p, logs_p, m_q, logs_q),
(hidden_x, logw, logw_),
g,
loss_commit,
) = net_g(
x,
@@ -454,7 +473,11 @@ def train_and_evaluate(
loss_disc_all = loss_disc
if net_dur_disc is not None:
y_dur_hat_r, y_dur_hat_g = net_dur_disc(
hidden_x.detach(), x_mask.detach(), logw.detach(), logw_.detach()
hidden_x.detach(),
x_mask.detach(),
logw.detach(),
logw_.detach(),
g.detach(),
)
with autocast(enabled=False):
# TODO: I think need to mean using the mask, but for now, just mean all
@@ -480,7 +503,9 @@ def train_and_evaluate(
# Generator
y_d_hat_r, y_d_hat_g, fmap_r, fmap_g = net_d(y, y_hat)
if net_dur_disc is not None:
y_dur_hat_r, y_dur_hat_g = net_dur_disc(hidden_x, x_mask, logw, logw_)
y_dur_hat_r, y_dur_hat_g = net_dur_disc(
hidden_x, x_mask, logw, logw_, g
)
with autocast(enabled=False):
loss_dur = torch.sum(l_length.float())
loss_mel = F.l1_loss(y_mel, y_hat_mel) * hps.train.c_mel
@@ -591,6 +616,7 @@ def train_and_evaluate(
)
global_step += 1
gc.collect()
torch.cuda.empty_cache()
if rank == 0:

1017
webui.py
View File

File diff suppressed because it is too large Load Diff

177
webui_preprocess.py Normal file
View File

@@ -0,0 +1,177 @@
import gradio as gr
import webbrowser
import os
import json
import subprocess
import shutil
def get_path(data_dir):
start_path = os.path.join("./data", data_dir)
lbl_path = os.path.join(start_path, "esd.list")
train_path = os.path.join(start_path, "train.list")
val_path = os.path.join(start_path, "val.list")
config_path = os.path.join(start_path, "configs", "config.json")
return start_path, lbl_path, train_path, val_path, config_path
def generate_config(data_dir, batch_size):
assert data_dir != "", "数据集名称不能为空"
start_path, _, train_path, val_path, config_path = get_path(data_dir)
if os.path.isfile(config_path):
config = json.load(open(config_path))
else:
config = json.load(open("configs/config.json"))
config["data"]["training_files"] = train_path
config["data"]["validation_files"] = val_path
config["train"]["batch_size"] = batch_size
out_path = os.path.join(start_path, "configs")
if not os.path.isdir(out_path):
os.mkdir(out_path)
model_path = os.path.join(start_path, "models")
if not os.path.isdir(model_path):
os.mkdir(model_path)
with open(config_path, "w", encoding="utf-8") as f:
json.dump(config, f, indent=4)
if not os.path.exists("config.yml"):
shutil.copy(src="default_config.yml", dst="config.yml")
return "配置文件生成完成"
def resample(data_dir):
assert data_dir != "", "数据集名称不能为空"
start_path, _, _, _, config_path = get_path(data_dir)
in_dir = os.path.join(start_path, "raw")
out_dir = os.path.join(start_path, "wavs")
subprocess.run(
f"python resample.py "
f"--sr 44100 "
f"--in_dir {in_dir} "
f"--out_dir {out_dir} ",
shell=True,
)
return "音频文件预处理完成"
def preprocess_text(data_dir):
assert data_dir != "", "数据集名称不能为空"
start_path, lbl_path, train_path, val_path, config_path = get_path(data_dir)
lines = open(lbl_path, "r", encoding="utf-8").readlines()
with open(lbl_path, "w", encoding="utf-8") as f:
for line in lines:
path, spk, language, text = line.strip().split("|")
path = os.path.join(start_path, "wavs", os.path.basename(path))
f.writelines(f"{path}|{spk}|{language}|{text}\n")
subprocess.run(
f"python preprocess_text.py "
f"--transcription-path {lbl_path} "
f"--train-path {train_path} "
f"--val-path {val_path} "
f"--config-path {config_path}",
shell=True,
)
return "标签文件预处理完成"
def bert_gen(data_dir):
assert data_dir != "", "数据集名称不能为空"
_, _, _, _, config_path = get_path(data_dir)
subprocess.run(
f"python bert_gen.py " f"--config {config_path}",
shell=True,
)
return "BERT 特征文件生成完成"
def clap_gen(data_dir):
assert data_dir != "", "数据集名称不能为空"
_, _, _, _, config_path = get_path(data_dir)
subprocess.run(
f"python clap_gen.py " f"--config {config_path}",
shell=True,
)
return "CLAP 特征文件生成完成"
if __name__ == "__main__":
with gr.Blocks() as app:
with gr.Row():
with gr.Column():
_ = gr.Markdown(
value="# Bert-VITS2 数据预处理\n"
"## 预先准备：\n"
"下载 BERT 和 CLAP 模型：\n"
"- [中文 RoBERTa](https://huggingface.co/hfl/chinese-roberta-wwm-ext-large)\n"
"- [日文 DeBERTa](https://huggingface.co/ku-nlp/deberta-v2-large-japanese-char-wwm)\n"
"- [英文 DeBERTa](https://huggingface.co/microsoft/deberta-v3-large)\n"
"- [CLAP](https://huggingface.co/laion/clap-htsat-fused)\n"
"\n"
"将 BERT 模型放置到 `bert` 文件夹下，CLAP 模型放置到 `emotional` 文件夹下，覆盖同名文件夹。\n"
"\n"
"数据准备：\n"
"将数据放置在 data 文件夹下，按照如下结构组织：\n"
"\n"
"```\n"
"├── data\n"
"│ ├── {你的数据集名称}\n"
"│ │ ├── esd.list\n"
"│ │ ├── raw\n"
"│ │ │ ├── ****.wav\n"
"│ │ │ ├── ****.wav\n"
"│ │ │ ├── ...\n"
"```\n"
"\n"
"其中，`raw` 文件夹下保存所有的音频文件，`esd.list` 文件为标签文本，格式为\n"
"\n"
"```\n"
"****.wav|{说话人名}|{语言 ID}|{标签文本}\n"
"```\n"
"\n"
"例如：\n"
"```\n"
"vo_ABDLQ001_1_paimon_02.wav|派蒙|ZH|没什么没什么，只是平时他总是站在这里，有点奇怪而已。\n"
"noa_501_0001.wav|NOA|JP|そうだね、油断しないのはとても大事なことだと思う\n"
"Albedo_vo_ABDLQ002_4_albedo_01.wav|Albedo|EN|Who are you? Why did you alarm them?\n"
"...\n"
"```\n"
)
data_dir = gr.Textbox(
label="数据集名称",
placeholder="你放置在 data 文件夹下的数据集所在文件夹的名称，如 data/genshin 则填 genshin",
)
info = gr.Textbox(label="状态信息")
_ = gr.Markdown(value="## 第一步：生成配置文件")
with gr.Row():
batch_size = gr.Slider(
label="批大小（Batch size）：24 GB 显存可用 12",
value=8,
minimum=1,
maximum=64,
step=1,
)
generate_config_btn = gr.Button(value="执行", variant="primary")
_ = gr.Markdown(value="## 第二步：预处理音频文件")
resample_btn = gr.Button(value="执行", variant="primary")
_ = gr.Markdown(value="## 第三步：预处理标签文件")
preprocess_text_btn = gr.Button(value="执行", variant="primary")
_ = gr.Markdown(value="## 第四步：生成 BERT 特征文件")
bert_gen_btn = gr.Button(value="执行", variant="primary")
_ = gr.Markdown(value="## 第五步：生成 CLAP 特征文件")
clap_gen_btn = gr.Button(value="执行", variant="primary")
_ = gr.Markdown(
value="## 训练模型及部署：\n"
"修改根目录下的 `config.yml` 中 `dataset_path` 一项为 `data/{你的数据集名称}`\n"
"- 训练：将[预训练模型文件](https://openi.pcl.ac.cn/Stardust_minus/Bert-VITS2/modelmanage/show_model)（`D_0.pth`、`DUR_0.pth` 和 `G_0.pth`）放到 `data/{你的数据集名称}/models` 文件夹下，执行 `torchrun --nproc_per_node=1 train_ms.py` 命令（多卡运行可参考 `run_MnodesAndMgpus.sh` 中的命令。\n"
"- 部署：修改根目录下的 `config.yml` 中 `webui` 下 `model` 一项为 `models/{权重文件名}.pth` （如 G_10000.pth），然后执行 `python webui.py`"
)
generate_config_btn.click(
generate_config, inputs=[data_dir, batch_size], outputs=[info]
)
resample_btn.click(resample, inputs=[data_dir], outputs=[info])
preprocess_text_btn.click(preprocess_text, inputs=[data_dir], outputs=[info])
bert_gen_btn.click(bert_gen, inputs=[data_dir], outputs=[info])
clap_gen_btn.click(clap_gen, inputs=[data_dir], outputs=[info])
webbrowser.open("http://127.0.0.1:7860")
app.launch(share=False, server_port=7860)