Dev no emo (#123)
* Create emo_gen.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update server.py, fix bugs in func get_text() and infer(). (#52) * Extract get_text() and infer() from webui.py. (#53) * Extract get_text() and infer() from webui.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * add emo emb * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * init emo gen * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * init emo * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * init emo * Delete bert/bert-base-japanese-v3 directory * Create .gitkeep * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Create add_punc.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix bug in bert_gen.py (#54) * Update README.md * fix bug in models.py (#56) * 更新 models.py * Fix japanese cleaner (#61) * 初步，睡觉明天继续写（ * 好好好放错分支了，熬夜是大忌 * [pre-commit.ci] pre-commit autoupdate (#55) * [pre-commit.ci] pre-commit autoupdate updates: - [github.com/pre-commit/pre-commit-hooks: v4.4.0 → v4.5.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.4.0...v4.5.0) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Create tokenizer_config.json * update preprocess_text.py：过滤一个音频匹配多个文本的情况 (#57) * update preprocess_text.py：过滤音频不存在的情况 (#58) * 修复日语cleaner和bert * better * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Stardust·减 <star_dust_chen@foxmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sora <atri@suzakuintsubaki.com> * Apply Code Formatter Change * Add config.yml for global configuration. (#62) * Add config.yml for global configuration. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix bug in webui.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Rename config.yml to default_config.yml. Add ./config.yml to gitignore. * Add config.py to parse config.yml * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update webui.py (#65) * Update webui.py: 1. Add auto translation from Chinese to Japanese. 2. Start to use config.py in webui.py to set config instead of using the command line. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix (#68) * 加上ー * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update infer.py and webui.py. Supports loading and inference models of 1.1.1 version. (#66) * Update infer.py and webui.py. Supports loading and inference models of 1.1.1 version. * Update config.json * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix bug in translate.py (#69) * Supports loading and inference models of 1.1、1.0.1、1.0 version. (#70) * Supports loading and inference models of 1.1、1.0.1、1.0 version. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Delete useless file in OldVersion --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update japanese.py (#71) Handling JA long pronunciations * 使用配置文件配置bert_gen.py, preprocess_text.py, resample.py (#72) * Update bert_gen.py, preprocess_text.py, resample.py. Support using config.yml in these files. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update bert_gen.py * Update bert_gen.py, fix bug. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Delete bert/bert-base-japanese-v3 directory * Create config.json * Create tokenizer_config.json * Create vocab.txt * Update server.py. 支持多版本多模型 (#76) * Update server.py. 支持多版本多模型 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Dev webui (#77) * 申请pr (#75) * 2023/10/11 update 界面优化 * Update webui.py 翻译英文页面为中文 * Update train_ms.py 单卡训练 * 加入图片 * Update extern_subprocess.py * Update asr_transcript.py * Update asr_transcript.py * Update asr_transcript.py * Update extern_subprocess.py * Update asr_transcript.py * Update asr_transcript.py * Update asr_transcript.py * Update all_process.py * Update extern_subprocess.py * Update all_process.py * Update all_process.py * Update asr_transcript.py * Update extern_subprocess.py * Update webui.py * Create re_matching.py * Update webui.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update all_process.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update all_process.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update all_process.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update asr_transcript.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Pack 'update' functions into a module * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update all_process.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update asr_transcript.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update extern_subprocess.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update all_process.py * Update asr_transcript.py * Update webui.py * Add files via upload * Update extern_subprocess.py * Update all_process.py * Update asr_transcript.py * Update bert_gen.py * Update extern_subprocess.py * Update preprocess_text.py * Update re_matching.py * Update resample.py * Update update_status.py * Update update_status.py * Update webui.py * Update all_process.py * Update preprocess_text.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update train_ms.py --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Stardust·减 <star_dust_chen@foxmail.com> Co-authored-by: innnky <67028263+innnky@users.noreply.github.com> * Delete all_process.py * Delete asr_transcript.py * Delete extern_subprocess.py --------- Co-authored-by: spicysama <122108331+AnyaCoder@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: innnky <67028263+innnky@users.noreply.github.com> * Create config.json * Create preprocessor_config.json * Create vocab.json * Delete emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim/.gitkeep * Update emo_gen.py * Delete add_punc.py * add emotion_clustering.i * Apply Code Formatter Change * Update models.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update preprocess_text.py (#78) * Update preprocess_text.py. 检测重复以及不存在的音频 (#79) * Handle Janpanese long pronunciations (#80) * Handle Janpanese long pronunciations * Update japanese.py * Update japanese.py * Use unified phonemes for Japanese long vowel (#82) * Use an unified phoneme for Japanese long vowel `symbol.py` has not been updated to ensure compatibility with older version models. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * 增加一个按钮，点击后可以按句子切分，添加“|” (#81) * Update re_matching.py * Update webui.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix phonemer bug (#83) * Fix phonemer bug * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix long vowel handler bug (#84) * Fix long vowel handler bug * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * 加入整合包管理器的特性：长文本合成可以自定义句间段间停顿 (#85) * Update webui.py * Update re_matching.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update train_ms.py * fix' * Update cleaner.py * add en * add en * Update english.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add en * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add en * add en * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add en * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * 更新 README.md * 更新 README.md * 更新 README.md * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Change phonemer to pyopenjtalk (#86) * Change phonemer to pyopenjtalk * 修改为openjtalk便于安装 --------- Co-authored-by: Stardust·减 <star_dust_chen@foxmail.com> * 更新 english.py * Fix english_bert_mock.py. (#87) * Add punctuation execptions (#88) * Add punctuation execptions * Ellipses exceptions * remove get bert * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix bug in oldVersion. (#89) * Update requirements.txt * change to large * rollback requirements.txt * Feat: Enable 1.1.1 models using fix-ver infer. (#91) * Feat: Enable 1.1.1 models using fix-ver infer. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add Japanese accent (high-low) (#90) * Add punctuation execptions * Ellipses exceptions * Add Japanese accent * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Do not replace iteration mark (#92) * Add punctuation execptions * Ellipses exceptions * Add Japanese accent * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not replace iteration mark --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix: fix import error in oldVersion (#93) * Refactor: reusing model loading in webui.py and server.py. (#94) * Feat: Enable using config.yml in train_ms.py (#96) * 更新 emo_gen.py * Change emo_gen.py (#97) * Fix emo_gen bugs * Add multiprocess * Fix queue (#98) * Fix emo_gen bugs * Add multiprocess * Del var * Fix queue * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix training bugs (#99) * Updatge cluster notebook * Fix train * Fix filename * Update infer.py (#100) * Update infer.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add reference audio (#101) * Add reference audio * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update * Update * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Stardust·减 <star_dust_chen@foxmail.com> * Fix: fix 1.1.1-fix (#102) * Fix infer bug (#103) * Feat: Add server_fastapi.py. (#104) * Feat: Add server_fastapi.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix: Update requirements.txt. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix: requirements.txt. (#105) * Swith to deberta-v3-large (#106) * Swith to deberta-v3-large * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Feat: Update config.py. (#107) * Feat: Update config.py. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Dev fix (#108) * fix bugs when deploying * fix bugs when deploying * fix bugs when deploying * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Revert "Dev fix (#108)" (#109) This reverts commit 685e18a10498d602b1a9a26079340d11925646f0. * Dev fix (#110) * fix bugs when deploying * fix bugs when deploying * fix bugs when deploying * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix fixed bugs * fix fixed bugs * fix fixed bug 3 * fix fixed bug 4 * fix fixed bug 5 * fix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add emo vec quantizer (#111) Co-authored-by: Stardust·减 <star_dust_chen@foxmail.com> * Clean req and gitignore (#112) * Clean req and gitignore * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Switch to deberta-v2-large-japanese (#113) * Switch to deberta-v2-large-japanese * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix emo bugs (#114) * Fix english (#115) * Remove emo (#117) * Don't train codebook * Remove emo * Update * Update * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Merge dev into no-emo (#122) * [pre-commit.ci] pre-commit autoupdate (#95) * [pre-commit.ci] pre-commit autoupdate updates: - [github.com/astral-sh/ruff-pre-commit: v0.0.292 → v0.1.1](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.292...v0.1.1) - [github.com/psf/black: 23.9.1 → 23.10.0](https://github.com/psf/black/compare/23.9.1...23.10.0) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Don't train codebook (#116) * Update requirements.txt * Update english_bert_mock.py * Fix: server_fastapi.py (#118) * Fix: server_fastapi.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix: don't print debug logging. (#119) * Fix: don't print debug logging. * Feat: support emo_gen config * Fix config * Apply Code Formatter Change * 更新，修正bug (#121) * Feat: Update infer.py preprocess_text.py server_fastapi.py. * Fix resample.py. Maintain same directory structure in out_dir as in_dir. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update resample.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update server_fastapi.py to no-emo ver * Update config.py, no emo config --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: OedoSoldier <31711261+OedoSoldier@users.noreply.github.com> Co-authored-by: Stardust·减 <star_dust_chen@foxmail.com> Co-authored-by: Stardust-minus <Stardust-minus@users.noreply.github.com> * Update train_ms.py * Update latest version info (#124) --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: jiangyuxiaoxiao <atri@suzakuintsubaki.com> Co-authored-by: AkitoLiu <39857739+Akito-UzukiP@users.noreply.github.com> Co-authored-by: Stardust-minus <Stardust-minus@users.noreply.github.com> Co-authored-by: OedoSoldier <31711261+OedoSoldier@users.noreply.github.com> Co-authored-by: spicysama <122108331+AnyaCoder@users.noreply.github.com> Co-authored-by: innnky <67028263+innnky@users.noreply.github.com> Co-authored-by: YYuX-1145 <138500330+YYuX-1145@users.noreply.github.com>
This commit is contained in:
Stardust·减
12
.gitignore
vendored
12
.gitignore
vendored
@@ -166,3 +166,15 @@ cython_debug/
|
||||
filelists/*
|
||||
!/filelists/esd.list
|
||||
data/*
|
||||
/config.yml
|
||||
/Web/
|
||||
/emotional/*/*.bin
|
||||
/bert/*/*.bin
|
||||
/bert/*/*.h5
|
||||
/bert/*/*.model
|
||||
/bert/*/*.safetensors
|
||||
/bert/*/*.msgpack
|
||||
asr_transcript.py
|
||||
extract_list.py
|
||||
/Data
|
||||
Data/*
|
||||
|
||||
0
.gitmodules
vendored
Normal file
0
.gitmodules
vendored
Normal file
12
README.md
12
README.md
@@ -10,16 +10,8 @@ VITS2 Backbone with bert
|
||||
[//]: # ()
|
||||
[//]: # (本仓库来源于之前朋友分享了ai峰哥的视频，本人被其中的效果惊艳，在自己尝试MassTTS以后发现fs在音质方面与vits有一定差距，并且training的pipeline比vits更复杂，因此按照其思路将bert)
|
||||
|
||||
[//]: # (与vits结合起来以获得更好的韵律。本身我们是出于兴趣玩开源项目，用爱发电，我们本无意与任何人起冲突，然而[MaxMax2016](https://github.com/MaxMax2016))
|
||||
|
||||
[//]: # (以及其organization[PlayVoice](https://github.com/PlayVoice)几次三番前来碰瓷，说本项目抄袭了他们的代码，甚至上法院云云，因此在Readme中特别声明，本项目与)
|
||||
|
||||
[//]: # ([PlayVoice/vits_chinese](https://github.com/PlayVoice/vits_chinese)没有任何关系，结合bert的思路方面也是完全来源于MassTTS)
|
||||
|
||||
|
||||
[//]: # (附：对面认为本项目抄袭了他代码的证据，诸位可以自行查看并做出判断，[bert_vits2引用的MassTTS的实际代码](https://github.com/PlayVoice/vits_chinese/tree/4781241520c6b9fdcf090fca289148719272e89f#bert_vits2%E5%BC%95%E7%94%A8%E7%9A%84masstts%E7%9A%84%E5%AE%9E%E9%99%85%E4%BB%A3%E7%A0%81) )
|
||||
|
||||
## 成熟的旅行者/开拓者/舰长/博士/sensei/猎魔人/喵喵露/V应当参阅代码自己学习如何训练。
|
||||
|
||||
### 严禁将此项目用于一切违反《中华人民共和国宪法》，《中华人民共和国刑法》，《中华人民共和国治安管理处罚法》和《中华人民共和国民法典》之用途。
|
||||
### 严禁用于任何政治相关用途。
|
||||
#### Video:https://www.bilibili.com/video/BV1hp4y1K78E
|
||||
@@ -30,6 +22,8 @@ VITS2 Backbone with bert
|
||||
+ [p0p4k/vits2_pytorch](https://github.com/p0p4k/vits2_pytorch)
|
||||
+ [svc-develop-team/so-vits-svc](https://github.com/svc-develop-team/so-vits-svc)
|
||||
+ [PaddlePaddle/PaddleSpeech](https://github.com/PaddlePaddle/PaddleSpeech)
|
||||
+ [emotional-vits](https://github.com/innnky/emotional-vits)
|
||||
+ [Bert-VITS2-en](https://github.com/xwan07017/Bert-VITS2-en)
|
||||
## 感谢所有贡献者作出的努力
|
||||
<a href="https://github.com/fishaudio/Bert-VITS2/graphs/contributors" target="_blank">
|
||||
<img src="https://contrib.rocks/image?repo=fishaudio/Bert-VITS2"/>
|
||||
|
||||
34
bert/bert-base-japanese-v3/.gitattributes
vendored
Normal file
34
bert/bert-base-japanese-v3/.gitattributes
vendored
Normal file
@@ -0,0 +1,34 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
34
bert/bert-large-japanese-v2/.gitattributes
vendored
Normal file
34
bert/bert-large-japanese-v2/.gitattributes
vendored
Normal file
@@ -0,0 +1,34 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
53
bert/bert-large-japanese-v2/README.md
Normal file
53
bert/bert-large-japanese-v2/README.md
Normal file
@@ -0,0 +1,53 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
datasets:
|
||||
- cc100
|
||||
- wikipedia
|
||||
language:
|
||||
- ja
|
||||
widget:
|
||||
- text: 東北大学で[MASK]の研究をしています。
|
||||
---
|
||||
|
||||
# BERT large Japanese (unidic-lite with whole word masking, CC-100 and jawiki-20230102)
|
||||
|
||||
This is a [BERT](https://github.com/google-research/bert) model pretrained on texts in the Japanese language.
|
||||
|
||||
This version of the model processes input texts with word-level tokenization based on the Unidic 2.1.2 dictionary (available in [unidic-lite](https://pypi.org/project/unidic-lite/) package), followed by the WordPiece subword tokenization.
|
||||
Additionally, the model is trained with the whole word masking enabled for the masked language modeling (MLM) objective.
|
||||
|
||||
The codes for the pretraining are available at [cl-tohoku/bert-japanese](https://github.com/cl-tohoku/bert-japanese/).
|
||||
|
||||
## Model architecture
|
||||
|
||||
The model architecture is the same as the original BERT large model; 24 layers, 1024 dimensions of hidden states, and 16 attention heads.
|
||||
|
||||
## Training Data
|
||||
|
||||
The model is trained on the Japanese portion of [CC-100 dataset](https://data.statmt.org/cc-100/) and the Japanese version of Wikipedia.
|
||||
For Wikipedia, we generated a text corpus from the [Wikipedia Cirrussearch dump file](https://dumps.wikimedia.org/other/cirrussearch/) as of January 2, 2023.
|
||||
The corpus files generated from CC-100 and Wikipedia are 74.3GB and 4.9GB in size and consist of approximately 392M and 34M sentences, respectively.
|
||||
|
||||
For the purpose of splitting texts into sentences, we used [fugashi](https://github.com/polm/fugashi) with [mecab-ipadic-NEologd](https://github.com/neologd/mecab-ipadic-neologd) dictionary (v0.0.7).
|
||||
|
||||
## Tokenization
|
||||
|
||||
The texts are first tokenized by MeCab with the Unidic 2.1.2 dictionary and then split into subwords by the WordPiece algorithm.
|
||||
The vocabulary size is 32768.
|
||||
|
||||
We used [fugashi](https://github.com/polm/fugashi) and [unidic-lite](https://github.com/polm/unidic-lite) packages for the tokenization.
|
||||
|
||||
## Training
|
||||
|
||||
We trained the model first on the CC-100 corpus for 1M steps and then on the Wikipedia corpus for another 1M steps.
|
||||
For training of the MLM (masked language modeling) objective, we introduced whole word masking in which all of the subword tokens corresponding to a single word (tokenized by MeCab) are masked at once.
|
||||
|
||||
For training of each model, we used a v3-8 instance of Cloud TPUs provided by [TPU Research Cloud](https://sites.research.google/trc/about/).
|
||||
|
||||
## Licenses
|
||||
|
||||
The pretrained models are distributed under the Apache License 2.0.
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
This model is trained with Cloud TPUs provided by [TPU Research Cloud](https://sites.research.google/trc/about/) program.
|
||||
19
bert/bert-large-japanese-v2/config.json
Normal file
19
bert/bert-large-japanese-v2/config.json
Normal file
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"architectures": [
|
||||
"BertForPreTraining"
|
||||
],
|
||||
"attention_probs_dropout_prob": 0.1,
|
||||
"hidden_act": "gelu",
|
||||
"hidden_dropout_prob": 0.1,
|
||||
"hidden_size": 1024,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 4096,
|
||||
"layer_norm_eps": 1e-12,
|
||||
"max_position_embeddings": 512,
|
||||
"model_type": "bert",
|
||||
"num_attention_heads": 16,
|
||||
"num_hidden_layers": 24,
|
||||
"pad_token_id": 0,
|
||||
"type_vocab_size": 2,
|
||||
"vocab_size": 32768
|
||||
}
|
||||
10
bert/bert-large-japanese-v2/tokenizer_config.json
Normal file
10
bert/bert-large-japanese-v2/tokenizer_config.json
Normal file
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"tokenizer_class": "BertJapaneseTokenizer",
|
||||
"model_max_length": 512,
|
||||
"do_lower_case": false,
|
||||
"word_tokenizer_type": "mecab",
|
||||
"subword_tokenizer_type": "wordpiece",
|
||||
"mecab_kwargs": {
|
||||
"mecab_dic": "unidic_lite"
|
||||
}
|
||||
}
|
||||
32768
bert/bert-large-japanese-v2/vocab.txt
Normal file
32768
bert/bert-large-japanese-v2/vocab.txt
Normal file
File diff suppressed because it is too large
Load Diff
@@ -1 +0,0 @@
|
||||
*.bin
|
||||
34
bert/deberta-v2-large-japanese/.gitattributes
vendored
Normal file
34
bert/deberta-v2-large-japanese/.gitattributes
vendored
Normal file
@@ -0,0 +1,34 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
111
bert/deberta-v2-large-japanese/README.md
Normal file
111
bert/deberta-v2-large-japanese/README.md
Normal file
@@ -0,0 +1,111 @@
|
||||
---
|
||||
language: ja
|
||||
license: cc-by-sa-4.0
|
||||
library_name: transformers
|
||||
tags:
|
||||
- deberta
|
||||
- deberta-v2
|
||||
- fill-mask
|
||||
datasets:
|
||||
- wikipedia
|
||||
- cc100
|
||||
- oscar
|
||||
metrics:
|
||||
- accuracy
|
||||
mask_token: "[MASK]"
|
||||
widget:
|
||||
- text: "京都 大学 で 自然 言語 処理 を [MASK] する 。"
|
||||
---
|
||||
|
||||
# Model Card for Japanese DeBERTa V2 large
|
||||
|
||||
## Model description
|
||||
|
||||
This is a Japanese DeBERTa V2 large model pre-trained on Japanese Wikipedia, the Japanese portion of CC-100, and the
|
||||
Japanese portion of OSCAR.
|
||||
|
||||
## How to use
|
||||
|
||||
You can use this model for masked language modeling as follows:
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer, AutoModelForMaskedLM
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained('ku-nlp/deberta-v2-large-japanese')
|
||||
model = AutoModelForMaskedLM.from_pretrained('ku-nlp/deberta-v2-large-japanese')
|
||||
|
||||
sentence = '京都 大学 で 自然 言語 処理 を [MASK] する 。' # input should be segmented into words by Juman++ in advance
|
||||
encoding = tokenizer(sentence, return_tensors='pt')
|
||||
...
|
||||
```
|
||||
|
||||
You can also fine-tune this model on downstream tasks.
|
||||
|
||||
## Tokenization
|
||||
|
||||
The input text should be segmented into words by [Juman++](https://github.com/ku-nlp/jumanpp) in
|
||||
advance. [Juman++ 2.0.0-rc3](https://github.com/ku-nlp/jumanpp/releases/tag/v2.0.0-rc3) was used for pre-training. Each
|
||||
word is tokenized into subwords by [sentencepiece](https://github.com/google/sentencepiece).
|
||||
|
||||
## Training data
|
||||
|
||||
We used the following corpora for pre-training:
|
||||
|
||||
- Japanese Wikipedia (as of 20221020, 3.2GB, 27M sentences, 1.3M documents)
|
||||
- Japanese portion of CC-100 (85GB, 619M sentences, 66M documents)
|
||||
- Japanese portion of OSCAR (54GB, 326M sentences, 25M documents)
|
||||
|
||||
Note that we filtered out documents annotated with "header", "footer", or "noisy" tags in OSCAR.
|
||||
Also note that Japanese Wikipedia was duplicated 10 times to make the total size of the corpus comparable to that of
|
||||
CC-100 and OSCAR. As a result, the total size of the training data is 171GB.
|
||||
|
||||
## Training procedure
|
||||
|
||||
We first segmented texts in the corpora into words using [Juman++](https://github.com/ku-nlp/jumanpp).
|
||||
Then, we built a sentencepiece model with 32000 tokens including words ([JumanDIC](https://github.com/ku-nlp/JumanDIC))
|
||||
and subwords induced by the unigram language model of [sentencepiece](https://github.com/google/sentencepiece).
|
||||
|
||||
We tokenized the segmented corpora into subwords using the sentencepiece model and trained the Japanese DeBERTa model
|
||||
using [transformers](https://github.com/huggingface/transformers) library.
|
||||
The training took 36 days using 8 NVIDIA A100-SXM4-40GB GPUs.
|
||||
|
||||
The following hyperparameters were used during pre-training:
|
||||
|
||||
- learning_rate: 1e-4
|
||||
- per_device_train_batch_size: 18
|
||||
- distributed_type: multi-GPU
|
||||
- num_devices: 8
|
||||
- gradient_accumulation_steps: 16
|
||||
- total_train_batch_size: 2,304
|
||||
- max_seq_length: 512
|
||||
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-06
|
||||
- lr_scheduler_type: linear schedule with warmup
|
||||
- training_steps: 300,000
|
||||
- warmup_steps: 10,000
|
||||
|
||||
The accuracy of the trained model on the masked language modeling task was 0.799.
|
||||
The evaluation set consists of 5,000 randomly sampled documents from each of the training corpora.
|
||||
|
||||
## Fine-tuning on NLU tasks
|
||||
|
||||
We fine-tuned the following models and evaluated them on the dev set of JGLUE.
|
||||
We tuned learning rate and training epochs for each model and task
|
||||
following [the JGLUE paper](https://www.jstage.jst.go.jp/article/jnlp/30/1/30_63/_pdf/-char/ja).
|
||||
|
||||
| Model | MARC-ja/acc | JSTS/pearson | JSTS/spearman | JNLI/acc | JSQuAD/EM | JSQuAD/F1 | JComQA/acc |
|
||||
|-------------------------------|-------------|--------------|---------------|----------|-----------|-----------|------------|
|
||||
| Waseda RoBERTa base | 0.965 | 0.913 | 0.876 | 0.905 | 0.853 | 0.916 | 0.853 |
|
||||
| Waseda RoBERTa large (seq512) | 0.969 | 0.925 | 0.890 | 0.928 | 0.910 | 0.955 | 0.900 |
|
||||
| LUKE Japanese base* | 0.965 | 0.916 | 0.877 | 0.912 | - | - | 0.842 |
|
||||
| LUKE Japanese large* | 0.965 | 0.932 | 0.902 | 0.927 | - | - | 0.893 |
|
||||
| DeBERTaV2 base | 0.970 | 0.922 | 0.886 | 0.922 | 0.899 | 0.951 | 0.873 |
|
||||
| DeBERTaV2 large | 0.968 | 0.925 | 0.892 | 0.924 | 0.912 | 0.959 | 0.890 |
|
||||
|
||||
*The scores of LUKE are from [the official repository](https://github.com/studio-ousia/luke).
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
This work was supported by Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures (
|
||||
JHPCN) through General Collaboration Project no. jh221004, "Developing a Platform for Constructing and Sharing of
|
||||
Large-Scale Japanese Language Models".
|
||||
For training models, we used the mdx: a platform for the data-driven future.
|
||||
38
bert/deberta-v2-large-japanese/config.json
Normal file
38
bert/deberta-v2-large-japanese/config.json
Normal file
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"_name_or_path": "configs/deberta_v2_large.json",
|
||||
"architectures": [
|
||||
"DebertaV2ForMaskedLM"
|
||||
],
|
||||
"attention_head_size": 64,
|
||||
"attention_probs_dropout_prob": 0.1,
|
||||
"conv_act": "gelu",
|
||||
"conv_kernel_size": 3,
|
||||
"hidden_act": "gelu",
|
||||
"hidden_dropout_prob": 0.1,
|
||||
"hidden_size": 1024,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 4096,
|
||||
"layer_norm_eps": 1e-07,
|
||||
"max_position_embeddings": 512,
|
||||
"max_relative_positions": -1,
|
||||
"model_type": "deberta-v2",
|
||||
"norm_rel_ebd": "layer_norm",
|
||||
"num_attention_heads": 16,
|
||||
"num_hidden_layers": 24,
|
||||
"pad_token_id": 0,
|
||||
"pooler_dropout": 0,
|
||||
"pooler_hidden_act": "gelu",
|
||||
"pooler_hidden_size": 1024,
|
||||
"pos_att_type": [
|
||||
"p2c",
|
||||
"c2p"
|
||||
],
|
||||
"position_biased_input": false,
|
||||
"position_buckets": 256,
|
||||
"relative_attention": true,
|
||||
"share_att_key": true,
|
||||
"torch_dtype": "float32",
|
||||
"transformers_version": "4.23.1",
|
||||
"type_vocab_size": 0,
|
||||
"vocab_size": 32000
|
||||
}
|
||||
9
bert/deberta-v2-large-japanese/special_tokens_map.json
Normal file
9
bert/deberta-v2-large-japanese/special_tokens_map.json
Normal file
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"bos_token": "[CLS]",
|
||||
"cls_token": "[CLS]",
|
||||
"eos_token": "[SEP]",
|
||||
"mask_token": "[MASK]",
|
||||
"pad_token": "[PAD]",
|
||||
"sep_token": "[SEP]",
|
||||
"unk_token": "[UNK]"
|
||||
}
|
||||
128169
bert/deberta-v2-large-japanese/tokenizer.json
Normal file
128169
bert/deberta-v2-large-japanese/tokenizer.json
Normal file
File diff suppressed because one or more lines are too long
15
bert/deberta-v2-large-japanese/tokenizer_config.json
Normal file
15
bert/deberta-v2-large-japanese/tokenizer_config.json
Normal file
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"bos_token": "[CLS]",
|
||||
"cls_token": "[CLS]",
|
||||
"do_lower_case": false,
|
||||
"eos_token": "[SEP]",
|
||||
"keep_accents": true,
|
||||
"mask_token": "[MASK]",
|
||||
"pad_token": "[PAD]",
|
||||
"sep_token": "[SEP]",
|
||||
"sp_model_kwargs": {},
|
||||
"special_tokens_map_file": null,
|
||||
"split_by_punct": false,
|
||||
"tokenizer_class": "DebertaV2Tokenizer",
|
||||
"unk_token": "[UNK]"
|
||||
}
|
||||
27
bert/deberta-v3-large/.gitattributes
vendored
Normal file
27
bert/deberta-v3-large/.gitattributes
vendored
Normal file
@@ -0,0 +1,27 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zstandard filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
93
bert/deberta-v3-large/README.md
Normal file
93
bert/deberta-v3-large/README.md
Normal file
@@ -0,0 +1,93 @@
|
||||
---
|
||||
language: en
|
||||
tags:
|
||||
- deberta
|
||||
- deberta-v3
|
||||
- fill-mask
|
||||
thumbnail: https://huggingface.co/front/thumbnails/microsoft.png
|
||||
license: mit
|
||||
---
|
||||
|
||||
## DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
|
||||
|
||||
[DeBERTa](https://arxiv.org/abs/2006.03654) improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data.
|
||||
|
||||
In [DeBERTa V3](https://arxiv.org/abs/2111.09543), we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our [paper](https://arxiv.org/abs/2111.09543).
|
||||
|
||||
Please check the [official repository](https://github.com/microsoft/DeBERTa) for more implementation details and updates.
|
||||
|
||||
The DeBERTa V3 large model comes with 24 layers and a hidden size of 1024. It has 304M backbone parameters with a vocabulary containing 128K tokens which introduces 131M parameters in the Embedding layer. This model was trained using the 160GB data as DeBERTa V2.
|
||||
|
||||
|
||||
#### Fine-tuning on NLU tasks
|
||||
|
||||
We present the dev results on SQuAD 2.0 and MNLI tasks.
|
||||
|
||||
| Model |Vocabulary(K)|Backbone #Params(M)| SQuAD 2.0(F1/EM) | MNLI-m/mm(ACC)|
|
||||
|-------------------|----------|-------------------|-----------|----------|
|
||||
| RoBERTa-large |50 |304 | 89.4/86.5 | 90.2 |
|
||||
| XLNet-large |32 |- | 90.6/87.9 | 90.8 |
|
||||
| DeBERTa-large |50 |- | 90.7/88.0 | 91.3 |
|
||||
| **DeBERTa-v3-large**|128|304 | **91.5/89.0**| **91.8/91.9**|
|
||||
|
||||
|
||||
#### Fine-tuning with HF transformers
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
cd transformers/examples/pytorch/text-classification/
|
||||
|
||||
pip install datasets
|
||||
export TASK_NAME=mnli
|
||||
|
||||
output_dir="ds_results"
|
||||
|
||||
num_gpus=8
|
||||
|
||||
batch_size=8
|
||||
|
||||
python -m torch.distributed.launch --nproc_per_node=${num_gpus} \
|
||||
run_glue.py \
|
||||
--model_name_or_path microsoft/deberta-v3-large \
|
||||
--task_name $TASK_NAME \
|
||||
--do_train \
|
||||
--do_eval \
|
||||
--evaluation_strategy steps \
|
||||
--max_seq_length 256 \
|
||||
--warmup_steps 50 \
|
||||
--per_device_train_batch_size ${batch_size} \
|
||||
--learning_rate 6e-6 \
|
||||
--num_train_epochs 2 \
|
||||
--output_dir $output_dir \
|
||||
--overwrite_output_dir \
|
||||
--logging_steps 1000 \
|
||||
--logging_dir $output_dir
|
||||
|
||||
```
|
||||
|
||||
### Citation
|
||||
|
||||
If you find DeBERTa useful for your work, please cite the following papers:
|
||||
|
||||
``` latex
|
||||
@misc{he2021debertav3,
|
||||
title={DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing},
|
||||
author={Pengcheng He and Jianfeng Gao and Weizhu Chen},
|
||||
year={2021},
|
||||
eprint={2111.09543},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CL}
|
||||
}
|
||||
```
|
||||
|
||||
``` latex
|
||||
@inproceedings{
|
||||
he2021deberta,
|
||||
title={DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION},
|
||||
author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
|
||||
booktitle={International Conference on Learning Representations},
|
||||
year={2021},
|
||||
url={https://openreview.net/forum?id=XPZIaotutsD}
|
||||
}
|
||||
```
|
||||
22
bert/deberta-v3-large/config.json
Normal file
22
bert/deberta-v3-large/config.json
Normal file
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"model_type": "deberta-v2",
|
||||
"attention_probs_dropout_prob": 0.1,
|
||||
"hidden_act": "gelu",
|
||||
"hidden_dropout_prob": 0.1,
|
||||
"hidden_size": 1024,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 4096,
|
||||
"max_position_embeddings": 512,
|
||||
"relative_attention": true,
|
||||
"position_buckets": 256,
|
||||
"norm_rel_ebd": "layer_norm",
|
||||
"share_att_key": true,
|
||||
"pos_att_type": "p2c|c2p",
|
||||
"layer_norm_eps": 1e-7,
|
||||
"max_relative_positions": -1,
|
||||
"position_biased_input": false,
|
||||
"num_attention_heads": 16,
|
||||
"num_hidden_layers": 24,
|
||||
"type_vocab_size": 0,
|
||||
"vocab_size": 128100
|
||||
}
|
||||
22
bert/deberta-v3-large/generator_config.json
Normal file
22
bert/deberta-v3-large/generator_config.json
Normal file
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"model_type": "deberta-v2",
|
||||
"attention_probs_dropout_prob": 0.1,
|
||||
"hidden_act": "gelu",
|
||||
"hidden_dropout_prob": 0.1,
|
||||
"hidden_size": 1024,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 4096,
|
||||
"max_position_embeddings": 512,
|
||||
"relative_attention": true,
|
||||
"position_buckets": 256,
|
||||
"norm_rel_ebd": "layer_norm",
|
||||
"share_att_key": true,
|
||||
"pos_att_type": "p2c|c2p",
|
||||
"layer_norm_eps": 1e-7,
|
||||
"max_relative_positions": -1,
|
||||
"position_biased_input": false,
|
||||
"num_attention_heads": 16,
|
||||
"num_hidden_layers": 12,
|
||||
"type_vocab_size": 0,
|
||||
"vocab_size": 128100
|
||||
}
|
||||
4
bert/deberta-v3-large/tokenizer_config.json
Normal file
4
bert/deberta-v3-large/tokenizer_config.json
Normal file
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"do_lower_case": false,
|
||||
"vocab_type": "spm"
|
||||
}
|
||||
39
bert_gen.py
39
bert_gen.py
@@ -6,14 +6,19 @@ from tqdm import tqdm
|
||||
from text import cleaned_text_to_sequence, get_bert
|
||||
import argparse
|
||||
import torch.multiprocessing as mp
|
||||
from config import config
|
||||
|
||||
|
||||
def process_line(line):
|
||||
rank = mp.current_process()._identity
|
||||
rank = rank[0] if len(rank) > 0 else 0
|
||||
if torch.cuda.is_available():
|
||||
gpu_id = rank % torch.cuda.device_count()
|
||||
device = torch.device(f"cuda:{gpu_id}")
|
||||
device = config.bert_gen_config.device
|
||||
if config.bert_gen_config.use_multi_device:
|
||||
rank = mp.current_process()._identity
|
||||
rank = rank[0] if len(rank) > 0 else 0
|
||||
if torch.cuda.is_available():
|
||||
gpu_id = rank % torch.cuda.device_count()
|
||||
device = torch.device(f"cuda:{gpu_id}")
|
||||
else:
|
||||
device = torch.device("cpu")
|
||||
wav_path, _, language_str, text, phones, tone, word2ph = line.strip().split("|")
|
||||
phone = phones.split(" ")
|
||||
tone = [int(i) for i in tone.split(" ")]
|
||||
@@ -28,7 +33,7 @@ def process_line(line):
|
||||
word2ph[i] = word2ph[i] * 2
|
||||
word2ph[0] += 1
|
||||
|
||||
bert_path = wav_path.replace(".wav", ".bert.pt")
|
||||
bert_path = wav_path.replace(".WAV", ".wav").replace(".wav", ".bert.pt")
|
||||
|
||||
try:
|
||||
bert = torch.load(bert_path)
|
||||
@@ -39,11 +44,17 @@ def process_line(line):
|
||||
torch.save(bert, bert_path)
|
||||
|
||||
|
||||
preprocess_text_config = config.preprocess_text_config
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("-c", "--config", type=str, default="configs/config.json")
|
||||
parser.add_argument("--num_processes", type=int, default=2)
|
||||
args = parser.parse_args()
|
||||
parser.add_argument(
|
||||
"-c", "--config", type=str, default=config.bert_gen_config.config_path
|
||||
)
|
||||
parser.add_argument(
|
||||
"--num_processes", type=int, default=config.bert_gen_config.num_processes
|
||||
)
|
||||
args, _ = parser.parse_known_args()
|
||||
config_path = args.config
|
||||
hps = utils.get_hparams_from_file(config_path)
|
||||
lines = []
|
||||
@@ -52,8 +63,10 @@ if __name__ == "__main__":
|
||||
|
||||
with open(hps.data.validation_files, encoding="utf-8") as f:
|
||||
lines.extend(f.readlines())
|
||||
if len(lines) != 0:
|
||||
num_processes = args.num_processes
|
||||
with Pool(processes=num_processes) as pool:
|
||||
for _ in tqdm(pool.imap_unordered(process_line, lines), total=len(lines)):
|
||||
pass
|
||||
|
||||
num_processes = args.num_processes
|
||||
with Pool(processes=num_processes) as pool:
|
||||
for _ in tqdm(pool.imap_unordered(process_line, lines), total=len(lines)):
|
||||
pass
|
||||
print(f"bert生成完毕!, 共有{len(lines)}个bert.pt生成!")
|
||||
|
||||
@@ -50,7 +50,13 @@ def slice_segments(x, ids_str, segment_size=4):
|
||||
for i in range(x.size(0)):
|
||||
idx_str = ids_str[i]
|
||||
idx_end = idx_str + segment_size
|
||||
ret[i] = x[i, :, idx_str:idx_end]
|
||||
if idx_str < 0:
|
||||
i1 = x.size(2) + idx_str
|
||||
r1 = x[i, :, i1:]
|
||||
r2 = x[i, :, :idx_end]
|
||||
ret[i] = torch.cat([r1, r2], dim=1)
|
||||
else:
|
||||
ret[i] = x[i, :, idx_str:idx_end]
|
||||
return ret
|
||||
|
||||
|
||||
|
||||
228
config.py
Normal file
228
config.py
Normal file
@@ -0,0 +1,228 @@
|
||||
"""
|
||||
@Desc: 全局配置文件读取
|
||||
"""
|
||||
import argparse
|
||||
import yaml
|
||||
from typing import Dict, List
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
|
||||
class Resample_config:
|
||||
"""重采样配置"""
|
||||
|
||||
def __init__(self, in_dir: str, out_dir: str, sampling_rate: int = 44100):
|
||||
self.sampling_rate: int = sampling_rate # 目标采样率
|
||||
self.in_dir: str = in_dir # 待处理音频目录路径
|
||||
self.out_dir: str = out_dir # 重采样输出路径
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
|
||||
"""从字典中生成实例"""
|
||||
|
||||
# 不检查路径是否有效，此逻辑在resample.py中处理
|
||||
data["in_dir"] = os.path.join(dataset_path, data["in_dir"])
|
||||
data["out_dir"] = os.path.join(dataset_path, data["out_dir"])
|
||||
|
||||
return cls(**data)
|
||||
|
||||
|
||||
class Preprocess_text_config:
|
||||
"""数据预处理配置"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
transcription_path: str,
|
||||
cleaned_path: str,
|
||||
train_path: str,
|
||||
val_path: str,
|
||||
config_path: str,
|
||||
val_per_spk: int = 5,
|
||||
max_val_total: int = 10000,
|
||||
clean: bool = True,
|
||||
):
|
||||
self.transcription_path: str = transcription_path # 原始文本文件路径，文本格式应为{wav_path}|{speaker_name}|{language}|{text}。
|
||||
self.cleaned_path: str = cleaned_path # 数据清洗后文本路径，可以不填。不填则将在原始文本目录生成
|
||||
self.train_path: str = train_path # 训练集路径，可以不填。不填则将在原始文本目录生成
|
||||
self.val_path: str = val_path # 验证集路径，可以不填。不填则将在原始文本目录生成
|
||||
self.config_path: str = config_path # 配置文件路径
|
||||
self.val_per_spk: int = val_per_spk # 每个speaker的验证集条数
|
||||
self.max_val_total: int = max_val_total # 验证集最大条数，多于的会被截断并放到训练集中
|
||||
self.clean: bool = clean # 是否进行数据清洗
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
|
||||
"""从字典中生成实例"""
|
||||
|
||||
data["transcription_path"] = os.path.join(
|
||||
dataset_path, data["transcription_path"]
|
||||
)
|
||||
if data["cleaned_path"] == "" or data["cleaned_path"] is None:
|
||||
data["cleaned_path"] = None
|
||||
else:
|
||||
data["cleaned_path"] = os.path.join(dataset_path, data["cleaned_path"])
|
||||
data["train_path"] = os.path.join(dataset_path, data["train_path"])
|
||||
data["val_path"] = os.path.join(dataset_path, data["val_path"])
|
||||
data["config_path"] = os.path.join(dataset_path, data["config_path"])
|
||||
|
||||
return cls(**data)
|
||||
|
||||
|
||||
class Bert_gen_config:
|
||||
"""bert_gen 配置"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config_path: str,
|
||||
num_processes: int = 2,
|
||||
device: str = "cuda",
|
||||
use_multi_device: bool = False,
|
||||
):
|
||||
self.config_path = config_path
|
||||
self.num_processes = num_processes
|
||||
self.device = device
|
||||
self.use_multi_device = use_multi_device
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
|
||||
data["config_path"] = os.path.join(dataset_path, data["config_path"])
|
||||
|
||||
return cls(**data)
|
||||
|
||||
|
||||
class Emo_gen_config:
|
||||
"""emo_gen 配置"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config_path: str,
|
||||
num_processes: int = 2,
|
||||
device: str = "cuda",
|
||||
):
|
||||
self.config_path = config_path
|
||||
self.num_processes = num_processes
|
||||
self.device = device
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
|
||||
data["config_path"] = os.path.join(dataset_path, data["config_path"])
|
||||
|
||||
return cls(**data)
|
||||
|
||||
|
||||
class Train_ms_config:
|
||||
"""训练配置"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config_path: str,
|
||||
env: Dict[str, any],
|
||||
model: str,
|
||||
):
|
||||
self.env = env # 需要加载的环境变量
|
||||
self.model = model # 训练模型存储目录，该路径为相对于dataset_path的路径，而非项目根目录
|
||||
self.config_path = config_path # 配置文件路径
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
|
||||
# data["model"] = os.path.join(dataset_path, data["model"])
|
||||
data["config_path"] = os.path.join(dataset_path, data["config_path"])
|
||||
|
||||
return cls(**data)
|
||||
|
||||
|
||||
class Webui_config:
|
||||
"""webui 配置"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
device: str,
|
||||
model: str,
|
||||
config_path: str,
|
||||
port: int = 7860,
|
||||
share: bool = False,
|
||||
debug: bool = False,
|
||||
):
|
||||
self.device: str = device
|
||||
self.model: str = model # 端口号
|
||||
self.config_path: str = config_path # 是否公开部署，对外网开放
|
||||
self.port: int = port # 是否开启debug模式
|
||||
self.share: bool = share # 模型路径
|
||||
self.debug: bool = debug # 配置文件路径
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, dataset_path: str, data: Dict[str, any]):
|
||||
data["config_path"] = os.path.join(dataset_path, data["config_path"])
|
||||
data["model"] = os.path.join(dataset_path, data["model"])
|
||||
return cls(**data)
|
||||
|
||||
|
||||
class Server_config:
|
||||
def __init__(
|
||||
self, models: List[Dict[str, any]], port: int = 5000, device: str = "cuda"
|
||||
):
|
||||
self.models: List[Dict[str, any]] = models # 需要加载的所有模型的配置
|
||||
self.port: int = port # 端口号
|
||||
self.device: str = device # 模型默认使用设备
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, any]):
|
||||
return cls(**data)
|
||||
|
||||
|
||||
class Translate_config:
|
||||
"""翻译api配置"""
|
||||
|
||||
def __init__(self, app_key: str, secret_key: str):
|
||||
self.app_key = app_key
|
||||
self.secret_key = secret_key
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, any]):
|
||||
return cls(**data)
|
||||
|
||||
|
||||
class Config:
|
||||
def __init__(self, config_path: str):
|
||||
if not os.path.isfile(config_path) and os.path.isfile("default_config.yml"):
|
||||
shutil.copy(src="default_config.yml", dst=config_path)
|
||||
print(
|
||||
f"已根据默认配置文件default_config.yml生成配置文件{config_path}。请按该配置文件的说明进行配置后重新运行。"
|
||||
)
|
||||
print("如无特殊需求，请勿修改default_config.yml或备份该文件。")
|
||||
sys.exit(0)
|
||||
with open(file=config_path, mode="r", encoding="utf-8") as file:
|
||||
yaml_config: Dict[str, any] = yaml.safe_load(file.read())
|
||||
dataset_path: str = yaml_config["dataset_path"]
|
||||
self.dataset_path: str = dataset_path
|
||||
self.resample_config: Resample_config = Resample_config.from_dict(
|
||||
dataset_path, yaml_config["resample"]
|
||||
)
|
||||
self.preprocess_text_config: Preprocess_text_config = (
|
||||
Preprocess_text_config.from_dict(
|
||||
dataset_path, yaml_config["preprocess_text"]
|
||||
)
|
||||
)
|
||||
self.bert_gen_config: Bert_gen_config = Bert_gen_config.from_dict(
|
||||
dataset_path, yaml_config["bert_gen"]
|
||||
)
|
||||
self.train_ms_config: Train_ms_config = Train_ms_config.from_dict(
|
||||
dataset_path, yaml_config["train_ms"]
|
||||
)
|
||||
self.webui_config: Webui_config = Webui_config.from_dict(
|
||||
dataset_path, yaml_config["webui"]
|
||||
)
|
||||
self.server_config: Server_config = Server_config.from_dict(
|
||||
yaml_config["server"]
|
||||
)
|
||||
self.translate_config: Translate_config = Translate_config.from_dict(
|
||||
yaml_config["translate"]
|
||||
)
|
||||
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
# 为避免与以前的config.json起冲突，将其更名如下
|
||||
parser.add_argument("-y", "--yml_config", type=str, default="config.yml")
|
||||
args, _ = parser.parse_known_args()
|
||||
config = Config(args.yml_config)
|
||||
@@ -1,342 +1,343 @@
|
||||
{
|
||||
"train": {
|
||||
"log_interval": 200,
|
||||
"eval_interval": 1000,
|
||||
"seed": 52,
|
||||
"epochs": 10000,
|
||||
"learning_rate": 0.0003,
|
||||
"betas": [
|
||||
0.8,
|
||||
0.99
|
||||
],
|
||||
"eps": 1e-09,
|
||||
"batch_size": 8,
|
||||
"fp16_run": false,
|
||||
"lr_decay": 0.999875,
|
||||
"segment_size": 16384,
|
||||
"init_lr_ratio": 1,
|
||||
"warmup_epochs": 0,
|
||||
"c_mel": 45,
|
||||
"c_kl": 1.0,
|
||||
"skip_optimizer": true
|
||||
},
|
||||
"data": {
|
||||
"training_files": "filelists/train.list",
|
||||
"validation_files": "filelists/val.list",
|
||||
"max_wav_value": 32768.0,
|
||||
"sampling_rate": 44100,
|
||||
"filter_length": 2048,
|
||||
"hop_length": 512,
|
||||
"win_length": 2048,
|
||||
"n_mel_channels": 128,
|
||||
"mel_fmin": 0.0,
|
||||
"mel_fmax": null,
|
||||
"add_blank": true,
|
||||
"n_speakers": 256,
|
||||
"cleaned_text": true,
|
||||
"spk2id": {
|
||||
"丹恒": 0,
|
||||
"克拉拉": 1,
|
||||
"穹": 2,
|
||||
"「信使」": 3,
|
||||
"史瓦罗": 4,
|
||||
"彦卿": 5,
|
||||
"晴霓": 6,
|
||||
"杰帕德": 7,
|
||||
"素裳": 8,
|
||||
"绿芙蓉": 9,
|
||||
"罗刹": 10,
|
||||
"艾丝妲": 11,
|
||||
"黑塔": 12,
|
||||
"丹枢": 13,
|
||||
"希露瓦": 14,
|
||||
"白露": 15,
|
||||
"费斯曼": 16,
|
||||
"停云": 17,
|
||||
"可可利亚": 18,
|
||||
"景元": 19,
|
||||
"螺丝咕姆": 20,
|
||||
"青镞": 21,
|
||||
"公输师傅": 22,
|
||||
"卡芙卡": 23,
|
||||
"大毫": 24,
|
||||
"驭空": 25,
|
||||
"半夏": 26,
|
||||
"奥列格": 27,
|
||||
"娜塔莎": 28,
|
||||
"桑博": 29,
|
||||
"瓦尔特": 30,
|
||||
"阿兰": 31,
|
||||
"伦纳德": 32,
|
||||
"佩拉": 33,
|
||||
"卡波特": 34,
|
||||
"帕姆": 35,
|
||||
"帕斯卡": 36,
|
||||
"青雀": 37,
|
||||
"三月七": 38,
|
||||
"刃": 39,
|
||||
"姬子": 40,
|
||||
"布洛妮娅": 41,
|
||||
"希儿": 42,
|
||||
"星": 43,
|
||||
"符玄": 44,
|
||||
"虎克": 45,
|
||||
"银狼": 46,
|
||||
"镜流": 47,
|
||||
"「博士」": 48,
|
||||
"「大肉丸」": 49,
|
||||
"九条裟罗": 50,
|
||||
"佐西摩斯": 51,
|
||||
"刻晴": 52,
|
||||
"博易": 53,
|
||||
"卡维": 54,
|
||||
"可莉": 55,
|
||||
"嘉玛": 56,
|
||||
"埃舍尔": 57,
|
||||
"塔杰·拉德卡尼": 58,
|
||||
"大慈树王": 59,
|
||||
"宵宫": 60,
|
||||
"康纳": 61,
|
||||
"影": 62,
|
||||
"枫原万叶": 63,
|
||||
"欧菲妮": 64,
|
||||
"玛乔丽": 65,
|
||||
"珊瑚": 66,
|
||||
"田铁嘴": 67,
|
||||
"砂糖": 68,
|
||||
"神里绫华": 69,
|
||||
"罗莎莉亚": 70,
|
||||
"荒泷一斗": 71,
|
||||
"莎拉": 72,
|
||||
"迪希雅": 73,
|
||||
"钟离": 74,
|
||||
"阿圆": 75,
|
||||
"阿娜耶": 76,
|
||||
"阿拉夫": 77,
|
||||
"雷泽": 78,
|
||||
"香菱": 79,
|
||||
"龙二": 80,
|
||||
"「公子」": 81,
|
||||
"「白老先生」": 82,
|
||||
"优菈": 83,
|
||||
"凯瑟琳": 84,
|
||||
"哲平": 85,
|
||||
"夏洛蒂": 86,
|
||||
"安柏": 87,
|
||||
"巴达维": 88,
|
||||
"式大将": 89,
|
||||
"斯坦利": 90,
|
||||
"毗伽尔": 91,
|
||||
"海妮耶": 92,
|
||||
"爱德琳": 93,
|
||||
"纳西妲": 94,
|
||||
"老孟": 95,
|
||||
"芙宁娜": 96,
|
||||
"阿守": 97,
|
||||
"阿祇": 98,
|
||||
"丹吉尔": 99,
|
||||
"丽莎": 100,
|
||||
"五郎": 101,
|
||||
"元太": 102,
|
||||
"克列门特": 103,
|
||||
"克罗索": 104,
|
||||
"北斗": 105,
|
||||
"埃勒曼": 106,
|
||||
"天目十五": 107,
|
||||
"奥兹": 108,
|
||||
"恶龙": 109,
|
||||
"早柚": 110,
|
||||
"杜拉夫": 111,
|
||||
"松浦": 112,
|
||||
"柊千里": 113,
|
||||
"甘雨": 114,
|
||||
"石头": 115,
|
||||
"纯水精灵？": 116,
|
||||
"羽生田千鹤": 117,
|
||||
"莱依拉": 118,
|
||||
"菲谢尔": 119,
|
||||
"言笑": 120,
|
||||
"诺艾尔": 121,
|
||||
"赛诺": 122,
|
||||
"辛焱": 123,
|
||||
"迪娜泽黛": 124,
|
||||
"那维莱特": 125,
|
||||
"八重神子": 126,
|
||||
"凯亚": 127,
|
||||
"吴船长": 128,
|
||||
"埃德": 129,
|
||||
"天叔": 130,
|
||||
"女士": 131,
|
||||
"恕筠": 132,
|
||||
"提纳里": 133,
|
||||
"派蒙": 134,
|
||||
"流浪者": 135,
|
||||
"深渊使徒": 136,
|
||||
"玛格丽特": 137,
|
||||
"珐露珊": 138,
|
||||
"琴": 139,
|
||||
"瑶瑶": 140,
|
||||
"留云借风真君": 141,
|
||||
"绮良良": 142,
|
||||
"舒伯特": 143,
|
||||
"荧": 144,
|
||||
"莫娜": 145,
|
||||
"行秋": 146,
|
||||
"迈勒斯": 147,
|
||||
"阿佩普": 148,
|
||||
"鹿野奈奈": 149,
|
||||
"七七": 150,
|
||||
"伊迪娅": 151,
|
||||
"博来": 152,
|
||||
"坎蒂丝": 153,
|
||||
"埃尔欣根": 154,
|
||||
"埃泽": 155,
|
||||
"塞琉斯": 156,
|
||||
"夜兰": 157,
|
||||
"常九爷": 158,
|
||||
"悦": 159,
|
||||
"戴因斯雷布": 160,
|
||||
"笼钓瓶一心": 161,
|
||||
"纳比尔": 162,
|
||||
"胡桃": 163,
|
||||
"艾尔海森": 164,
|
||||
"艾莉丝": 165,
|
||||
"菲米尼": 166,
|
||||
"蒂玛乌斯": 167,
|
||||
"迪奥娜": 168,
|
||||
"阿晃": 169,
|
||||
"阿洛瓦": 170,
|
||||
"陆行岩本真蕈·元素生命": 171,
|
||||
"雷电将军": 172,
|
||||
"魈": 173,
|
||||
"鹿野院平藏": 174,
|
||||
"「女士」": 175,
|
||||
"「散兵」": 176,
|
||||
"凝光": 177,
|
||||
"妮露": 178,
|
||||
"娜维娅": 179,
|
||||
"宛烟": 180,
|
||||
"慧心": 181,
|
||||
"托克": 182,
|
||||
"托马": 183,
|
||||
"掇星攫辰天君": 184,
|
||||
"旁白": 185,
|
||||
"浮游水蕈兽·元素生命": 186,
|
||||
"烟绯": 187,
|
||||
"玛塞勒": 188,
|
||||
"百闻": 189,
|
||||
"知易": 190,
|
||||
"米卡": 191,
|
||||
"西拉杰": 192,
|
||||
"迪卢克": 193,
|
||||
"重云": 194,
|
||||
"阿扎尔": 195,
|
||||
"霍夫曼": 196,
|
||||
"上杉": 197,
|
||||
"久利须": 198,
|
||||
"嘉良": 199,
|
||||
"回声海螺": 200,
|
||||
"多莉": 201,
|
||||
"安西": 202,
|
||||
"德沃沙克": 203,
|
||||
"拉赫曼": 204,
|
||||
"林尼": 205,
|
||||
"查尔斯": 206,
|
||||
"深渊法师": 207,
|
||||
"温迪": 208,
|
||||
"爱贝尔": 209,
|
||||
"珊瑚宫心海": 210,
|
||||
"班尼特": 211,
|
||||
"琳妮特": 212,
|
||||
"申鹤": 213,
|
||||
"神里绫人": 214,
|
||||
"艾伯特": 215,
|
||||
"萍姥姥": 216,
|
||||
"萨赫哈蒂": 217,
|
||||
"萨齐因": 218,
|
||||
"阿尔卡米": 219,
|
||||
"阿贝多": 220,
|
||||
"anzai": 221,
|
||||
"久岐忍": 222,
|
||||
"九条镰治": 223,
|
||||
"云堇": 224,
|
||||
"伊利亚斯": 225,
|
||||
"埃洛伊": 226,
|
||||
"塞塔蕾": 227,
|
||||
"拉齐": 228,
|
||||
"昆钧": 229,
|
||||
"柯莱": 230,
|
||||
"沙扎曼": 231,
|
||||
"海芭夏": 232,
|
||||
"白术": 233,
|
||||
"空": 234,
|
||||
"艾文": 235,
|
||||
"芭芭拉": 236,
|
||||
"莫塞伊思": 237,
|
||||
"莺儿": 238,
|
||||
"达达利亚": 239,
|
||||
"迈蒙": 240,
|
||||
"长生": 241,
|
||||
"阿巴图伊": 242,
|
||||
"陆景和": 243,
|
||||
"莫弈": 244,
|
||||
"夏彦": 245,
|
||||
"左然": 246,
|
||||
"标贝": 247
|
||||
}
|
||||
},
|
||||
"model": {
|
||||
"use_spk_conditioned_encoder": true,
|
||||
"use_noise_scaled_mas": true,
|
||||
"use_mel_posterior_encoder": false,
|
||||
"use_duration_discriminator": true,
|
||||
"inter_channels": 192,
|
||||
"hidden_channels": 192,
|
||||
"filter_channels": 768,
|
||||
"n_heads": 2,
|
||||
"n_layers": 6,
|
||||
"kernel_size": 3,
|
||||
"p_dropout": 0.1,
|
||||
"resblock": "1",
|
||||
"resblock_kernel_sizes": [
|
||||
3,
|
||||
7,
|
||||
11
|
||||
],
|
||||
"resblock_dilation_sizes": [
|
||||
[
|
||||
1,
|
||||
3,
|
||||
5
|
||||
],
|
||||
[
|
||||
1,
|
||||
3,
|
||||
5
|
||||
],
|
||||
[
|
||||
1,
|
||||
3,
|
||||
5
|
||||
]
|
||||
],
|
||||
"upsample_rates": [
|
||||
8,
|
||||
8,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
],
|
||||
"upsample_initial_channel": 512,
|
||||
"upsample_kernel_sizes": [
|
||||
16,
|
||||
16,
|
||||
8,
|
||||
2,
|
||||
2
|
||||
],
|
||||
"n_layers_q": 3,
|
||||
"use_spectral_norm": false,
|
||||
"gin_channels": 256
|
||||
}
|
||||
"train": {
|
||||
"log_interval": 200,
|
||||
"eval_interval": 1000,
|
||||
"seed": 52,
|
||||
"epochs": 10000,
|
||||
"learning_rate": 0.0003,
|
||||
"betas": [
|
||||
0.8,
|
||||
0.99
|
||||
],
|
||||
"eps": 1e-09,
|
||||
"batch_size": 8,
|
||||
"fp16_run": false,
|
||||
"lr_decay": 0.999875,
|
||||
"segment_size": 16384,
|
||||
"init_lr_ratio": 1,
|
||||
"warmup_epochs": 0,
|
||||
"c_mel": 45,
|
||||
"c_kl": 1.0,
|
||||
"skip_optimizer": true
|
||||
},
|
||||
"data": {
|
||||
"training_files": "filelists/train.list",
|
||||
"validation_files": "filelists/val.list",
|
||||
"max_wav_value": 32768.0,
|
||||
"sampling_rate": 44100,
|
||||
"filter_length": 2048,
|
||||
"hop_length": 512,
|
||||
"win_length": 2048,
|
||||
"n_mel_channels": 128,
|
||||
"mel_fmin": 0.0,
|
||||
"mel_fmax": null,
|
||||
"add_blank": true,
|
||||
"n_speakers": 256,
|
||||
"cleaned_text": true,
|
||||
"spk2id": {
|
||||
"丹恒": 0,
|
||||
"克拉拉": 1,
|
||||
"穹": 2,
|
||||
"「信使」": 3,
|
||||
"史瓦罗": 4,
|
||||
"彦卿": 5,
|
||||
"晴霓": 6,
|
||||
"杰帕德": 7,
|
||||
"素裳": 8,
|
||||
"绿芙蓉": 9,
|
||||
"罗刹": 10,
|
||||
"艾丝妲": 11,
|
||||
"黑塔": 12,
|
||||
"丹枢": 13,
|
||||
"希露瓦": 14,
|
||||
"白露": 15,
|
||||
"费斯曼": 16,
|
||||
"停云": 17,
|
||||
"可可利亚": 18,
|
||||
"景元": 19,
|
||||
"螺丝咕姆": 20,
|
||||
"青镞": 21,
|
||||
"公输师傅": 22,
|
||||
"卡芙卡": 23,
|
||||
"大毫": 24,
|
||||
"驭空": 25,
|
||||
"半夏": 26,
|
||||
"奥列格": 27,
|
||||
"娜塔莎": 28,
|
||||
"桑博": 29,
|
||||
"瓦尔特": 30,
|
||||
"阿兰": 31,
|
||||
"伦纳德": 32,
|
||||
"佩拉": 33,
|
||||
"卡波特": 34,
|
||||
"帕姆": 35,
|
||||
"帕斯卡": 36,
|
||||
"青雀": 37,
|
||||
"三月七": 38,
|
||||
"刃": 39,
|
||||
"姬子": 40,
|
||||
"布洛妮娅": 41,
|
||||
"希儿": 42,
|
||||
"星": 43,
|
||||
"符玄": 44,
|
||||
"虎克": 45,
|
||||
"银狼": 46,
|
||||
"镜流": 47,
|
||||
"「博士」": 48,
|
||||
"「大肉丸」": 49,
|
||||
"九条裟罗": 50,
|
||||
"佐西摩斯": 51,
|
||||
"刻晴": 52,
|
||||
"博易": 53,
|
||||
"卡维": 54,
|
||||
"可莉": 55,
|
||||
"嘉玛": 56,
|
||||
"埃舍尔": 57,
|
||||
"塔杰·拉德卡尼": 58,
|
||||
"大慈树王": 59,
|
||||
"宵宫": 60,
|
||||
"康纳": 61,
|
||||
"影": 62,
|
||||
"枫原万叶": 63,
|
||||
"欧菲妮": 64,
|
||||
"玛乔丽": 65,
|
||||
"珊瑚": 66,
|
||||
"田铁嘴": 67,
|
||||
"砂糖": 68,
|
||||
"神里绫华": 69,
|
||||
"罗莎莉亚": 70,
|
||||
"荒泷一斗": 71,
|
||||
"莎拉": 72,
|
||||
"迪希雅": 73,
|
||||
"钟离": 74,
|
||||
"阿圆": 75,
|
||||
"阿娜耶": 76,
|
||||
"阿拉夫": 77,
|
||||
"雷泽": 78,
|
||||
"香菱": 79,
|
||||
"龙二": 80,
|
||||
"「公子」": 81,
|
||||
"「白老先生」": 82,
|
||||
"优菈": 83,
|
||||
"凯瑟琳": 84,
|
||||
"哲平": 85,
|
||||
"夏洛蒂": 86,
|
||||
"安柏": 87,
|
||||
"巴达维": 88,
|
||||
"式大将": 89,
|
||||
"斯坦利": 90,
|
||||
"毗伽尔": 91,
|
||||
"海妮耶": 92,
|
||||
"爱德琳": 93,
|
||||
"纳西妲": 94,
|
||||
"老孟": 95,
|
||||
"芙宁娜": 96,
|
||||
"阿守": 97,
|
||||
"阿祇": 98,
|
||||
"丹吉尔": 99,
|
||||
"丽莎": 100,
|
||||
"五郎": 101,
|
||||
"元太": 102,
|
||||
"克列门特": 103,
|
||||
"克罗索": 104,
|
||||
"北斗": 105,
|
||||
"埃勒曼": 106,
|
||||
"天目十五": 107,
|
||||
"奥兹": 108,
|
||||
"恶龙": 109,
|
||||
"早柚": 110,
|
||||
"杜拉夫": 111,
|
||||
"松浦": 112,
|
||||
"柊千里": 113,
|
||||
"甘雨": 114,
|
||||
"石头": 115,
|
||||
"纯水精灵？": 116,
|
||||
"羽生田千鹤": 117,
|
||||
"莱依拉": 118,
|
||||
"菲谢尔": 119,
|
||||
"言笑": 120,
|
||||
"诺艾尔": 121,
|
||||
"赛诺": 122,
|
||||
"辛焱": 123,
|
||||
"迪娜泽黛": 124,
|
||||
"那维莱特": 125,
|
||||
"八重神子": 126,
|
||||
"凯亚": 127,
|
||||
"吴船长": 128,
|
||||
"埃德": 129,
|
||||
"天叔": 130,
|
||||
"女士": 131,
|
||||
"恕筠": 132,
|
||||
"提纳里": 133,
|
||||
"派蒙": 134,
|
||||
"流浪者": 135,
|
||||
"深渊使徒": 136,
|
||||
"玛格丽特": 137,
|
||||
"珐露珊": 138,
|
||||
"琴": 139,
|
||||
"瑶瑶": 140,
|
||||
"留云借风真君": 141,
|
||||
"绮良良": 142,
|
||||
"舒伯特": 143,
|
||||
"荧": 144,
|
||||
"莫娜": 145,
|
||||
"行秋": 146,
|
||||
"迈勒斯": 147,
|
||||
"阿佩普": 148,
|
||||
"鹿野奈奈": 149,
|
||||
"七七": 150,
|
||||
"伊迪娅": 151,
|
||||
"博来": 152,
|
||||
"坎蒂丝": 153,
|
||||
"埃尔欣根": 154,
|
||||
"埃泽": 155,
|
||||
"塞琉斯": 156,
|
||||
"夜兰": 157,
|
||||
"常九爷": 158,
|
||||
"悦": 159,
|
||||
"戴因斯雷布": 160,
|
||||
"笼钓瓶一心": 161,
|
||||
"纳比尔": 162,
|
||||
"胡桃": 163,
|
||||
"艾尔海森": 164,
|
||||
"艾莉丝": 165,
|
||||
"菲米尼": 166,
|
||||
"蒂玛乌斯": 167,
|
||||
"迪奥娜": 168,
|
||||
"阿晃": 169,
|
||||
"阿洛瓦": 170,
|
||||
"陆行岩本真蕈·元素生命": 171,
|
||||
"雷电将军": 172,
|
||||
"魈": 173,
|
||||
"鹿野院平藏": 174,
|
||||
"「女士」": 175,
|
||||
"「散兵」": 176,
|
||||
"凝光": 177,
|
||||
"妮露": 178,
|
||||
"娜维娅": 179,
|
||||
"宛烟": 180,
|
||||
"慧心": 181,
|
||||
"托克": 182,
|
||||
"托马": 183,
|
||||
"掇星攫辰天君": 184,
|
||||
"旁白": 185,
|
||||
"浮游水蕈兽·元素生命": 186,
|
||||
"烟绯": 187,
|
||||
"玛塞勒": 188,
|
||||
"百闻": 189,
|
||||
"知易": 190,
|
||||
"米卡": 191,
|
||||
"西拉杰": 192,
|
||||
"迪卢克": 193,
|
||||
"重云": 194,
|
||||
"阿扎尔": 195,
|
||||
"霍夫曼": 196,
|
||||
"上杉": 197,
|
||||
"久利须": 198,
|
||||
"嘉良": 199,
|
||||
"回声海螺": 200,
|
||||
"多莉": 201,
|
||||
"安西": 202,
|
||||
"德沃沙克": 203,
|
||||
"拉赫曼": 204,
|
||||
"林尼": 205,
|
||||
"查尔斯": 206,
|
||||
"深渊法师": 207,
|
||||
"温迪": 208,
|
||||
"爱贝尔": 209,
|
||||
"珊瑚宫心海": 210,
|
||||
"班尼特": 211,
|
||||
"琳妮特": 212,
|
||||
"申鹤": 213,
|
||||
"神里绫人": 214,
|
||||
"艾伯特": 215,
|
||||
"萍姥姥": 216,
|
||||
"萨赫哈蒂": 217,
|
||||
"萨齐因": 218,
|
||||
"阿尔卡米": 219,
|
||||
"阿贝多": 220,
|
||||
"anzai": 221,
|
||||
"久岐忍": 222,
|
||||
"九条镰治": 223,
|
||||
"云堇": 224,
|
||||
"伊利亚斯": 225,
|
||||
"埃洛伊": 226,
|
||||
"塞塔蕾": 227,
|
||||
"拉齐": 228,
|
||||
"昆钧": 229,
|
||||
"柯莱": 230,
|
||||
"沙扎曼": 231,
|
||||
"海芭夏": 232,
|
||||
"白术": 233,
|
||||
"空": 234,
|
||||
"艾文": 235,
|
||||
"芭芭拉": 236,
|
||||
"莫塞伊思": 237,
|
||||
"莺儿": 238,
|
||||
"达达利亚": 239,
|
||||
"迈蒙": 240,
|
||||
"长生": 241,
|
||||
"阿巴图伊": 242,
|
||||
"陆景和": 243,
|
||||
"莫弈": 244,
|
||||
"夏彦": 245,
|
||||
"左然": 246,
|
||||
"标贝": 247
|
||||
}
|
||||
},
|
||||
"model": {
|
||||
"use_spk_conditioned_encoder": true,
|
||||
"use_noise_scaled_mas": true,
|
||||
"use_mel_posterior_encoder": false,
|
||||
"use_duration_discriminator": true,
|
||||
"inter_channels": 192,
|
||||
"hidden_channels": 192,
|
||||
"filter_channels": 768,
|
||||
"n_heads": 2,
|
||||
"n_layers": 6,
|
||||
"kernel_size": 3,
|
||||
"p_dropout": 0.1,
|
||||
"resblock": "1",
|
||||
"resblock_kernel_sizes": [
|
||||
3,
|
||||
7,
|
||||
11
|
||||
],
|
||||
"resblock_dilation_sizes": [
|
||||
[
|
||||
1,
|
||||
3,
|
||||
5
|
||||
],
|
||||
[
|
||||
1,
|
||||
3,
|
||||
5
|
||||
],
|
||||
[
|
||||
1,
|
||||
3,
|
||||
5
|
||||
]
|
||||
],
|
||||
"upsample_rates": [
|
||||
8,
|
||||
8,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
],
|
||||
"upsample_initial_channel": 512,
|
||||
"upsample_kernel_sizes": [
|
||||
16,
|
||||
16,
|
||||
8,
|
||||
2,
|
||||
2
|
||||
],
|
||||
"n_layers_q": 3,
|
||||
"use_spectral_norm": false,
|
||||
"gin_channels": 256
|
||||
},
|
||||
"version": "2.0"
|
||||
}
|
||||
|
||||
@@ -7,7 +7,7 @@ from loguru import logger
|
||||
import commons
|
||||
from mel_processing import spectrogram_torch, mel_spectrogram_torch
|
||||
from utils import load_wav_to_torch, load_filepaths_and_text
|
||||
from text import cleaned_text_to_sequence, get_bert
|
||||
from text import cleaned_text_to_sequence
|
||||
|
||||
"""Multi speaker version"""
|
||||
|
||||
@@ -85,13 +85,13 @@ class TextAudioSpeakerLoader(torch.utils.data.Dataset):
|
||||
# separate filename, speaker_id and text
|
||||
audiopath, sid, language, text, phones, tone, word2ph = audiopath_sid_text
|
||||
|
||||
bert, ja_bert, phones, tone, language = self.get_text(
|
||||
bert, ja_bert, en_bert, phones, tone, language = self.get_text(
|
||||
text, word2ph, phones, tone, language, audiopath
|
||||
)
|
||||
|
||||
spec, wav = self.get_audio(audiopath)
|
||||
sid = torch.LongTensor([int(self.spk_map[sid])])
|
||||
return (phones, spec, wav, sid, tone, language, bert, ja_bert)
|
||||
return (phones, spec, wav, sid, tone, language, bert, ja_bert, en_bert)
|
||||
|
||||
def get_audio(self, filename):
|
||||
audio, sampling_rate = load_wav_to_torch(filename)
|
||||
@@ -147,20 +147,22 @@ class TextAudioSpeakerLoader(torch.utils.data.Dataset):
|
||||
try:
|
||||
bert = torch.load(bert_path)
|
||||
assert bert.shape[-1] == len(phone)
|
||||
except:
|
||||
bert = get_bert(text, word2ph, language_str)
|
||||
torch.save(bert, bert_path)
|
||||
assert bert.shape[-1] == len(phone), phone
|
||||
except Exception as e:
|
||||
logger.warn("Bert load Failed")
|
||||
logger.warn(e)
|
||||
|
||||
if language_str == "ZH":
|
||||
bert = bert
|
||||
ja_bert = torch.zeros(768, len(phone))
|
||||
ja_bert = torch.zeros(1024, len(phone))
|
||||
en_bert = torch.zeros(1024, len(phone))
|
||||
elif language_str == "JP":
|
||||
bert = torch.zeros(1024, len(phone))
|
||||
ja_bert = bert
|
||||
en_bert = torch.zeros(1024, len(phone))
|
||||
elif language_str == "EN":
|
||||
bert = torch.zeros(1024, len(phone))
|
||||
else:
|
||||
bert = torch.zeros(1024, len(phone))
|
||||
ja_bert = torch.zeros(768, len(phone))
|
||||
ja_bert = torch.zeros(1024, len(phone))
|
||||
en_bert = bert
|
||||
assert bert.shape[-1] == len(phone), (
|
||||
bert.shape,
|
||||
len(phone),
|
||||
@@ -178,7 +180,7 @@ class TextAudioSpeakerLoader(torch.utils.data.Dataset):
|
||||
phone = torch.LongTensor(phone)
|
||||
tone = torch.LongTensor(tone)
|
||||
language = torch.LongTensor(language)
|
||||
return bert, ja_bert, phone, tone, language
|
||||
return bert, ja_bert, en_bert, phone, tone, language
|
||||
|
||||
def get_sid(self, sid):
|
||||
sid = torch.LongTensor([int(sid)])
|
||||
@@ -221,7 +223,8 @@ class TextAudioSpeakerCollate:
|
||||
tone_padded = torch.LongTensor(len(batch), max_text_len)
|
||||
language_padded = torch.LongTensor(len(batch), max_text_len)
|
||||
bert_padded = torch.FloatTensor(len(batch), 1024, max_text_len)
|
||||
ja_bert_padded = torch.FloatTensor(len(batch), 768, max_text_len)
|
||||
ja_bert_padded = torch.FloatTensor(len(batch), 1024, max_text_len)
|
||||
en_bert_padded = torch.FloatTensor(len(batch), 1024, max_text_len)
|
||||
|
||||
spec_padded = torch.FloatTensor(len(batch), batch[0][1].size(0), max_spec_len)
|
||||
wav_padded = torch.FloatTensor(len(batch), 1, max_wav_len)
|
||||
@@ -232,6 +235,8 @@ class TextAudioSpeakerCollate:
|
||||
wav_padded.zero_()
|
||||
bert_padded.zero_()
|
||||
ja_bert_padded.zero_()
|
||||
en_bert_padded.zero_()
|
||||
|
||||
for i in range(len(ids_sorted_decreasing)):
|
||||
row = batch[ids_sorted_decreasing[i]]
|
||||
|
||||
@@ -261,6 +266,9 @@ class TextAudioSpeakerCollate:
|
||||
ja_bert = row[7]
|
||||
ja_bert_padded[i, :, : ja_bert.size(1)] = ja_bert
|
||||
|
||||
en_bert = row[8]
|
||||
en_bert_padded[i, :, : en_bert.size(1)] = en_bert
|
||||
|
||||
return (
|
||||
text_padded,
|
||||
text_lengths,
|
||||
@@ -273,6 +281,7 @@ class TextAudioSpeakerCollate:
|
||||
language_padded,
|
||||
bert_padded,
|
||||
ja_bert_padded,
|
||||
en_bert_padded,
|
||||
)
|
||||
|
||||
|
||||
|
||||
150
default_config.yml
Normal file
150
default_config.yml
Normal file
@@ -0,0 +1,150 @@
|
||||
# 全局配置
|
||||
# 对于希望在同一时间使用多个配置文件的情况，例如两个GPU同时跑两个训练集：通过环境变量指定配置文件，不指定则默认为./config.yml
|
||||
|
||||
# 拟提供通用路径配置，统一存放数据，避免数据放得很乱
|
||||
# 每个数据集与其对应的模型存放至统一路径下，后续所有的路径配置均为相对于datasetPath的路径
|
||||
# 不填或者填空则路径为相对于项目根目录的路径
|
||||
dataset_path: "Data/你的数据集"
|
||||
|
||||
|
||||
# resample 音频重采样配置
|
||||
# 注意， “:” 后需要加空格
|
||||
resample:
|
||||
# 目标重采样率
|
||||
sampling_rate: 44100
|
||||
# 音频文件输入路径，重采样会将该路径下所有.wav音频文件重采样
|
||||
# 请填入相对于datasetPath的相对路径
|
||||
in_dir: "audios/raw" # 相对于根目录的路径为 /datasetPath/in_dir
|
||||
# 音频文件重采样后输出路径
|
||||
out_dir: "audios/wavs"
|
||||
|
||||
|
||||
# preprocess_text 数据集预处理相关配置
|
||||
# 注意， “:” 后需要加空格
|
||||
preprocess_text:
|
||||
# 原始文本文件路径，文本格式应为{wav_path}|{speaker_name}|{language}|{text}。
|
||||
transcription_path: "filelists/你的数据集文本.list"
|
||||
# 数据清洗后文本路径，可以不填。不填则将在原始文本目录生成
|
||||
cleaned_path: ""
|
||||
# 训练集路径
|
||||
train_path: "filelists/train.list"
|
||||
# 验证集路径
|
||||
val_path: "filelists/val.list"
|
||||
# 配置文件路径
|
||||
config_path: "config.json"
|
||||
# 每个speaker的验证集条数
|
||||
val_per_spk: 5
|
||||
# 验证集最大条数，多于的会被截断并放到训练集中
|
||||
max_val_total: 8
|
||||
# 是否进行数据清洗
|
||||
clean: true
|
||||
|
||||
|
||||
# bert_gen 相关配置
|
||||
# 注意， “:” 后需要加空格
|
||||
bert_gen:
|
||||
# 训练数据集配置文件路径
|
||||
config_path: "config.json"
|
||||
# 并行数
|
||||
num_processes: 2
|
||||
# 使用设备：可选项 "cuda" 显卡推理，"cpu" cpu推理
|
||||
# 该选项同时决定了get_bert_feature的默认设备
|
||||
device: "cuda"
|
||||
# 使用多卡推理
|
||||
use_multi_device: false
|
||||
|
||||
|
||||
# train 训练配置
|
||||
# 注意， “:” 后需要加空格
|
||||
train_ms:
|
||||
# 需要加载的环境变量，多显卡训练时RANK请手动在环境变量填写
|
||||
# 环境变量对应名称环境变量不存在时加载，也就是说手动添加的环境变量优先级更高，会覆盖本配置文件
|
||||
env:
|
||||
MASTER_ADDR: "localhost"
|
||||
MASTER_PORT: 10086
|
||||
WORLD_SIZE: 1
|
||||
RANK: 0
|
||||
# 可以填写任意名的环境变量
|
||||
THE_ENV_VAR_YOU_NEED_TO_USE: "1234567"
|
||||
# 训练模型存储目录：与旧版本的区别，原先数据集是存放在logs/model_name下的，现在改为统一存放在Data/你的数据集/models下
|
||||
model: "models"
|
||||
# 配置文件路径
|
||||
config_path: "config.json"
|
||||
|
||||
|
||||
# webui webui配置
|
||||
# 注意， “:” 后需要加空格
|
||||
webui:
|
||||
# 推理设备
|
||||
device: "cuda"
|
||||
# 模型路径
|
||||
model: "models/G_8000.pth"
|
||||
# 配置文件路径
|
||||
config_path: "config.json"
|
||||
# 端口号
|
||||
port: 7860
|
||||
# 是否公开部署，对外网开放
|
||||
share: false
|
||||
# 是否开启debug模式
|
||||
debug: false
|
||||
|
||||
|
||||
# server api配置
|
||||
# 注意， “:” 后需要加空格
|
||||
# 注意，本配置下的所有配置均为相对于根目录的路径
|
||||
server:
|
||||
# 端口号
|
||||
port: 5000
|
||||
# 模型默认使用设备：但是当前并没有实现这个配置。
|
||||
device: "cuda"
|
||||
# 需要加载的所有模型的配置
|
||||
# 注意，所有模型都必须正确配置model与config的路径，空路径会导致加载错误。
|
||||
models:
|
||||
- # 模型的路径
|
||||
model: ""
|
||||
# 模型config.json的路径
|
||||
config: ""
|
||||
# 模型使用设备，若填写则会覆盖默认配置
|
||||
device: "cuda"
|
||||
# 模型默认使用的语言
|
||||
language: "ZH"
|
||||
# 模型人物默认参数
|
||||
# 不必填写所有人物，不填的使用默认值
|
||||
# 暂时不用填写，当前尚未实现按人区分配置
|
||||
speakers:
|
||||
- speaker: "科比"
|
||||
sdp_ratio: 0.2
|
||||
noise_scale: 0.6
|
||||
noise_scale_w: 0.8
|
||||
length_scale: 1
|
||||
- speaker: "五条悟"
|
||||
sdp_ratio: 0.3
|
||||
noise_scale: 0.7
|
||||
noise_scale_w: 0.8
|
||||
length_scale: 0.5
|
||||
- speaker: "安倍晋三"
|
||||
sdp_ratio: 0.2
|
||||
noise_scale: 0.6
|
||||
noise_scale_w: 0.8
|
||||
length_scale: 1.2
|
||||
- # 模型的路径
|
||||
model: ""
|
||||
# 模型config.json的路径
|
||||
config: ""
|
||||
# 模型使用设备，若填写则会覆盖默认配置
|
||||
device: "cpu"
|
||||
# 模型默认使用的语言
|
||||
language: "JP"
|
||||
# 模型人物默认参数
|
||||
# 不必填写所有人物，不填的使用默认值
|
||||
speakers: [ ] # 也可以不填
|
||||
|
||||
|
||||
# 百度翻译开放平台 api配置
|
||||
# api接入文档 https://api.fanyi.baidu.com/doc/21
|
||||
# 请不要在github等网站公开分享你的app id 与 key
|
||||
translate:
|
||||
# 你的APPID
|
||||
"app_key": ""
|
||||
# 你的密钥
|
||||
"secret_key": ""
|
||||
28
emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim/.gitattributes
vendored
Normal file
28
emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim/.gitattributes
vendored
Normal file
@@ -0,0 +1,28 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zstandard filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
437
emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim/LICENSE
Normal file
437
emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim/LICENSE
Normal file
@@ -0,0 +1,437 @@
|
||||
Attribution-NonCommercial-ShareAlike 4.0 International
|
||||
|
||||
=======================================================================
|
||||
|
||||
Creative Commons Corporation ("Creative Commons") is not a law firm and
|
||||
does not provide legal services or legal advice. Distribution of
|
||||
Creative Commons public licenses does not create a lawyer-client or
|
||||
other relationship. Creative Commons makes its licenses and related
|
||||
information available on an "as-is" basis. Creative Commons gives no
|
||||
warranties regarding its licenses, any material licensed under their
|
||||
terms and conditions, or any related information. Creative Commons
|
||||
disclaims all liability for damages resulting from their use to the
|
||||
fullest extent possible.
|
||||
|
||||
Using Creative Commons Public Licenses
|
||||
|
||||
Creative Commons public licenses provide a standard set of terms and
|
||||
conditions that creators and other rights holders may use to share
|
||||
original works of authorship and other material subject to copyright
|
||||
and certain other rights specified in the public license below. The
|
||||
following considerations are for informational purposes only, are not
|
||||
exhaustive, and do not form part of our licenses.
|
||||
|
||||
Considerations for licensors: Our public licenses are
|
||||
intended for use by those authorized to give the public
|
||||
permission to use material in ways otherwise restricted by
|
||||
copyright and certain other rights. Our licenses are
|
||||
irrevocable. Licensors should read and understand the terms
|
||||
and conditions of the license they choose before applying it.
|
||||
Licensors should also secure all rights necessary before
|
||||
applying our licenses so that the public can reuse the
|
||||
material as expected. Licensors should clearly mark any
|
||||
material not subject to the license. This includes other CC-
|
||||
licensed material, or material used under an exception or
|
||||
limitation to copyright. More considerations for licensors:
|
||||
wiki.creativecommons.org/Considerations_for_licensors
|
||||
|
||||
Considerations for the public: By using one of our public
|
||||
licenses, a licensor grants the public permission to use the
|
||||
licensed material under specified terms and conditions. If
|
||||
the licensor's permission is not necessary for any reason--for
|
||||
example, because of any applicable exception or limitation to
|
||||
copyright--then that use is not regulated by the license. Our
|
||||
licenses grant only permissions under copyright and certain
|
||||
other rights that a licensor has authority to grant. Use of
|
||||
the licensed material may still be restricted for other
|
||||
reasons, including because others have copyright or other
|
||||
rights in the material. A licensor may make special requests,
|
||||
such as asking that all changes be marked or described.
|
||||
Although not required by our licenses, you are encouraged to
|
||||
respect those requests where reasonable. More considerations
|
||||
for the public:
|
||||
wiki.creativecommons.org/Considerations_for_licensees
|
||||
|
||||
=======================================================================
|
||||
|
||||
Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International
|
||||
Public License
|
||||
|
||||
By exercising the Licensed Rights (defined below), You accept and agree
|
||||
to be bound by the terms and conditions of this Creative Commons
|
||||
Attribution-NonCommercial-ShareAlike 4.0 International Public License
|
||||
("Public License"). To the extent this Public License may be
|
||||
interpreted as a contract, You are granted the Licensed Rights in
|
||||
consideration of Your acceptance of these terms and conditions, and the
|
||||
Licensor grants You such rights in consideration of benefits the
|
||||
Licensor receives from making the Licensed Material available under
|
||||
these terms and conditions.
|
||||
|
||||
|
||||
Section 1 -- Definitions.
|
||||
|
||||
a. Adapted Material means material subject to Copyright and Similar
|
||||
Rights that is derived from or based upon the Licensed Material
|
||||
and in which the Licensed Material is translated, altered,
|
||||
arranged, transformed, or otherwise modified in a manner requiring
|
||||
permission under the Copyright and Similar Rights held by the
|
||||
Licensor. For purposes of this Public License, where the Licensed
|
||||
Material is a musical work, performance, or sound recording,
|
||||
Adapted Material is always produced where the Licensed Material is
|
||||
synched in timed relation with a moving image.
|
||||
|
||||
b. Adapter's License means the license You apply to Your Copyright
|
||||
and Similar Rights in Your contributions to Adapted Material in
|
||||
accordance with the terms and conditions of this Public License.
|
||||
|
||||
c. BY-NC-SA Compatible License means a license listed at
|
||||
creativecommons.org/compatiblelicenses, approved by Creative
|
||||
Commons as essentially the equivalent of this Public License.
|
||||
|
||||
d. Copyright and Similar Rights means copyright and/or similar rights
|
||||
closely related to copyright including, without limitation,
|
||||
performance, broadcast, sound recording, and Sui Generis Database
|
||||
Rights, without regard to how the rights are labeled or
|
||||
categorized. For purposes of this Public License, the rights
|
||||
specified in Section 2(b)(1)-(2) are not Copyright and Similar
|
||||
Rights.
|
||||
|
||||
e. Effective Technological Measures means those measures that, in the
|
||||
absence of proper authority, may not be circumvented under laws
|
||||
fulfilling obligations under Article 11 of the WIPO Copyright
|
||||
Treaty adopted on December 20, 1996, and/or similar international
|
||||
agreements.
|
||||
|
||||
f. Exceptions and Limitations means fair use, fair dealing, and/or
|
||||
any other exception or limitation to Copyright and Similar Rights
|
||||
that applies to Your use of the Licensed Material.
|
||||
|
||||
g. License Elements means the license attributes listed in the name
|
||||
of a Creative Commons Public License. The License Elements of this
|
||||
Public License are Attribution, NonCommercial, and ShareAlike.
|
||||
|
||||
h. Licensed Material means the artistic or literary work, database,
|
||||
or other material to which the Licensor applied this Public
|
||||
License.
|
||||
|
||||
i. Licensed Rights means the rights granted to You subject to the
|
||||
terms and conditions of this Public License, which are limited to
|
||||
all Copyright and Similar Rights that apply to Your use of the
|
||||
Licensed Material and that the Licensor has authority to license.
|
||||
|
||||
j. Licensor means the individual(s) or entity(ies) granting rights
|
||||
under this Public License.
|
||||
|
||||
k. NonCommercial means not primarily intended for or directed towards
|
||||
commercial advantage or monetary compensation. For purposes of
|
||||
this Public License, the exchange of the Licensed Material for
|
||||
other material subject to Copyright and Similar Rights by digital
|
||||
file-sharing or similar means is NonCommercial provided there is
|
||||
no payment of monetary compensation in connection with the
|
||||
exchange.
|
||||
|
||||
l. Share means to provide material to the public by any means or
|
||||
process that requires permission under the Licensed Rights, such
|
||||
as reproduction, public display, public performance, distribution,
|
||||
dissemination, communication, or importation, and to make material
|
||||
available to the public including in ways that members of the
|
||||
public may access the material from a place and at a time
|
||||
individually chosen by them.
|
||||
|
||||
m. Sui Generis Database Rights means rights other than copyright
|
||||
resulting from Directive 96/9/EC of the European Parliament and of
|
||||
the Council of 11 March 1996 on the legal protection of databases,
|
||||
as amended and/or succeeded, as well as other essentially
|
||||
equivalent rights anywhere in the world.
|
||||
|
||||
n. You means the individual or entity exercising the Licensed Rights
|
||||
under this Public License. Your has a corresponding meaning.
|
||||
|
||||
|
||||
Section 2 -- Scope.
|
||||
|
||||
a. License grant.
|
||||
|
||||
1. Subject to the terms and conditions of this Public License,
|
||||
the Licensor hereby grants You a worldwide, royalty-free,
|
||||
non-sublicensable, non-exclusive, irrevocable license to
|
||||
exercise the Licensed Rights in the Licensed Material to:
|
||||
|
||||
a. reproduce and Share the Licensed Material, in whole or
|
||||
in part, for NonCommercial purposes only; and
|
||||
|
||||
b. produce, reproduce, and Share Adapted Material for
|
||||
NonCommercial purposes only.
|
||||
|
||||
2. Exceptions and Limitations. For the avoidance of doubt, where
|
||||
Exceptions and Limitations apply to Your use, this Public
|
||||
License does not apply, and You do not need to comply with
|
||||
its terms and conditions.
|
||||
|
||||
3. Term. The term of this Public License is specified in Section
|
||||
6(a).
|
||||
|
||||
4. Media and formats; technical modifications allowed. The
|
||||
Licensor authorizes You to exercise the Licensed Rights in
|
||||
all media and formats whether now known or hereafter created,
|
||||
and to make technical modifications necessary to do so. The
|
||||
Licensor waives and/or agrees not to assert any right or
|
||||
authority to forbid You from making technical modifications
|
||||
necessary to exercise the Licensed Rights, including
|
||||
technical modifications necessary to circumvent Effective
|
||||
Technological Measures. For purposes of this Public License,
|
||||
simply making modifications authorized by this Section 2(a)
|
||||
(4) never produces Adapted Material.
|
||||
|
||||
5. Downstream recipients.
|
||||
|
||||
a. Offer from the Licensor -- Licensed Material. Every
|
||||
recipient of the Licensed Material automatically
|
||||
receives an offer from the Licensor to exercise the
|
||||
Licensed Rights under the terms and conditions of this
|
||||
Public License.
|
||||
|
||||
b. Additional offer from the Licensor -- Adapted Material.
|
||||
Every recipient of Adapted Material from You
|
||||
automatically receives an offer from the Licensor to
|
||||
exercise the Licensed Rights in the Adapted Material
|
||||
under the conditions of the Adapter's License You apply.
|
||||
|
||||
c. No downstream restrictions. You may not offer or impose
|
||||
any additional or different terms or conditions on, or
|
||||
apply any Effective Technological Measures to, the
|
||||
Licensed Material if doing so restricts exercise of the
|
||||
Licensed Rights by any recipient of the Licensed
|
||||
Material.
|
||||
|
||||
6. No endorsement. Nothing in this Public License constitutes or
|
||||
may be construed as permission to assert or imply that You
|
||||
are, or that Your use of the Licensed Material is, connected
|
||||
with, or sponsored, endorsed, or granted official status by,
|
||||
the Licensor or others designated to receive attribution as
|
||||
provided in Section 3(a)(1)(A)(i).
|
||||
|
||||
b. Other rights.
|
||||
|
||||
1. Moral rights, such as the right of integrity, are not
|
||||
licensed under this Public License, nor are publicity,
|
||||
privacy, and/or other similar personality rights; however, to
|
||||
the extent possible, the Licensor waives and/or agrees not to
|
||||
assert any such rights held by the Licensor to the limited
|
||||
extent necessary to allow You to exercise the Licensed
|
||||
Rights, but not otherwise.
|
||||
|
||||
2. Patent and trademark rights are not licensed under this
|
||||
Public License.
|
||||
|
||||
3. To the extent possible, the Licensor waives any right to
|
||||
collect royalties from You for the exercise of the Licensed
|
||||
Rights, whether directly or through a collecting society
|
||||
under any voluntary or waivable statutory or compulsory
|
||||
licensing scheme. In all other cases the Licensor expressly
|
||||
reserves any right to collect such royalties, including when
|
||||
the Licensed Material is used other than for NonCommercial
|
||||
purposes.
|
||||
|
||||
|
||||
Section 3 -- License Conditions.
|
||||
|
||||
Your exercise of the Licensed Rights is expressly made subject to the
|
||||
following conditions.
|
||||
|
||||
a. Attribution.
|
||||
|
||||
1. If You Share the Licensed Material (including in modified
|
||||
form), You must:
|
||||
|
||||
a. retain the following if it is supplied by the Licensor
|
||||
with the Licensed Material:
|
||||
|
||||
i. identification of the creator(s) of the Licensed
|
||||
Material and any others designated to receive
|
||||
attribution, in any reasonable manner requested by
|
||||
the Licensor (including by pseudonym if
|
||||
designated);
|
||||
|
||||
ii. a copyright notice;
|
||||
|
||||
iii. a notice that refers to this Public License;
|
||||
|
||||
iv. a notice that refers to the disclaimer of
|
||||
warranties;
|
||||
|
||||
v. a URI or hyperlink to the Licensed Material to the
|
||||
extent reasonably practicable;
|
||||
|
||||
b. indicate if You modified the Licensed Material and
|
||||
retain an indication of any previous modifications; and
|
||||
|
||||
c. indicate the Licensed Material is licensed under this
|
||||
Public License, and include the text of, or the URI or
|
||||
hyperlink to, this Public License.
|
||||
|
||||
2. You may satisfy the conditions in Section 3(a)(1) in any
|
||||
reasonable manner based on the medium, means, and context in
|
||||
which You Share the Licensed Material. For example, it may be
|
||||
reasonable to satisfy the conditions by providing a URI or
|
||||
hyperlink to a resource that includes the required
|
||||
information.
|
||||
3. If requested by the Licensor, You must remove any of the
|
||||
information required by Section 3(a)(1)(A) to the extent
|
||||
reasonably practicable.
|
||||
|
||||
b. ShareAlike.
|
||||
|
||||
In addition to the conditions in Section 3(a), if You Share
|
||||
Adapted Material You produce, the following conditions also apply.
|
||||
|
||||
1. The Adapter's License You apply must be a Creative Commons
|
||||
license with the same License Elements, this version or
|
||||
later, or a BY-NC-SA Compatible License.
|
||||
|
||||
2. You must include the text of, or the URI or hyperlink to, the
|
||||
Adapter's License You apply. You may satisfy this condition
|
||||
in any reasonable manner based on the medium, means, and
|
||||
context in which You Share Adapted Material.
|
||||
|
||||
3. You may not offer or impose any additional or different terms
|
||||
or conditions on, or apply any Effective Technological
|
||||
Measures to, Adapted Material that restrict exercise of the
|
||||
rights granted under the Adapter's License You apply.
|
||||
|
||||
|
||||
Section 4 -- Sui Generis Database Rights.
|
||||
|
||||
Where the Licensed Rights include Sui Generis Database Rights that
|
||||
apply to Your use of the Licensed Material:
|
||||
|
||||
a. for the avoidance of doubt, Section 2(a)(1) grants You the right
|
||||
to extract, reuse, reproduce, and Share all or a substantial
|
||||
portion of the contents of the database for NonCommercial purposes
|
||||
only;
|
||||
|
||||
b. if You include all or a substantial portion of the database
|
||||
contents in a database in which You have Sui Generis Database
|
||||
Rights, then the database in which You have Sui Generis Database
|
||||
Rights (but not its individual contents) is Adapted Material,
|
||||
including for purposes of Section 3(b); and
|
||||
|
||||
c. You must comply with the conditions in Section 3(a) if You Share
|
||||
all or a substantial portion of the contents of the database.
|
||||
|
||||
For the avoidance of doubt, this Section 4 supplements and does not
|
||||
replace Your obligations under this Public License where the Licensed
|
||||
Rights include other Copyright and Similar Rights.
|
||||
|
||||
|
||||
Section 5 -- Disclaimer of Warranties and Limitation of Liability.
|
||||
|
||||
a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
|
||||
EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
|
||||
AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
|
||||
ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
|
||||
IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
|
||||
WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
|
||||
PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
|
||||
ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
|
||||
KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
|
||||
ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
|
||||
|
||||
b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
|
||||
TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
|
||||
NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
|
||||
INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
|
||||
COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
|
||||
USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
|
||||
ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
|
||||
DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
|
||||
IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
|
||||
|
||||
c. The disclaimer of warranties and limitation of liability provided
|
||||
above shall be interpreted in a manner that, to the extent
|
||||
possible, most closely approximates an absolute disclaimer and
|
||||
waiver of all liability.
|
||||
|
||||
|
||||
Section 6 -- Term and Termination.
|
||||
|
||||
a. This Public License applies for the term of the Copyright and
|
||||
Similar Rights licensed here. However, if You fail to comply with
|
||||
this Public License, then Your rights under this Public License
|
||||
terminate automatically.
|
||||
|
||||
b. Where Your right to use the Licensed Material has terminated under
|
||||
Section 6(a), it reinstates:
|
||||
|
||||
1. automatically as of the date the violation is cured, provided
|
||||
it is cured within 30 days of Your discovery of the
|
||||
violation; or
|
||||
|
||||
2. upon express reinstatement by the Licensor.
|
||||
|
||||
For the avoidance of doubt, this Section 6(b) does not affect any
|
||||
right the Licensor may have to seek remedies for Your violations
|
||||
of this Public License.
|
||||
|
||||
c. For the avoidance of doubt, the Licensor may also offer the
|
||||
Licensed Material under separate terms or conditions or stop
|
||||
distributing the Licensed Material at any time; however, doing so
|
||||
will not terminate this Public License.
|
||||
|
||||
d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
|
||||
License.
|
||||
|
||||
|
||||
Section 7 -- Other Terms and Conditions.
|
||||
|
||||
a. The Licensor shall not be bound by any additional or different
|
||||
terms or conditions communicated by You unless expressly agreed.
|
||||
|
||||
b. Any arrangements, understandings, or agreements regarding the
|
||||
Licensed Material not stated herein are separate from and
|
||||
independent of the terms and conditions of this Public License.
|
||||
|
||||
|
||||
Section 8 -- Interpretation.
|
||||
|
||||
a. For the avoidance of doubt, this Public License does not, and
|
||||
shall not be interpreted to, reduce, limit, restrict, or impose
|
||||
conditions on any use of the Licensed Material that could lawfully
|
||||
be made without permission under this Public License.
|
||||
|
||||
b. To the extent possible, if any provision of this Public License is
|
||||
deemed unenforceable, it shall be automatically reformed to the
|
||||
minimum extent necessary to make it enforceable. If the provision
|
||||
cannot be reformed, it shall be severed from this Public License
|
||||
without affecting the enforceability of the remaining terms and
|
||||
conditions.
|
||||
|
||||
c. No term or condition of this Public License will be waived and no
|
||||
failure to comply consented to unless expressly agreed to by the
|
||||
Licensor.
|
||||
|
||||
d. Nothing in this Public License constitutes or may be interpreted
|
||||
as a limitation upon, or waiver of, any privileges and immunities
|
||||
that apply to the Licensor or You, including from the legal
|
||||
processes of any jurisdiction or authority.
|
||||
|
||||
=======================================================================
|
||||
|
||||
Creative Commons is not a party to its public
|
||||
licenses. Notwithstanding, Creative Commons may elect to apply one of
|
||||
its public licenses to material it publishes and in those instances
|
||||
will be considered the “Licensor.” The text of the Creative Commons
|
||||
public licenses is dedicated to the public domain under the CC0 Public
|
||||
Domain Dedication. Except for the limited purpose of indicating that
|
||||
material is shared under a Creative Commons public license or as
|
||||
otherwise permitted by the Creative Commons policies published at
|
||||
creativecommons.org/policies, Creative Commons does not authorize the
|
||||
use of the trademark "Creative Commons" or any other trademark or logo
|
||||
of Creative Commons without its prior written consent including,
|
||||
without limitation, in connection with any unauthorized modifications
|
||||
to any of its public licenses or any other arrangements,
|
||||
understandings, or agreements concerning use of licensed material. For
|
||||
the avoidance of doubt, this paragraph does not form part of the
|
||||
public licenses.
|
||||
|
||||
Creative Commons may be contacted at creativecommons.org.
|
||||
127
emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim/README.md
Normal file
127
emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim/README.md
Normal file
@@ -0,0 +1,127 @@
|
||||
---
|
||||
language: en
|
||||
datasets:
|
||||
- msp-podcast
|
||||
inference: true
|
||||
tags:
|
||||
- speech
|
||||
- audio
|
||||
- wav2vec2
|
||||
- audio-classification
|
||||
- emotion-recognition
|
||||
license: cc-by-nc-sa-4.0
|
||||
pipeline_tag: audio-classification
|
||||
---
|
||||
|
||||
# Model for Dimensional Speech Emotion Recognition based on Wav2vec 2.0
|
||||
|
||||
The model expects a raw audio signal as input and outputs predictions for arousal, dominance and valence in a range of approximately 0...1. In addition, it also provides the pooled states of the last transformer layer. The model was created by fine-tuning [
|
||||
Wav2Vec2-Large-Robust](https://huggingface.co/facebook/wav2vec2-large-robust) on [MSP-Podcast](https://ecs.utdallas.edu/research/researchlabs/msp-lab/MSP-Podcast.html) (v1.7). The model was pruned from 24 to 12 transformer layers before fine-tuning. An [ONNX](https://onnx.ai/") export of the model is available from [doi:10.5281/zenodo.6221127](https://zenodo.org/record/6221127). Further details are given in the associated [paper](https://arxiv.org/abs/2203.07378) and [tutorial](https://github.com/audeering/w2v2-how-to).
|
||||
|
||||
# Usage
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
from transformers import Wav2Vec2Processor
|
||||
from transformers.models.wav2vec2.modeling_wav2vec2 import (
|
||||
Wav2Vec2Model,
|
||||
Wav2Vec2PreTrainedModel,
|
||||
)
|
||||
|
||||
|
||||
class RegressionHead(nn.Module):
|
||||
r"""Classification head."""
|
||||
|
||||
def __init__(self, config):
|
||||
|
||||
super().__init__()
|
||||
|
||||
self.dense = nn.Linear(config.hidden_size, config.hidden_size)
|
||||
self.dropout = nn.Dropout(config.final_dropout)
|
||||
self.out_proj = nn.Linear(config.hidden_size, config.num_labels)
|
||||
|
||||
def forward(self, features, **kwargs):
|
||||
|
||||
x = features
|
||||
x = self.dropout(x)
|
||||
x = self.dense(x)
|
||||
x = torch.tanh(x)
|
||||
x = self.dropout(x)
|
||||
x = self.out_proj(x)
|
||||
|
||||
return x
|
||||
|
||||
|
||||
class EmotionModel(Wav2Vec2PreTrainedModel):
|
||||
r"""Speech emotion classifier."""
|
||||
|
||||
def __init__(self, config):
|
||||
|
||||
super().__init__(config)
|
||||
|
||||
self.config = config
|
||||
self.wav2vec2 = Wav2Vec2Model(config)
|
||||
self.classifier = RegressionHead(config)
|
||||
self.init_weights()
|
||||
|
||||
def forward(
|
||||
self,
|
||||
input_values,
|
||||
):
|
||||
|
||||
outputs = self.wav2vec2(input_values)
|
||||
hidden_states = outputs[0]
|
||||
hidden_states = torch.mean(hidden_states, dim=1)
|
||||
logits = self.classifier(hidden_states)
|
||||
|
||||
return hidden_states, logits
|
||||
|
||||
|
||||
|
||||
# load model from hub
|
||||
device = 'cpu'
|
||||
model_name = 'audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim'
|
||||
processor = Wav2Vec2Processor.from_pretrained(model_name)
|
||||
model = EmotionModel.from_pretrained(model_name)
|
||||
|
||||
# dummy signal
|
||||
sampling_rate = 16000
|
||||
signal = np.zeros((1, sampling_rate), dtype=np.float32)
|
||||
|
||||
|
||||
def process_func(
|
||||
x: np.ndarray,
|
||||
sampling_rate: int,
|
||||
embeddings: bool = False,
|
||||
) -> np.ndarray:
|
||||
r"""Predict emotions or extract embeddings from raw audio signal."""
|
||||
|
||||
# run through processor to normalize signal
|
||||
# always returns a batch, so we just get the first entry
|
||||
# then we put it on the device
|
||||
y = processor(x, sampling_rate=sampling_rate)
|
||||
y = y['input_values'][0]
|
||||
y = y.reshape(1, -1)
|
||||
y = torch.from_numpy(y).to(device)
|
||||
|
||||
# run through model
|
||||
with torch.no_grad():
|
||||
y = model(y)[0 if embeddings else 1]
|
||||
|
||||
# convert to numpy
|
||||
y = y.detach().cpu().numpy()
|
||||
|
||||
return y
|
||||
|
||||
|
||||
print(process_func(signal, sampling_rate))
|
||||
# Arousal dominance valence
|
||||
# [[0.5460754 0.6062266 0.40431657]]
|
||||
|
||||
print(process_func(signal, sampling_rate, embeddings=True))
|
||||
# Pooled hidden states of last transformer layer
|
||||
# [[-0.00752167 0.0065819 -0.00746342 ... 0.00663632 0.00848748
|
||||
# 0.00599211]]
|
||||
```
|
||||
@@ -0,0 +1,122 @@
|
||||
{
|
||||
"_name_or_path": "torch",
|
||||
"activation_dropout": 0.1,
|
||||
"adapter_kernel_size": 3,
|
||||
"adapter_stride": 2,
|
||||
"add_adapter": false,
|
||||
"apply_spec_augment": true,
|
||||
"architectures": [
|
||||
"Wav2Vec2ForSpeechClassification"
|
||||
],
|
||||
"attention_dropout": 0.1,
|
||||
"bos_token_id": 1,
|
||||
"classifier_proj_size": 256,
|
||||
"codevector_dim": 768,
|
||||
"contrastive_logits_temperature": 0.1,
|
||||
"conv_bias": true,
|
||||
"conv_dim": [
|
||||
512,
|
||||
512,
|
||||
512,
|
||||
512,
|
||||
512,
|
||||
512,
|
||||
512
|
||||
],
|
||||
"conv_kernel": [
|
||||
10,
|
||||
3,
|
||||
3,
|
||||
3,
|
||||
3,
|
||||
2,
|
||||
2
|
||||
],
|
||||
"conv_stride": [
|
||||
5,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
],
|
||||
"ctc_loss_reduction": "sum",
|
||||
"ctc_zero_infinity": false,
|
||||
"diversity_loss_weight": 0.1,
|
||||
"do_stable_layer_norm": true,
|
||||
"eos_token_id": 2,
|
||||
"feat_extract_activation": "gelu",
|
||||
"feat_extract_dropout": 0.0,
|
||||
"feat_extract_norm": "layer",
|
||||
"feat_proj_dropout": 0.1,
|
||||
"feat_quantizer_dropout": 0.0,
|
||||
"final_dropout": 0.1,
|
||||
"finetuning_task": "wav2vec2_reg",
|
||||
"gradient_checkpointing": false,
|
||||
"hidden_act": "gelu",
|
||||
"hidden_dropout": 0.1,
|
||||
"hidden_dropout_prob": 0.1,
|
||||
"hidden_size": 1024,
|
||||
"id2label": {
|
||||
"0": "arousal",
|
||||
"1": "dominance",
|
||||
"2": "valence"
|
||||
},
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 4096,
|
||||
"label2id": {
|
||||
"arousal": 0,
|
||||
"dominance": 1,
|
||||
"valence": 2
|
||||
},
|
||||
"layer_norm_eps": 1e-05,
|
||||
"layerdrop": 0.1,
|
||||
"mask_feature_length": 10,
|
||||
"mask_feature_min_masks": 0,
|
||||
"mask_feature_prob": 0.0,
|
||||
"mask_time_length": 10,
|
||||
"mask_time_min_masks": 2,
|
||||
"mask_time_prob": 0.05,
|
||||
"model_type": "wav2vec2",
|
||||
"num_adapter_layers": 3,
|
||||
"num_attention_heads": 16,
|
||||
"num_codevector_groups": 2,
|
||||
"num_codevectors_per_group": 320,
|
||||
"num_conv_pos_embedding_groups": 16,
|
||||
"num_conv_pos_embeddings": 128,
|
||||
"num_feat_extract_layers": 7,
|
||||
"num_hidden_layers": 12,
|
||||
"num_negatives": 100,
|
||||
"output_hidden_size": 1024,
|
||||
"pad_token_id": 0,
|
||||
"pooling_mode": "mean",
|
||||
"problem_type": "regression",
|
||||
"proj_codevector_dim": 768,
|
||||
"tdnn_dilation": [
|
||||
1,
|
||||
2,
|
||||
3,
|
||||
1,
|
||||
1
|
||||
],
|
||||
"tdnn_dim": [
|
||||
512,
|
||||
512,
|
||||
512,
|
||||
512,
|
||||
1500
|
||||
],
|
||||
"tdnn_kernel": [
|
||||
5,
|
||||
3,
|
||||
3,
|
||||
1,
|
||||
1
|
||||
],
|
||||
"torch_dtype": "float32",
|
||||
"transformers_version": "4.17.0.dev0",
|
||||
"use_weighted_layer_sum": false,
|
||||
"vocab_size": null,
|
||||
"xvector_output_dim": 512
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"do_normalize": true,
|
||||
"feature_extractor_type": "Wav2Vec2FeatureExtractor",
|
||||
"feature_size": 1,
|
||||
"padding_side": "right",
|
||||
"padding_value": 0.0,
|
||||
"return_attention_mask": true,
|
||||
"sampling_rate": 16000
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{}
|
||||
BIN
img/yuyu.png
Normal file
BIN
img/yuyu.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 241 KiB |
BIN
img/参数说明.png
Normal file
BIN
img/参数说明.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 202 KiB |
BIN
img/宵宫.png
Normal file
BIN
img/宵宫.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 337 KiB |
BIN
img/微信图片_20231010105112.png
Normal file
BIN
img/微信图片_20231010105112.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 232 KiB |
BIN
img/神里绫华.png
Normal file
BIN
img/神里绫华.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 280 KiB |
BIN
img/纳西妲.png
Normal file
BIN
img/纳西妲.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 310 KiB |
206
infer.py
Normal file
206
infer.py
Normal file
@@ -0,0 +1,206 @@
|
||||
"""
|
||||
版本管理、兼容推理及模型加载实现。
|
||||
版本说明：
|
||||
1. 版本号与github的release版本号对应，使用哪个release版本训练的模型即对应其版本号
|
||||
2. 请在模型的config.json中显示声明版本号，添加一个字段"version" : "你的版本号"
|
||||
特殊版本说明：
|
||||
1.1.1-fix： 1.1.1版本训练的模型，但是在推理时使用dev的日语修复
|
||||
1.1.1-dev： dev开发
|
||||
2.0：当前版本
|
||||
"""
|
||||
import torch
|
||||
import commons
|
||||
from text import cleaned_text_to_sequence, get_bert
|
||||
from text.cleaner import clean_text
|
||||
import utils
|
||||
|
||||
from models import SynthesizerTrn
|
||||
from text.symbols import symbols
|
||||
from oldVersion.V111.models import SynthesizerTrn as V111SynthesizerTrn
|
||||
from oldVersion.V111.text import symbols as V111symbols
|
||||
from oldVersion.V110.models import SynthesizerTrn as V110SynthesizerTrn
|
||||
from oldVersion.V110.text import symbols as V110symbols
|
||||
from oldVersion.V101.models import SynthesizerTrn as V101SynthesizerTrn
|
||||
from oldVersion.V101.text import symbols as V101symbols
|
||||
|
||||
from oldVersion import V111, V110, V101
|
||||
|
||||
# 当前版本信息
|
||||
latest_version = "2.0"
|
||||
|
||||
# 版本兼容
|
||||
SynthesizerTrnMap = {
|
||||
"1.1.1-fix": V111SynthesizerTrn,
|
||||
"1.1.1": V111SynthesizerTrn,
|
||||
"1.1": V110SynthesizerTrn,
|
||||
"1.1.0": V110SynthesizerTrn,
|
||||
"1.0.1": V101SynthesizerTrn,
|
||||
"1.0": V101SynthesizerTrn,
|
||||
"1.0.0": V101SynthesizerTrn,
|
||||
}
|
||||
|
||||
symbolsMap = {
|
||||
"1.1.1-fix": V111symbols,
|
||||
"1.1.1": V111symbols,
|
||||
"1.1": V110symbols,
|
||||
"1.1.0": V110symbols,
|
||||
"1.0.1": V101symbols,
|
||||
"1.0": V101symbols,
|
||||
"1.0.0": V101symbols,
|
||||
}
|
||||
|
||||
|
||||
def get_net_g(model_path: str, version: str, device: str, hps):
|
||||
if version != latest_version:
|
||||
net_g = SynthesizerTrnMap[version](
|
||||
len(symbolsMap[version]),
|
||||
hps.data.filter_length // 2 + 1,
|
||||
hps.train.segment_size // hps.data.hop_length,
|
||||
n_speakers=hps.data.n_speakers,
|
||||
**hps.model,
|
||||
).to(device)
|
||||
else:
|
||||
# 当前版本模型 net_g
|
||||
net_g = SynthesizerTrn(
|
||||
len(symbols),
|
||||
hps.data.filter_length // 2 + 1,
|
||||
hps.train.segment_size // hps.data.hop_length,
|
||||
n_speakers=hps.data.n_speakers,
|
||||
**hps.model,
|
||||
).to(device)
|
||||
_ = net_g.eval()
|
||||
_ = utils.load_checkpoint(model_path, net_g, None, skip_optimizer=True)
|
||||
return net_g
|
||||
|
||||
|
||||
def get_text(text, language_str, hps, device):
|
||||
# 在此处实现当前版本的get_text
|
||||
norm_text, phone, tone, word2ph = clean_text(text, language_str)
|
||||
phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
|
||||
|
||||
if hps.data.add_blank:
|
||||
phone = commons.intersperse(phone, 0)
|
||||
tone = commons.intersperse(tone, 0)
|
||||
language = commons.intersperse(language, 0)
|
||||
for i in range(len(word2ph)):
|
||||
word2ph[i] = word2ph[i] * 2
|
||||
word2ph[0] += 1
|
||||
bert = get_bert(norm_text, word2ph, language_str, device)
|
||||
del word2ph
|
||||
assert bert.shape[-1] == len(phone), phone
|
||||
|
||||
if language_str == "ZH":
|
||||
bert = bert
|
||||
ja_bert = torch.zeros(1024, len(phone))
|
||||
en_bert = torch.zeros(1024, len(phone))
|
||||
elif language_str == "JP":
|
||||
bert = torch.zeros(1024, len(phone))
|
||||
ja_bert = bert
|
||||
en_bert = torch.zeros(1024, len(phone))
|
||||
elif language_str == "EN":
|
||||
bert = torch.zeros(1024, len(phone))
|
||||
ja_bert = torch.zeros(1024, len(phone))
|
||||
en_bert = bert
|
||||
else:
|
||||
raise ValueError("language_str should be ZH, JP or EN")
|
||||
|
||||
assert bert.shape[-1] == len(
|
||||
phone
|
||||
), f"Bert seq len {bert.shape[-1]} != {len(phone)}"
|
||||
|
||||
phone = torch.LongTensor(phone)
|
||||
tone = torch.LongTensor(tone)
|
||||
language = torch.LongTensor(language)
|
||||
return bert, ja_bert, en_bert, phone, tone, language
|
||||
|
||||
|
||||
def infer(
|
||||
text,
|
||||
sdp_ratio,
|
||||
noise_scale,
|
||||
noise_scale_w,
|
||||
length_scale,
|
||||
sid,
|
||||
language,
|
||||
hps,
|
||||
net_g,
|
||||
device,
|
||||
):
|
||||
# 支持中日双语版本
|
||||
inferMap_V2 = {
|
||||
"1.1.1-fix": V111.infer_fix,
|
||||
"1.1.1": V111.infer,
|
||||
"1.1": V110.infer,
|
||||
"1.1.0": V110.infer,
|
||||
}
|
||||
# 仅支持中文版本
|
||||
# 在测试中，并未发现两个版本的模型不能互相通用
|
||||
inferMap_V1 = {
|
||||
"1.0.1": V101.infer,
|
||||
"1.0": V101.infer,
|
||||
"1.0.0": V101.infer,
|
||||
}
|
||||
version = hps.version if hasattr(hps, "version") else latest_version
|
||||
# 非当前版本，根据版本号选择合适的infer
|
||||
if version != latest_version:
|
||||
if version in inferMap_V2.keys():
|
||||
return inferMap_V2[version](
|
||||
text,
|
||||
sdp_ratio,
|
||||
noise_scale,
|
||||
noise_scale_w,
|
||||
length_scale,
|
||||
sid,
|
||||
language,
|
||||
hps,
|
||||
net_g,
|
||||
device,
|
||||
)
|
||||
if version in inferMap_V1.keys():
|
||||
return inferMap_V1[version](
|
||||
text,
|
||||
sdp_ratio,
|
||||
noise_scale,
|
||||
noise_scale_w,
|
||||
length_scale,
|
||||
sid,
|
||||
hps,
|
||||
net_g,
|
||||
device,
|
||||
)
|
||||
# 在此处实现当前版本的推理
|
||||
bert, ja_bert, en_bert, phones, tones, lang_ids = get_text(
|
||||
text, language, hps, device
|
||||
)
|
||||
with torch.no_grad():
|
||||
x_tst = phones.to(device).unsqueeze(0)
|
||||
tones = tones.to(device).unsqueeze(0)
|
||||
lang_ids = lang_ids.to(device).unsqueeze(0)
|
||||
bert = bert.to(device).unsqueeze(0)
|
||||
ja_bert = ja_bert.to(device).unsqueeze(0)
|
||||
en_bert = en_bert.to(device).unsqueeze(0)
|
||||
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
|
||||
del phones
|
||||
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
|
||||
audio = (
|
||||
net_g.infer(
|
||||
x_tst,
|
||||
x_tst_lengths,
|
||||
speakers,
|
||||
tones,
|
||||
lang_ids,
|
||||
bert,
|
||||
ja_bert,
|
||||
en_bert,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise_scale,
|
||||
noise_scale_w=noise_scale_w,
|
||||
length_scale=length_scale,
|
||||
)[0][0, 0]
|
||||
.data.cpu()
|
||||
.float()
|
||||
.numpy()
|
||||
)
|
||||
del x_tst, tones, lang_ids, bert, x_tst_lengths, speakers
|
||||
torch.cuda.empty_cache()
|
||||
return audio
|
||||
28
models.py
28
models.py
@@ -340,7 +340,8 @@ class TextEncoder(nn.Module):
|
||||
self.language_emb = nn.Embedding(num_languages, hidden_channels)
|
||||
nn.init.normal_(self.language_emb.weight, 0.0, hidden_channels**-0.5)
|
||||
self.bert_proj = nn.Conv1d(1024, hidden_channels, 1)
|
||||
self.ja_bert_proj = nn.Conv1d(768, hidden_channels, 1)
|
||||
self.ja_bert_proj = nn.Conv1d(1024, hidden_channels, 1)
|
||||
self.en_bert_proj = nn.Conv1d(1024, hidden_channels, 1)
|
||||
|
||||
self.encoder = attentions.Encoder(
|
||||
hidden_channels,
|
||||
@@ -353,15 +354,17 @@ class TextEncoder(nn.Module):
|
||||
)
|
||||
self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
|
||||
|
||||
def forward(self, x, x_lengths, tone, language, bert, ja_bert, g=None):
|
||||
def forward(self, x, x_lengths, tone, language, bert, ja_bert, en_bert, g=None):
|
||||
bert_emb = self.bert_proj(bert).transpose(1, 2)
|
||||
ja_bert_emb = self.ja_bert_proj(ja_bert).transpose(1, 2)
|
||||
en_bert_emb = self.en_bert_proj(en_bert).transpose(1, 2)
|
||||
x = (
|
||||
self.emb(x)
|
||||
+ self.tone_emb(tone)
|
||||
+ self.language_emb(language)
|
||||
+ bert_emb
|
||||
+ ja_bert_emb
|
||||
+ en_bert_emb
|
||||
) * math.sqrt(
|
||||
self.hidden_channels
|
||||
) # [b, t, h]
|
||||
@@ -763,7 +766,7 @@ class SynthesizerTrn(nn.Module):
|
||||
gin_channels=256,
|
||||
use_sdp=True,
|
||||
n_flow_layer=4,
|
||||
n_layers_trans_flow=6,
|
||||
n_layers_trans_flow=4,
|
||||
flow_share_parameter=False,
|
||||
use_transformer_flow=True,
|
||||
**kwargs
|
||||
@@ -862,13 +865,25 @@ class SynthesizerTrn(nn.Module):
|
||||
else:
|
||||
self.ref_enc = ReferenceEncoder(spec_channels, gin_channels)
|
||||
|
||||
def forward(self, x, x_lengths, y, y_lengths, sid, tone, language, bert, ja_bert):
|
||||
def forward(
|
||||
self,
|
||||
x,
|
||||
x_lengths,
|
||||
y,
|
||||
y_lengths,
|
||||
sid,
|
||||
tone,
|
||||
language,
|
||||
bert,
|
||||
ja_bert,
|
||||
en_bert,
|
||||
):
|
||||
if self.n_speakers > 0:
|
||||
g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
|
||||
else:
|
||||
g = self.ref_enc(y.transpose(1, 2)).unsqueeze(-1)
|
||||
x, m_p, logs_p, x_mask = self.enc_p(
|
||||
x, x_lengths, tone, language, bert, ja_bert, g=g
|
||||
x, x_lengths, tone, language, bert, ja_bert, en_bert, g=g
|
||||
)
|
||||
z, m_q, logs_q, y_mask = self.enc_q(y, y_lengths, g=g)
|
||||
z_p = self.flow(z, y_mask, g=g)
|
||||
@@ -945,6 +960,7 @@ class SynthesizerTrn(nn.Module):
|
||||
language,
|
||||
bert,
|
||||
ja_bert,
|
||||
en_bert,
|
||||
noise_scale=0.667,
|
||||
length_scale=1,
|
||||
noise_scale_w=0.8,
|
||||
@@ -959,7 +975,7 @@ class SynthesizerTrn(nn.Module):
|
||||
else:
|
||||
g = self.ref_enc(y.transpose(1, 2)).unsqueeze(-1)
|
||||
x, m_p, logs_p, x_mask = self.enc_p(
|
||||
x, x_lengths, tone, language, bert, ja_bert, g=g
|
||||
x, x_lengths, tone, language, bert, ja_bert, en_bert, g=g
|
||||
)
|
||||
logw = self.sdp(x, x_mask, g=g, reverse=True, noise_scale=noise_scale_w) * (
|
||||
sdp_ratio
|
||||
|
||||
73
oldVersion/V101/__init__.py
Normal file
73
oldVersion/V101/__init__.py
Normal file
@@ -0,0 +1,73 @@
|
||||
"""
|
||||
1.0.1 版本兼容
|
||||
https://github.com/fishaudio/Bert-VITS2/releases/tag/1.0.1
|
||||
"""
|
||||
import torch
|
||||
import commons
|
||||
from .text.cleaner import clean_text
|
||||
from .text import cleaned_text_to_sequence
|
||||
from oldVersion.V111.text import get_bert
|
||||
|
||||
|
||||
def get_text(text, language_str, hps, device):
|
||||
norm_text, phone, tone, word2ph = clean_text(text, language_str)
|
||||
phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
|
||||
|
||||
if hps.data.add_blank:
|
||||
phone = commons.intersperse(phone, 0)
|
||||
tone = commons.intersperse(tone, 0)
|
||||
language = commons.intersperse(language, 0)
|
||||
for i in range(len(word2ph)):
|
||||
word2ph[i] = word2ph[i] * 2
|
||||
word2ph[0] += 1
|
||||
bert = get_bert(norm_text, word2ph, language_str, device)
|
||||
del word2ph
|
||||
|
||||
assert bert.shape[-1] == len(phone)
|
||||
|
||||
phone = torch.LongTensor(phone)
|
||||
tone = torch.LongTensor(tone)
|
||||
language = torch.LongTensor(language)
|
||||
|
||||
return bert, phone, tone, language
|
||||
|
||||
|
||||
def infer(
|
||||
text,
|
||||
sdp_ratio,
|
||||
noise_scale,
|
||||
noise_scale_w,
|
||||
length_scale,
|
||||
sid,
|
||||
hps,
|
||||
net_g,
|
||||
device,
|
||||
):
|
||||
bert, phones, tones, lang_ids = get_text(text, "ZH", hps, device)
|
||||
with torch.no_grad():
|
||||
x_tst = phones.to(device).unsqueeze(0)
|
||||
tones = tones.to(device).unsqueeze(0)
|
||||
lang_ids = lang_ids.to(device).unsqueeze(0)
|
||||
bert = bert.to(device).unsqueeze(0)
|
||||
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
|
||||
del phones
|
||||
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
|
||||
audio = (
|
||||
net_g.infer(
|
||||
x_tst,
|
||||
x_tst_lengths,
|
||||
speakers,
|
||||
tones,
|
||||
lang_ids,
|
||||
bert,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise_scale,
|
||||
noise_scale_w=noise_scale_w,
|
||||
length_scale=length_scale,
|
||||
)[0][0, 0]
|
||||
.data.cpu()
|
||||
.float()
|
||||
.numpy()
|
||||
)
|
||||
del x_tst, tones, lang_ids, bert, x_tst_lengths, speakers
|
||||
return audio
|
||||
977
oldVersion/V101/models.py
Normal file
977
oldVersion/V101/models.py
Normal file
@@ -0,0 +1,977 @@
|
||||
import math
|
||||
import torch
|
||||
from torch import nn
|
||||
from torch.nn import functional as F
|
||||
|
||||
import commons
|
||||
import modules
|
||||
import attentions
|
||||
import monotonic_align
|
||||
|
||||
from torch.nn import Conv1d, ConvTranspose1d, Conv2d
|
||||
from torch.nn.utils import weight_norm, remove_weight_norm, spectral_norm
|
||||
|
||||
from commons import init_weights, get_padding
|
||||
from .text import symbols, num_tones, num_languages
|
||||
|
||||
|
||||
class DurationDiscriminator(nn.Module): # vits2
|
||||
def __init__(
|
||||
self, in_channels, filter_channels, kernel_size, p_dropout, gin_channels=0
|
||||
):
|
||||
super().__init__()
|
||||
|
||||
self.in_channels = in_channels
|
||||
self.filter_channels = filter_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.p_dropout = p_dropout
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.drop = nn.Dropout(p_dropout)
|
||||
self.conv_1 = nn.Conv1d(
|
||||
in_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.norm_1 = modules.LayerNorm(filter_channels)
|
||||
self.conv_2 = nn.Conv1d(
|
||||
filter_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.norm_2 = modules.LayerNorm(filter_channels)
|
||||
self.dur_proj = nn.Conv1d(1, filter_channels, 1)
|
||||
|
||||
self.pre_out_conv_1 = nn.Conv1d(
|
||||
2 * filter_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.pre_out_norm_1 = modules.LayerNorm(filter_channels)
|
||||
self.pre_out_conv_2 = nn.Conv1d(
|
||||
filter_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.pre_out_norm_2 = modules.LayerNorm(filter_channels)
|
||||
|
||||
if gin_channels != 0:
|
||||
self.cond = nn.Conv1d(gin_channels, in_channels, 1)
|
||||
|
||||
self.output_layer = nn.Sequential(nn.Linear(filter_channels, 1), nn.Sigmoid())
|
||||
|
||||
def forward_probability(self, x, x_mask, dur, g=None):
|
||||
dur = self.dur_proj(dur)
|
||||
x = torch.cat([x, dur], dim=1)
|
||||
x = self.pre_out_conv_1(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.pre_out_norm_1(x)
|
||||
x = self.drop(x)
|
||||
x = self.pre_out_conv_2(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.pre_out_norm_2(x)
|
||||
x = self.drop(x)
|
||||
x = x * x_mask
|
||||
x = x.transpose(1, 2)
|
||||
output_prob = self.output_layer(x)
|
||||
return output_prob
|
||||
|
||||
def forward(self, x, x_mask, dur_r, dur_hat, g=None):
|
||||
x = torch.detach(x)
|
||||
if g is not None:
|
||||
g = torch.detach(g)
|
||||
x = x + self.cond(g)
|
||||
x = self.conv_1(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.norm_1(x)
|
||||
x = self.drop(x)
|
||||
x = self.conv_2(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.norm_2(x)
|
||||
x = self.drop(x)
|
||||
|
||||
output_probs = []
|
||||
for dur in [dur_r, dur_hat]:
|
||||
output_prob = self.forward_probability(x, x_mask, dur, g)
|
||||
output_probs.append(output_prob)
|
||||
|
||||
return output_probs
|
||||
|
||||
|
||||
class TransformerCouplingBlock(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
channels,
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
n_flows=4,
|
||||
gin_channels=0,
|
||||
share_parameter=False,
|
||||
):
|
||||
super().__init__()
|
||||
self.channels = channels
|
||||
self.hidden_channels = hidden_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.n_layers = n_layers
|
||||
self.n_flows = n_flows
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.flows = nn.ModuleList()
|
||||
|
||||
self.wn = (
|
||||
attentions.FFT(
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
isflow=True,
|
||||
gin_channels=self.gin_channels,
|
||||
)
|
||||
if share_parameter
|
||||
else None
|
||||
)
|
||||
|
||||
for i in range(n_flows):
|
||||
self.flows.append(
|
||||
modules.TransformerCouplingLayer(
|
||||
channels,
|
||||
hidden_channels,
|
||||
kernel_size,
|
||||
n_layers,
|
||||
n_heads,
|
||||
p_dropout,
|
||||
filter_channels,
|
||||
mean_only=True,
|
||||
wn_sharing_parameter=self.wn,
|
||||
gin_channels=self.gin_channels,
|
||||
)
|
||||
)
|
||||
self.flows.append(modules.Flip())
|
||||
|
||||
def forward(self, x, x_mask, g=None, reverse=False):
|
||||
if not reverse:
|
||||
for flow in self.flows:
|
||||
x, _ = flow(x, x_mask, g=g, reverse=reverse)
|
||||
else:
|
||||
for flow in reversed(self.flows):
|
||||
x = flow(x, x_mask, g=g, reverse=reverse)
|
||||
return x
|
||||
|
||||
|
||||
class StochasticDurationPredictor(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
in_channels,
|
||||
filter_channels,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
n_flows=4,
|
||||
gin_channels=0,
|
||||
):
|
||||
super().__init__()
|
||||
filter_channels = in_channels # it needs to be removed from future version.
|
||||
self.in_channels = in_channels
|
||||
self.filter_channels = filter_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.p_dropout = p_dropout
|
||||
self.n_flows = n_flows
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.log_flow = modules.Log()
|
||||
self.flows = nn.ModuleList()
|
||||
self.flows.append(modules.ElementwiseAffine(2))
|
||||
for i in range(n_flows):
|
||||
self.flows.append(
|
||||
modules.ConvFlow(2, filter_channels, kernel_size, n_layers=3)
|
||||
)
|
||||
self.flows.append(modules.Flip())
|
||||
|
||||
self.post_pre = nn.Conv1d(1, filter_channels, 1)
|
||||
self.post_proj = nn.Conv1d(filter_channels, filter_channels, 1)
|
||||
self.post_convs = modules.DDSConv(
|
||||
filter_channels, kernel_size, n_layers=3, p_dropout=p_dropout
|
||||
)
|
||||
self.post_flows = nn.ModuleList()
|
||||
self.post_flows.append(modules.ElementwiseAffine(2))
|
||||
for i in range(4):
|
||||
self.post_flows.append(
|
||||
modules.ConvFlow(2, filter_channels, kernel_size, n_layers=3)
|
||||
)
|
||||
self.post_flows.append(modules.Flip())
|
||||
|
||||
self.pre = nn.Conv1d(in_channels, filter_channels, 1)
|
||||
self.proj = nn.Conv1d(filter_channels, filter_channels, 1)
|
||||
self.convs = modules.DDSConv(
|
||||
filter_channels, kernel_size, n_layers=3, p_dropout=p_dropout
|
||||
)
|
||||
if gin_channels != 0:
|
||||
self.cond = nn.Conv1d(gin_channels, filter_channels, 1)
|
||||
|
||||
def forward(self, x, x_mask, w=None, g=None, reverse=False, noise_scale=1.0):
|
||||
x = torch.detach(x)
|
||||
x = self.pre(x)
|
||||
if g is not None:
|
||||
g = torch.detach(g)
|
||||
x = x + self.cond(g)
|
||||
x = self.convs(x, x_mask)
|
||||
x = self.proj(x) * x_mask
|
||||
|
||||
if not reverse:
|
||||
flows = self.flows
|
||||
assert w is not None
|
||||
|
||||
logdet_tot_q = 0
|
||||
h_w = self.post_pre(w)
|
||||
h_w = self.post_convs(h_w, x_mask)
|
||||
h_w = self.post_proj(h_w) * x_mask
|
||||
e_q = (
|
||||
torch.randn(w.size(0), 2, w.size(2)).to(device=x.device, dtype=x.dtype)
|
||||
* x_mask
|
||||
)
|
||||
z_q = e_q
|
||||
for flow in self.post_flows:
|
||||
z_q, logdet_q = flow(z_q, x_mask, g=(x + h_w))
|
||||
logdet_tot_q += logdet_q
|
||||
z_u, z1 = torch.split(z_q, [1, 1], 1)
|
||||
u = torch.sigmoid(z_u) * x_mask
|
||||
z0 = (w - u) * x_mask
|
||||
logdet_tot_q += torch.sum(
|
||||
(F.logsigmoid(z_u) + F.logsigmoid(-z_u)) * x_mask, [1, 2]
|
||||
)
|
||||
logq = (
|
||||
torch.sum(-0.5 * (math.log(2 * math.pi) + (e_q**2)) * x_mask, [1, 2])
|
||||
- logdet_tot_q
|
||||
)
|
||||
|
||||
logdet_tot = 0
|
||||
z0, logdet = self.log_flow(z0, x_mask)
|
||||
logdet_tot += logdet
|
||||
z = torch.cat([z0, z1], 1)
|
||||
for flow in flows:
|
||||
z, logdet = flow(z, x_mask, g=x, reverse=reverse)
|
||||
logdet_tot = logdet_tot + logdet
|
||||
nll = (
|
||||
torch.sum(0.5 * (math.log(2 * math.pi) + (z**2)) * x_mask, [1, 2])
|
||||
- logdet_tot
|
||||
)
|
||||
return nll + logq # [b]
|
||||
else:
|
||||
flows = list(reversed(self.flows))
|
||||
flows = flows[:-2] + [flows[-1]] # remove a useless vflow
|
||||
z = (
|
||||
torch.randn(x.size(0), 2, x.size(2)).to(device=x.device, dtype=x.dtype)
|
||||
* noise_scale
|
||||
)
|
||||
for flow in flows:
|
||||
z = flow(z, x_mask, g=x, reverse=reverse)
|
||||
z0, z1 = torch.split(z, [1, 1], 1)
|
||||
logw = z0
|
||||
return logw
|
||||
|
||||
|
||||
class DurationPredictor(nn.Module):
|
||||
def __init__(
|
||||
self, in_channels, filter_channels, kernel_size, p_dropout, gin_channels=0
|
||||
):
|
||||
super().__init__()
|
||||
|
||||
self.in_channels = in_channels
|
||||
self.filter_channels = filter_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.p_dropout = p_dropout
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.drop = nn.Dropout(p_dropout)
|
||||
self.conv_1 = nn.Conv1d(
|
||||
in_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.norm_1 = modules.LayerNorm(filter_channels)
|
||||
self.conv_2 = nn.Conv1d(
|
||||
filter_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.norm_2 = modules.LayerNorm(filter_channels)
|
||||
self.proj = nn.Conv1d(filter_channels, 1, 1)
|
||||
|
||||
if gin_channels != 0:
|
||||
self.cond = nn.Conv1d(gin_channels, in_channels, 1)
|
||||
|
||||
def forward(self, x, x_mask, g=None):
|
||||
x = torch.detach(x)
|
||||
if g is not None:
|
||||
g = torch.detach(g)
|
||||
x = x + self.cond(g)
|
||||
x = self.conv_1(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.norm_1(x)
|
||||
x = self.drop(x)
|
||||
x = self.conv_2(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.norm_2(x)
|
||||
x = self.drop(x)
|
||||
x = self.proj(x * x_mask)
|
||||
return x * x_mask
|
||||
|
||||
|
||||
class TextEncoder(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
n_vocab,
|
||||
out_channels,
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
gin_channels=0,
|
||||
):
|
||||
super().__init__()
|
||||
self.n_vocab = n_vocab
|
||||
self.out_channels = out_channels
|
||||
self.hidden_channels = hidden_channels
|
||||
self.filter_channels = filter_channels
|
||||
self.n_heads = n_heads
|
||||
self.n_layers = n_layers
|
||||
self.kernel_size = kernel_size
|
||||
self.p_dropout = p_dropout
|
||||
self.gin_channels = gin_channels
|
||||
self.emb = nn.Embedding(len(symbols), hidden_channels)
|
||||
nn.init.normal_(self.emb.weight, 0.0, hidden_channels**-0.5)
|
||||
self.tone_emb = nn.Embedding(num_tones, hidden_channels)
|
||||
nn.init.normal_(self.tone_emb.weight, 0.0, hidden_channels**-0.5)
|
||||
self.language_emb = nn.Embedding(num_languages, hidden_channels)
|
||||
nn.init.normal_(self.language_emb.weight, 0.0, hidden_channels**-0.5)
|
||||
self.bert_proj = nn.Conv1d(1024, hidden_channels, 1)
|
||||
|
||||
self.encoder = attentions.Encoder(
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
gin_channels=self.gin_channels,
|
||||
)
|
||||
self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
|
||||
|
||||
def forward(self, x, x_lengths, tone, language, bert, g=None):
|
||||
x = (
|
||||
self.emb(x)
|
||||
+ self.tone_emb(tone)
|
||||
+ self.language_emb(language)
|
||||
+ self.bert_proj(bert).transpose(1, 2)
|
||||
) * math.sqrt(
|
||||
self.hidden_channels
|
||||
) # [b, t, h]
|
||||
x = torch.transpose(x, 1, -1) # [b, h, t]
|
||||
x_mask = torch.unsqueeze(commons.sequence_mask(x_lengths, x.size(2)), 1).to(
|
||||
x.dtype
|
||||
)
|
||||
|
||||
x = self.encoder(x * x_mask, x_mask, g=g)
|
||||
stats = self.proj(x) * x_mask
|
||||
|
||||
m, logs = torch.split(stats, self.out_channels, dim=1)
|
||||
return x, m, logs, x_mask
|
||||
|
||||
|
||||
class ResidualCouplingBlock(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
channels,
|
||||
hidden_channels,
|
||||
kernel_size,
|
||||
dilation_rate,
|
||||
n_layers,
|
||||
n_flows=4,
|
||||
gin_channels=0,
|
||||
):
|
||||
super().__init__()
|
||||
self.channels = channels
|
||||
self.hidden_channels = hidden_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.dilation_rate = dilation_rate
|
||||
self.n_layers = n_layers
|
||||
self.n_flows = n_flows
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.flows = nn.ModuleList()
|
||||
for i in range(n_flows):
|
||||
self.flows.append(
|
||||
modules.ResidualCouplingLayer(
|
||||
channels,
|
||||
hidden_channels,
|
||||
kernel_size,
|
||||
dilation_rate,
|
||||
n_layers,
|
||||
gin_channels=gin_channels,
|
||||
mean_only=True,
|
||||
)
|
||||
)
|
||||
self.flows.append(modules.Flip())
|
||||
|
||||
def forward(self, x, x_mask, g=None, reverse=False):
|
||||
if not reverse:
|
||||
for flow in self.flows:
|
||||
x, _ = flow(x, x_mask, g=g, reverse=reverse)
|
||||
else:
|
||||
for flow in reversed(self.flows):
|
||||
x = flow(x, x_mask, g=g, reverse=reverse)
|
||||
return x
|
||||
|
||||
|
||||
class PosteriorEncoder(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
in_channels,
|
||||
out_channels,
|
||||
hidden_channels,
|
||||
kernel_size,
|
||||
dilation_rate,
|
||||
n_layers,
|
||||
gin_channels=0,
|
||||
):
|
||||
super().__init__()
|
||||
self.in_channels = in_channels
|
||||
self.out_channels = out_channels
|
||||
self.hidden_channels = hidden_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.dilation_rate = dilation_rate
|
||||
self.n_layers = n_layers
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.pre = nn.Conv1d(in_channels, hidden_channels, 1)
|
||||
self.enc = modules.WN(
|
||||
hidden_channels,
|
||||
kernel_size,
|
||||
dilation_rate,
|
||||
n_layers,
|
||||
gin_channels=gin_channels,
|
||||
)
|
||||
self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
|
||||
|
||||
def forward(self, x, x_lengths, g=None):
|
||||
x_mask = torch.unsqueeze(commons.sequence_mask(x_lengths, x.size(2)), 1).to(
|
||||
x.dtype
|
||||
)
|
||||
x = self.pre(x) * x_mask
|
||||
x = self.enc(x, x_mask, g=g)
|
||||
stats = self.proj(x) * x_mask
|
||||
m, logs = torch.split(stats, self.out_channels, dim=1)
|
||||
z = (m + torch.randn_like(m) * torch.exp(logs)) * x_mask
|
||||
return z, m, logs, x_mask
|
||||
|
||||
|
||||
class Generator(torch.nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
initial_channel,
|
||||
resblock,
|
||||
resblock_kernel_sizes,
|
||||
resblock_dilation_sizes,
|
||||
upsample_rates,
|
||||
upsample_initial_channel,
|
||||
upsample_kernel_sizes,
|
||||
gin_channels=0,
|
||||
):
|
||||
super(Generator, self).__init__()
|
||||
self.num_kernels = len(resblock_kernel_sizes)
|
||||
self.num_upsamples = len(upsample_rates)
|
||||
self.conv_pre = Conv1d(
|
||||
initial_channel, upsample_initial_channel, 7, 1, padding=3
|
||||
)
|
||||
resblock = modules.ResBlock1 if resblock == "1" else modules.ResBlock2
|
||||
|
||||
self.ups = nn.ModuleList()
|
||||
for i, (u, k) in enumerate(zip(upsample_rates, upsample_kernel_sizes)):
|
||||
self.ups.append(
|
||||
weight_norm(
|
||||
ConvTranspose1d(
|
||||
upsample_initial_channel // (2**i),
|
||||
upsample_initial_channel // (2 ** (i + 1)),
|
||||
k,
|
||||
u,
|
||||
padding=(k - u) // 2,
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
self.resblocks = nn.ModuleList()
|
||||
for i in range(len(self.ups)):
|
||||
ch = upsample_initial_channel // (2 ** (i + 1))
|
||||
for j, (k, d) in enumerate(
|
||||
zip(resblock_kernel_sizes, resblock_dilation_sizes)
|
||||
):
|
||||
self.resblocks.append(resblock(ch, k, d))
|
||||
|
||||
self.conv_post = Conv1d(ch, 1, 7, 1, padding=3, bias=False)
|
||||
self.ups.apply(init_weights)
|
||||
|
||||
if gin_channels != 0:
|
||||
self.cond = nn.Conv1d(gin_channels, upsample_initial_channel, 1)
|
||||
|
||||
def forward(self, x, g=None):
|
||||
x = self.conv_pre(x)
|
||||
if g is not None:
|
||||
x = x + self.cond(g)
|
||||
|
||||
for i in range(self.num_upsamples):
|
||||
x = F.leaky_relu(x, modules.LRELU_SLOPE)
|
||||
x = self.ups[i](x)
|
||||
xs = None
|
||||
for j in range(self.num_kernels):
|
||||
if xs is None:
|
||||
xs = self.resblocks[i * self.num_kernels + j](x)
|
||||
else:
|
||||
xs += self.resblocks[i * self.num_kernels + j](x)
|
||||
x = xs / self.num_kernels
|
||||
x = F.leaky_relu(x)
|
||||
x = self.conv_post(x)
|
||||
x = torch.tanh(x)
|
||||
|
||||
return x
|
||||
|
||||
def remove_weight_norm(self):
|
||||
print("Removing weight norm...")
|
||||
for l in self.ups:
|
||||
remove_weight_norm(l)
|
||||
for l in self.resblocks:
|
||||
l.remove_weight_norm()
|
||||
|
||||
|
||||
class DiscriminatorP(torch.nn.Module):
|
||||
def __init__(self, period, kernel_size=5, stride=3, use_spectral_norm=False):
|
||||
super(DiscriminatorP, self).__init__()
|
||||
self.period = period
|
||||
self.use_spectral_norm = use_spectral_norm
|
||||
norm_f = weight_norm if use_spectral_norm is False else spectral_norm
|
||||
self.convs = nn.ModuleList(
|
||||
[
|
||||
norm_f(
|
||||
Conv2d(
|
||||
1,
|
||||
32,
|
||||
(kernel_size, 1),
|
||||
(stride, 1),
|
||||
padding=(get_padding(kernel_size, 1), 0),
|
||||
)
|
||||
),
|
||||
norm_f(
|
||||
Conv2d(
|
||||
32,
|
||||
128,
|
||||
(kernel_size, 1),
|
||||
(stride, 1),
|
||||
padding=(get_padding(kernel_size, 1), 0),
|
||||
)
|
||||
),
|
||||
norm_f(
|
||||
Conv2d(
|
||||
128,
|
||||
512,
|
||||
(kernel_size, 1),
|
||||
(stride, 1),
|
||||
padding=(get_padding(kernel_size, 1), 0),
|
||||
)
|
||||
),
|
||||
norm_f(
|
||||
Conv2d(
|
||||
512,
|
||||
1024,
|
||||
(kernel_size, 1),
|
||||
(stride, 1),
|
||||
padding=(get_padding(kernel_size, 1), 0),
|
||||
)
|
||||
),
|
||||
norm_f(
|
||||
Conv2d(
|
||||
1024,
|
||||
1024,
|
||||
(kernel_size, 1),
|
||||
1,
|
||||
padding=(get_padding(kernel_size, 1), 0),
|
||||
)
|
||||
),
|
||||
]
|
||||
)
|
||||
self.conv_post = norm_f(Conv2d(1024, 1, (3, 1), 1, padding=(1, 0)))
|
||||
|
||||
def forward(self, x):
|
||||
fmap = []
|
||||
|
||||
# 1d to 2d
|
||||
b, c, t = x.shape
|
||||
if t % self.period != 0: # pad first
|
||||
n_pad = self.period - (t % self.period)
|
||||
x = F.pad(x, (0, n_pad), "reflect")
|
||||
t = t + n_pad
|
||||
x = x.view(b, c, t // self.period, self.period)
|
||||
|
||||
for l in self.convs:
|
||||
x = l(x)
|
||||
x = F.leaky_relu(x, modules.LRELU_SLOPE)
|
||||
fmap.append(x)
|
||||
x = self.conv_post(x)
|
||||
fmap.append(x)
|
||||
x = torch.flatten(x, 1, -1)
|
||||
|
||||
return x, fmap
|
||||
|
||||
|
||||
class DiscriminatorS(torch.nn.Module):
|
||||
def __init__(self, use_spectral_norm=False):
|
||||
super(DiscriminatorS, self).__init__()
|
||||
norm_f = weight_norm if use_spectral_norm is False else spectral_norm
|
||||
self.convs = nn.ModuleList(
|
||||
[
|
||||
norm_f(Conv1d(1, 16, 15, 1, padding=7)),
|
||||
norm_f(Conv1d(16, 64, 41, 4, groups=4, padding=20)),
|
||||
norm_f(Conv1d(64, 256, 41, 4, groups=16, padding=20)),
|
||||
norm_f(Conv1d(256, 1024, 41, 4, groups=64, padding=20)),
|
||||
norm_f(Conv1d(1024, 1024, 41, 4, groups=256, padding=20)),
|
||||
norm_f(Conv1d(1024, 1024, 5, 1, padding=2)),
|
||||
]
|
||||
)
|
||||
self.conv_post = norm_f(Conv1d(1024, 1, 3, 1, padding=1))
|
||||
|
||||
def forward(self, x):
|
||||
fmap = []
|
||||
|
||||
for l in self.convs:
|
||||
x = l(x)
|
||||
x = F.leaky_relu(x, modules.LRELU_SLOPE)
|
||||
fmap.append(x)
|
||||
x = self.conv_post(x)
|
||||
fmap.append(x)
|
||||
x = torch.flatten(x, 1, -1)
|
||||
|
||||
return x, fmap
|
||||
|
||||
|
||||
class MultiPeriodDiscriminator(torch.nn.Module):
|
||||
def __init__(self, use_spectral_norm=False):
|
||||
super(MultiPeriodDiscriminator, self).__init__()
|
||||
periods = [2, 3, 5, 7, 11]
|
||||
|
||||
discs = [DiscriminatorS(use_spectral_norm=use_spectral_norm)]
|
||||
discs = discs + [
|
||||
DiscriminatorP(i, use_spectral_norm=use_spectral_norm) for i in periods
|
||||
]
|
||||
self.discriminators = nn.ModuleList(discs)
|
||||
|
||||
def forward(self, y, y_hat):
|
||||
y_d_rs = []
|
||||
y_d_gs = []
|
||||
fmap_rs = []
|
||||
fmap_gs = []
|
||||
for i, d in enumerate(self.discriminators):
|
||||
y_d_r, fmap_r = d(y)
|
||||
y_d_g, fmap_g = d(y_hat)
|
||||
y_d_rs.append(y_d_r)
|
||||
y_d_gs.append(y_d_g)
|
||||
fmap_rs.append(fmap_r)
|
||||
fmap_gs.append(fmap_g)
|
||||
|
||||
return y_d_rs, y_d_gs, fmap_rs, fmap_gs
|
||||
|
||||
|
||||
class ReferenceEncoder(nn.Module):
|
||||
"""
|
||||
inputs --- [N, Ty/r, n_mels*r] mels
|
||||
outputs --- [N, ref_enc_gru_size]
|
||||
"""
|
||||
|
||||
def __init__(self, spec_channels, gin_channels=0):
|
||||
super().__init__()
|
||||
self.spec_channels = spec_channels
|
||||
ref_enc_filters = [32, 32, 64, 64, 128, 128]
|
||||
K = len(ref_enc_filters)
|
||||
filters = [1] + ref_enc_filters
|
||||
convs = [
|
||||
weight_norm(
|
||||
nn.Conv2d(
|
||||
in_channels=filters[i],
|
||||
out_channels=filters[i + 1],
|
||||
kernel_size=(3, 3),
|
||||
stride=(2, 2),
|
||||
padding=(1, 1),
|
||||
)
|
||||
)
|
||||
for i in range(K)
|
||||
]
|
||||
self.convs = nn.ModuleList(convs)
|
||||
# self.wns = nn.ModuleList([weight_norm(num_features=ref_enc_filters[i]) for i in range(K)])
|
||||
|
||||
out_channels = self.calculate_channels(spec_channels, 3, 2, 1, K)
|
||||
self.gru = nn.GRU(
|
||||
input_size=ref_enc_filters[-1] * out_channels,
|
||||
hidden_size=256 // 2,
|
||||
batch_first=True,
|
||||
)
|
||||
self.proj = nn.Linear(128, gin_channels)
|
||||
|
||||
def forward(self, inputs, mask=None):
|
||||
N = inputs.size(0)
|
||||
out = inputs.view(N, 1, -1, self.spec_channels) # [N, 1, Ty, n_freqs]
|
||||
for conv in self.convs:
|
||||
out = conv(out)
|
||||
# out = wn(out)
|
||||
out = F.relu(out) # [N, 128, Ty//2^K, n_mels//2^K]
|
||||
|
||||
out = out.transpose(1, 2) # [N, Ty//2^K, 128, n_mels//2^K]
|
||||
T = out.size(1)
|
||||
N = out.size(0)
|
||||
out = out.contiguous().view(N, T, -1) # [N, Ty//2^K, 128*n_mels//2^K]
|
||||
|
||||
self.gru.flatten_parameters()
|
||||
memory, out = self.gru(out) # out --- [1, N, 128]
|
||||
|
||||
return self.proj(out.squeeze(0))
|
||||
|
||||
def calculate_channels(self, L, kernel_size, stride, pad, n_convs):
|
||||
for i in range(n_convs):
|
||||
L = (L - kernel_size + 2 * pad) // stride + 1
|
||||
return L
|
||||
|
||||
|
||||
class SynthesizerTrn(nn.Module):
|
||||
"""
|
||||
Synthesizer for Training
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
n_vocab,
|
||||
spec_channels,
|
||||
segment_size,
|
||||
inter_channels,
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
resblock,
|
||||
resblock_kernel_sizes,
|
||||
resblock_dilation_sizes,
|
||||
upsample_rates,
|
||||
upsample_initial_channel,
|
||||
upsample_kernel_sizes,
|
||||
n_speakers=256,
|
||||
gin_channels=256,
|
||||
use_sdp=True,
|
||||
n_flow_layer=4,
|
||||
n_layers_trans_flow=3,
|
||||
flow_share_parameter=False,
|
||||
use_transformer_flow=True,
|
||||
**kwargs
|
||||
):
|
||||
super().__init__()
|
||||
self.n_vocab = n_vocab
|
||||
self.spec_channels = spec_channels
|
||||
self.inter_channels = inter_channels
|
||||
self.hidden_channels = hidden_channels
|
||||
self.filter_channels = filter_channels
|
||||
self.n_heads = n_heads
|
||||
self.n_layers = n_layers
|
||||
self.kernel_size = kernel_size
|
||||
self.p_dropout = p_dropout
|
||||
self.resblock = resblock
|
||||
self.resblock_kernel_sizes = resblock_kernel_sizes
|
||||
self.resblock_dilation_sizes = resblock_dilation_sizes
|
||||
self.upsample_rates = upsample_rates
|
||||
self.upsample_initial_channel = upsample_initial_channel
|
||||
self.upsample_kernel_sizes = upsample_kernel_sizes
|
||||
self.segment_size = segment_size
|
||||
self.n_speakers = n_speakers
|
||||
self.gin_channels = gin_channels
|
||||
self.n_layers_trans_flow = n_layers_trans_flow
|
||||
self.use_spk_conditioned_encoder = kwargs.get(
|
||||
"use_spk_conditioned_encoder", True
|
||||
)
|
||||
self.use_sdp = use_sdp
|
||||
self.use_noise_scaled_mas = kwargs.get("use_noise_scaled_mas", False)
|
||||
self.mas_noise_scale_initial = kwargs.get("mas_noise_scale_initial", 0.01)
|
||||
self.noise_scale_delta = kwargs.get("noise_scale_delta", 2e-6)
|
||||
self.current_mas_noise_scale = self.mas_noise_scale_initial
|
||||
if self.use_spk_conditioned_encoder and gin_channels > 0:
|
||||
self.enc_gin_channels = gin_channels
|
||||
self.enc_p = TextEncoder(
|
||||
n_vocab,
|
||||
inter_channels,
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
gin_channels=self.enc_gin_channels,
|
||||
)
|
||||
self.dec = Generator(
|
||||
inter_channels,
|
||||
resblock,
|
||||
resblock_kernel_sizes,
|
||||
resblock_dilation_sizes,
|
||||
upsample_rates,
|
||||
upsample_initial_channel,
|
||||
upsample_kernel_sizes,
|
||||
gin_channels=gin_channels,
|
||||
)
|
||||
self.enc_q = PosteriorEncoder(
|
||||
spec_channels,
|
||||
inter_channels,
|
||||
hidden_channels,
|
||||
5,
|
||||
1,
|
||||
16,
|
||||
gin_channels=gin_channels,
|
||||
)
|
||||
if use_transformer_flow:
|
||||
self.flow = TransformerCouplingBlock(
|
||||
inter_channels,
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers_trans_flow,
|
||||
5,
|
||||
p_dropout,
|
||||
n_flow_layer,
|
||||
gin_channels=gin_channels,
|
||||
share_parameter=flow_share_parameter,
|
||||
)
|
||||
else:
|
||||
self.flow = ResidualCouplingBlock(
|
||||
inter_channels,
|
||||
hidden_channels,
|
||||
5,
|
||||
1,
|
||||
n_flow_layer,
|
||||
gin_channels=gin_channels,
|
||||
)
|
||||
self.sdp = StochasticDurationPredictor(
|
||||
hidden_channels, 192, 3, 0.5, 4, gin_channels=gin_channels
|
||||
)
|
||||
self.dp = DurationPredictor(
|
||||
hidden_channels, 256, 3, 0.5, gin_channels=gin_channels
|
||||
)
|
||||
|
||||
if n_speakers > 0:
|
||||
self.emb_g = nn.Embedding(n_speakers, gin_channels)
|
||||
else:
|
||||
self.ref_enc = ReferenceEncoder(spec_channels, gin_channels)
|
||||
|
||||
def forward(self, x, x_lengths, y, y_lengths, sid, tone, language, bert):
|
||||
if self.n_speakers >= 0:
|
||||
g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
|
||||
else:
|
||||
g = self.ref_enc(y.transpose(1, 2)).unsqueeze(-1)
|
||||
x, m_p, logs_p, x_mask = self.enc_p(x, x_lengths, tone, language, bert, g=g)
|
||||
z, m_q, logs_q, y_mask = self.enc_q(y, y_lengths, g=g)
|
||||
z_p = self.flow(z, y_mask, g=g)
|
||||
|
||||
with torch.no_grad():
|
||||
# negative cross-entropy
|
||||
s_p_sq_r = torch.exp(-2 * logs_p) # [b, d, t]
|
||||
neg_cent1 = torch.sum(
|
||||
-0.5 * math.log(2 * math.pi) - logs_p, [1], keepdim=True
|
||||
) # [b, 1, t_s]
|
||||
neg_cent2 = torch.matmul(
|
||||
-0.5 * (z_p**2).transpose(1, 2), s_p_sq_r
|
||||
) # [b, t_t, d] x [b, d, t_s] = [b, t_t, t_s]
|
||||
neg_cent3 = torch.matmul(
|
||||
z_p.transpose(1, 2), (m_p * s_p_sq_r)
|
||||
) # [b, t_t, d] x [b, d, t_s] = [b, t_t, t_s]
|
||||
neg_cent4 = torch.sum(
|
||||
-0.5 * (m_p**2) * s_p_sq_r, [1], keepdim=True
|
||||
) # [b, 1, t_s]
|
||||
neg_cent = neg_cent1 + neg_cent2 + neg_cent3 + neg_cent4
|
||||
if self.use_noise_scaled_mas:
|
||||
epsilon = (
|
||||
torch.std(neg_cent)
|
||||
* torch.randn_like(neg_cent)
|
||||
* self.current_mas_noise_scale
|
||||
)
|
||||
neg_cent = neg_cent + epsilon
|
||||
|
||||
attn_mask = torch.unsqueeze(x_mask, 2) * torch.unsqueeze(y_mask, -1)
|
||||
attn = (
|
||||
monotonic_align.maximum_path(neg_cent, attn_mask.squeeze(1))
|
||||
.unsqueeze(1)
|
||||
.detach()
|
||||
)
|
||||
|
||||
w = attn.sum(2)
|
||||
|
||||
l_length_sdp = self.sdp(x, x_mask, w, g=g)
|
||||
l_length_sdp = l_length_sdp / torch.sum(x_mask)
|
||||
|
||||
logw_ = torch.log(w + 1e-6) * x_mask
|
||||
logw = self.dp(x, x_mask, g=g)
|
||||
l_length_dp = torch.sum((logw - logw_) ** 2, [1, 2]) / torch.sum(
|
||||
x_mask
|
||||
) # for averaging
|
||||
|
||||
l_length = l_length_dp + l_length_sdp
|
||||
|
||||
# expand prior
|
||||
m_p = torch.matmul(attn.squeeze(1), m_p.transpose(1, 2)).transpose(1, 2)
|
||||
logs_p = torch.matmul(attn.squeeze(1), logs_p.transpose(1, 2)).transpose(1, 2)
|
||||
|
||||
z_slice, ids_slice = commons.rand_slice_segments(
|
||||
z, y_lengths, self.segment_size
|
||||
)
|
||||
o = self.dec(z_slice, g=g)
|
||||
return (
|
||||
o,
|
||||
l_length,
|
||||
attn,
|
||||
ids_slice,
|
||||
x_mask,
|
||||
y_mask,
|
||||
(z, z_p, m_p, logs_p, m_q, logs_q),
|
||||
(x, logw, logw_),
|
||||
)
|
||||
|
||||
def infer(
|
||||
self,
|
||||
x,
|
||||
x_lengths,
|
||||
sid,
|
||||
tone,
|
||||
language,
|
||||
bert,
|
||||
noise_scale=0.667,
|
||||
length_scale=1,
|
||||
noise_scale_w=0.8,
|
||||
max_len=None,
|
||||
sdp_ratio=0,
|
||||
y=None,
|
||||
):
|
||||
# x, m_p, logs_p, x_mask = self.enc_p(x, x_lengths, tone, language, bert)
|
||||
# g = self.gst(y)
|
||||
if self.n_speakers > 0:
|
||||
g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
|
||||
else:
|
||||
g = self.ref_enc(y.transpose(1, 2)).unsqueeze(-1)
|
||||
x, m_p, logs_p, x_mask = self.enc_p(x, x_lengths, tone, language, bert, g=g)
|
||||
logw = self.sdp(x, x_mask, g=g, reverse=True, noise_scale=noise_scale_w) * (
|
||||
sdp_ratio
|
||||
) + self.dp(x, x_mask, g=g) * (1 - sdp_ratio)
|
||||
w = torch.exp(logw) * x_mask * length_scale
|
||||
w_ceil = torch.ceil(w)
|
||||
y_lengths = torch.clamp_min(torch.sum(w_ceil, [1, 2]), 1).long()
|
||||
y_mask = torch.unsqueeze(commons.sequence_mask(y_lengths, None), 1).to(
|
||||
x_mask.dtype
|
||||
)
|
||||
attn_mask = torch.unsqueeze(x_mask, 2) * torch.unsqueeze(y_mask, -1)
|
||||
attn = commons.generate_path(w_ceil, attn_mask)
|
||||
|
||||
m_p = torch.matmul(attn.squeeze(1), m_p.transpose(1, 2)).transpose(
|
||||
1, 2
|
||||
) # [b, t', t], [b, t, d] -> [b, d, t']
|
||||
logs_p = torch.matmul(attn.squeeze(1), logs_p.transpose(1, 2)).transpose(
|
||||
1, 2
|
||||
) # [b, t', t], [b, t, d] -> [b, d, t']
|
||||
|
||||
z_p = m_p + torch.randn_like(m_p) * torch.exp(logs_p) * noise_scale
|
||||
z = self.flow(z_p, y_mask, g=g, reverse=True)
|
||||
o = self.dec((z * y_mask)[:, :, :max_len], g=g)
|
||||
return o, attn, y_mask, (z, z_p, m_p, logs_p)
|
||||
28
oldVersion/V101/text/__init__.py
Normal file
28
oldVersion/V101/text/__init__.py
Normal file
@@ -0,0 +1,28 @@
|
||||
from .symbols import *
|
||||
|
||||
|
||||
_symbol_to_id = {s: i for i, s in enumerate(symbols)}
|
||||
|
||||
|
||||
def cleaned_text_to_sequence(cleaned_text, tones, language):
|
||||
"""Converts a string of text to a sequence of IDs corresponding to the symbols in the text.
|
||||
Args:
|
||||
text: string to convert to a sequence
|
||||
Returns:
|
||||
List of integers corresponding to the symbols in the text
|
||||
"""
|
||||
phones = [_symbol_to_id[symbol] for symbol in cleaned_text]
|
||||
tone_start = language_tone_start_map[language]
|
||||
tones = [i + tone_start for i in tones]
|
||||
lang_id = language_id_map[language]
|
||||
lang_ids = [lang_id for i in phones]
|
||||
return phones, tones, lang_ids
|
||||
|
||||
|
||||
def get_bert(norm_text, word2ph, language):
|
||||
from .chinese_bert import get_bert_feature as zh_bert
|
||||
from .english_bert_mock import get_bert_feature as en_bert
|
||||
|
||||
lang_bert_func_map = {"ZH": zh_bert, "EN": en_bert}
|
||||
bert = lang_bert_func_map[language](norm_text, word2ph)
|
||||
return bert
|
||||
199
oldVersion/V101/text/chinese.py
Normal file
199
oldVersion/V101/text/chinese.py
Normal file
@@ -0,0 +1,199 @@
|
||||
import os
|
||||
import re
|
||||
|
||||
import cn2an
|
||||
from pypinyin import lazy_pinyin, Style
|
||||
|
||||
|
||||
from .symbols import punctuation
|
||||
from .tone_sandhi import ToneSandhi
|
||||
|
||||
current_file_path = os.path.dirname(__file__)
|
||||
pinyin_to_symbol_map = {
|
||||
line.split("\t")[0]: line.strip().split("\t")[1]
|
||||
for line in open(os.path.join(current_file_path, "opencpop-strict.txt")).readlines()
|
||||
}
|
||||
|
||||
import jieba.posseg as psg
|
||||
|
||||
|
||||
rep_map = {
|
||||
"：": ",",
|
||||
"；": ",",
|
||||
"，": ",",
|
||||
"。": ".",
|
||||
"！": "!",
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"·": ",",
|
||||
"、": ",",
|
||||
"...": "…",
|
||||
"$": ".",
|
||||
"“": "'",
|
||||
"”": "'",
|
||||
"‘": "'",
|
||||
"’": "'",
|
||||
"（": "'",
|
||||
"）": "'",
|
||||
"(": "'",
|
||||
")": "'",
|
||||
"《": "'",
|
||||
"》": "'",
|
||||
"【": "'",
|
||||
"】": "'",
|
||||
"[": "'",
|
||||
"]": "'",
|
||||
"—": "-",
|
||||
"～": "-",
|
||||
"~": "-",
|
||||
"「": "'",
|
||||
"」": "'",
|
||||
}
|
||||
|
||||
tone_modifier = ToneSandhi()
|
||||
|
||||
|
||||
def replace_punctuation(text):
|
||||
text = text.replace("嗯", "恩").replace("呣", "母")
|
||||
pattern = re.compile("|".join(re.escape(p) for p in rep_map.keys()))
|
||||
|
||||
replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
|
||||
|
||||
replaced_text = re.sub(
|
||||
r"[^\u4e00-\u9fa5" + "".join(punctuation) + r"]+", "", replaced_text
|
||||
)
|
||||
|
||||
return replaced_text
|
||||
|
||||
|
||||
def g2p(text):
|
||||
pattern = r"(?<=[{0}])\s*".format("".join(punctuation))
|
||||
sentences = [i for i in re.split(pattern, text) if i.strip() != ""]
|
||||
phones, tones, word2ph = _g2p(sentences)
|
||||
assert sum(word2ph) == len(phones)
|
||||
assert len(word2ph) == len(text) # Sometimes it will crash,you can add a try-catch.
|
||||
phones = ["_"] + phones + ["_"]
|
||||
tones = [0] + tones + [0]
|
||||
word2ph = [1] + word2ph + [1]
|
||||
return phones, tones, word2ph
|
||||
|
||||
|
||||
def _get_initials_finals(word):
|
||||
initials = []
|
||||
finals = []
|
||||
orig_initials = lazy_pinyin(word, neutral_tone_with_five=True, style=Style.INITIALS)
|
||||
orig_finals = lazy_pinyin(
|
||||
word, neutral_tone_with_five=True, style=Style.FINALS_TONE3
|
||||
)
|
||||
for c, v in zip(orig_initials, orig_finals):
|
||||
initials.append(c)
|
||||
finals.append(v)
|
||||
return initials, finals
|
||||
|
||||
|
||||
def _g2p(segments):
|
||||
phones_list = []
|
||||
tones_list = []
|
||||
word2ph = []
|
||||
for seg in segments:
|
||||
# Replace all English words in the sentence
|
||||
seg = re.sub("[a-zA-Z]+", "", seg)
|
||||
seg_cut = psg.lcut(seg)
|
||||
initials = []
|
||||
finals = []
|
||||
seg_cut = tone_modifier.pre_merge_for_modify(seg_cut)
|
||||
for word, pos in seg_cut:
|
||||
if pos == "eng":
|
||||
continue
|
||||
sub_initials, sub_finals = _get_initials_finals(word)
|
||||
sub_finals = tone_modifier.modified_tone(word, pos, sub_finals)
|
||||
initials.append(sub_initials)
|
||||
finals.append(sub_finals)
|
||||
|
||||
# assert len(sub_initials) == len(sub_finals) == len(word)
|
||||
initials = sum(initials, [])
|
||||
finals = sum(finals, [])
|
||||
#
|
||||
for c, v in zip(initials, finals):
|
||||
raw_pinyin = c + v
|
||||
# NOTE: post process for pypinyin outputs
|
||||
# we discriminate i, ii and iii
|
||||
if c == v:
|
||||
assert c in punctuation
|
||||
phone = [c]
|
||||
tone = "0"
|
||||
word2ph.append(1)
|
||||
else:
|
||||
v_without_tone = v[:-1]
|
||||
tone = v[-1]
|
||||
|
||||
pinyin = c + v_without_tone
|
||||
assert tone in "12345"
|
||||
|
||||
if c:
|
||||
# 多音节
|
||||
v_rep_map = {
|
||||
"uei": "ui",
|
||||
"iou": "iu",
|
||||
"uen": "un",
|
||||
}
|
||||
if v_without_tone in v_rep_map.keys():
|
||||
pinyin = c + v_rep_map[v_without_tone]
|
||||
else:
|
||||
# 单音节
|
||||
pinyin_rep_map = {
|
||||
"ing": "ying",
|
||||
"i": "yi",
|
||||
"in": "yin",
|
||||
"u": "wu",
|
||||
}
|
||||
if pinyin in pinyin_rep_map.keys():
|
||||
pinyin = pinyin_rep_map[pinyin]
|
||||
else:
|
||||
single_rep_map = {
|
||||
"v": "yu",
|
||||
"e": "e",
|
||||
"i": "y",
|
||||
"u": "w",
|
||||
}
|
||||
if pinyin[0] in single_rep_map.keys():
|
||||
pinyin = single_rep_map[pinyin[0]] + pinyin[1:]
|
||||
|
||||
assert pinyin in pinyin_to_symbol_map.keys(), (pinyin, seg, raw_pinyin)
|
||||
phone = pinyin_to_symbol_map[pinyin].split(" ")
|
||||
word2ph.append(len(phone))
|
||||
|
||||
phones_list += phone
|
||||
tones_list += [int(tone)] * len(phone)
|
||||
return phones_list, tones_list, word2ph
|
||||
|
||||
|
||||
def text_normalize(text):
|
||||
numbers = re.findall(r"\d+(?:\.?\d+)?", text)
|
||||
for number in numbers:
|
||||
text = text.replace(number, cn2an.an2cn(number), 1)
|
||||
text = replace_punctuation(text)
|
||||
return text
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph):
|
||||
from text import chinese_bert
|
||||
|
||||
return chinese_bert.get_bert_feature(text, word2ph)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from text.chinese_bert import get_bert_feature
|
||||
|
||||
text = "啊！但是《原神》是由,米哈\游自主， [研发]的一款全.新开放世界.冒险游戏"
|
||||
text = text_normalize(text)
|
||||
print(text)
|
||||
phones, tones, word2ph = g2p(text)
|
||||
bert = get_bert_feature(text, word2ph)
|
||||
|
||||
print(phones, tones, word2ph, bert.shape)
|
||||
|
||||
|
||||
# # 示例用法
|
||||
# text = "这是一个示例文本：,你好！这是一个测试...."
|
||||
# print(g2p_paddle(text)) # 输出: 这是一个示例文本你好这是一个测试
|
||||
100
oldVersion/V101/text/chinese_bert.py
Normal file
100
oldVersion/V101/text/chinese_bert.py
Normal file
@@ -0,0 +1,100 @@
|
||||
import torch
|
||||
import sys
|
||||
from transformers import AutoTokenizer, AutoModelForMaskedLM
|
||||
|
||||
device = torch.device(
|
||||
"cuda"
|
||||
if torch.cuda.is_available()
|
||||
else (
|
||||
"mps"
|
||||
if sys.platform == "darwin" and torch.backends.mps.is_available()
|
||||
else "cpu"
|
||||
)
|
||||
)
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/chinese-roberta-wwm-ext-large")
|
||||
model = AutoModelForMaskedLM.from_pretrained("./bert/chinese-roberta-wwm-ext-large").to(
|
||||
device
|
||||
)
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph):
|
||||
with torch.no_grad():
|
||||
inputs = tokenizer(text, return_tensors="pt")
|
||||
for i in inputs:
|
||||
inputs[i] = inputs[i].to(device)
|
||||
res = model(**inputs, output_hidden_states=True)
|
||||
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
|
||||
|
||||
assert len(word2ph) == len(text) + 2
|
||||
word2phone = word2ph
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
repeat_feature = res[i].repeat(word2phone[i], 1)
|
||||
phone_level_feature.append(repeat_feature)
|
||||
|
||||
phone_level_feature = torch.cat(phone_level_feature, dim=0)
|
||||
|
||||
return phone_level_feature.T
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# feature = get_bert_feature('你好,我是说的道理。')
|
||||
import torch
|
||||
|
||||
word_level_feature = torch.rand(38, 1024) # 12个词,每个词1024维特征
|
||||
word2phone = [
|
||||
1,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
]
|
||||
|
||||
# 计算总帧数
|
||||
total_frames = sum(word2phone)
|
||||
print(word_level_feature.shape)
|
||||
print(word2phone)
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
print(word_level_feature[i].shape)
|
||||
|
||||
# 对每个词重复word2phone[i]次
|
||||
repeat_feature = word_level_feature[i].repeat(word2phone[i], 1)
|
||||
phone_level_feature.append(repeat_feature)
|
||||
|
||||
phone_level_feature = torch.cat(phone_level_feature, dim=0)
|
||||
print(phone_level_feature.shape) # torch.Size([36, 1024])
|
||||
28
oldVersion/V101/text/cleaner.py
Normal file
28
oldVersion/V101/text/cleaner.py
Normal file
@@ -0,0 +1,28 @@
|
||||
from . import chinese, cleaned_text_to_sequence
|
||||
|
||||
|
||||
language_module_map = {"ZH": chinese}
|
||||
|
||||
|
||||
def clean_text(text, language):
|
||||
language_module = language_module_map[language]
|
||||
norm_text = language_module.text_normalize(text)
|
||||
phones, tones, word2ph = language_module.g2p(norm_text)
|
||||
return norm_text, phones, tones, word2ph
|
||||
|
||||
|
||||
def clean_text_bert(text, language):
|
||||
language_module = language_module_map[language]
|
||||
norm_text = language_module.text_normalize(text)
|
||||
phones, tones, word2ph = language_module.g2p(norm_text)
|
||||
bert = language_module.get_bert_feature(norm_text, word2ph)
|
||||
return phones, tones, bert
|
||||
|
||||
|
||||
def text_to_sequence(text, language):
|
||||
norm_text, phones, tones, word2ph = clean_text(text, language)
|
||||
return cleaned_text_to_sequence(phones, tones, language)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pass
|
||||
214
oldVersion/V101/text/english.py
Normal file
214
oldVersion/V101/text/english.py
Normal file
@@ -0,0 +1,214 @@
|
||||
import pickle
|
||||
import os
|
||||
import re
|
||||
from g2p_en import G2p
|
||||
|
||||
from text import symbols
|
||||
|
||||
current_file_path = os.path.dirname(__file__)
|
||||
CMU_DICT_PATH = os.path.join(current_file_path, "cmudict.rep")
|
||||
CACHE_PATH = os.path.join(current_file_path, "cmudict_cache.pickle")
|
||||
_g2p = G2p()
|
||||
|
||||
arpa = {
|
||||
"AH0",
|
||||
"S",
|
||||
"AH1",
|
||||
"EY2",
|
||||
"AE2",
|
||||
"EH0",
|
||||
"OW2",
|
||||
"UH0",
|
||||
"NG",
|
||||
"B",
|
||||
"G",
|
||||
"AY0",
|
||||
"M",
|
||||
"AA0",
|
||||
"F",
|
||||
"AO0",
|
||||
"ER2",
|
||||
"UH1",
|
||||
"IY1",
|
||||
"AH2",
|
||||
"DH",
|
||||
"IY0",
|
||||
"EY1",
|
||||
"IH0",
|
||||
"K",
|
||||
"N",
|
||||
"W",
|
||||
"IY2",
|
||||
"T",
|
||||
"AA1",
|
||||
"ER1",
|
||||
"EH2",
|
||||
"OY0",
|
||||
"UH2",
|
||||
"UW1",
|
||||
"Z",
|
||||
"AW2",
|
||||
"AW1",
|
||||
"V",
|
||||
"UW2",
|
||||
"AA2",
|
||||
"ER",
|
||||
"AW0",
|
||||
"UW0",
|
||||
"R",
|
||||
"OW1",
|
||||
"EH1",
|
||||
"ZH",
|
||||
"AE0",
|
||||
"IH2",
|
||||
"IH",
|
||||
"Y",
|
||||
"JH",
|
||||
"P",
|
||||
"AY1",
|
||||
"EY0",
|
||||
"OY2",
|
||||
"TH",
|
||||
"HH",
|
||||
"D",
|
||||
"ER0",
|
||||
"CH",
|
||||
"AO1",
|
||||
"AE1",
|
||||
"AO2",
|
||||
"OY1",
|
||||
"AY2",
|
||||
"IH1",
|
||||
"OW0",
|
||||
"L",
|
||||
"SH",
|
||||
}
|
||||
|
||||
|
||||
def post_replace_ph(ph):
|
||||
rep_map = {
|
||||
"：": ",",
|
||||
"；": ",",
|
||||
"，": ",",
|
||||
"。": ".",
|
||||
"！": "!",
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"·": ",",
|
||||
"、": ",",
|
||||
"...": "…",
|
||||
"v": "V",
|
||||
}
|
||||
if ph in rep_map.keys():
|
||||
ph = rep_map[ph]
|
||||
if ph in symbols:
|
||||
return ph
|
||||
if ph not in symbols:
|
||||
ph = "UNK"
|
||||
return ph
|
||||
|
||||
|
||||
def read_dict():
|
||||
g2p_dict = {}
|
||||
start_line = 49
|
||||
with open(CMU_DICT_PATH) as f:
|
||||
line = f.readline()
|
||||
line_index = 1
|
||||
while line:
|
||||
if line_index >= start_line:
|
||||
line = line.strip()
|
||||
word_split = line.split(" ")
|
||||
word = word_split[0]
|
||||
|
||||
syllable_split = word_split[1].split(" - ")
|
||||
g2p_dict[word] = []
|
||||
for syllable in syllable_split:
|
||||
phone_split = syllable.split(" ")
|
||||
g2p_dict[word].append(phone_split)
|
||||
|
||||
line_index = line_index + 1
|
||||
line = f.readline()
|
||||
|
||||
return g2p_dict
|
||||
|
||||
|
||||
def cache_dict(g2p_dict, file_path):
|
||||
with open(file_path, "wb") as pickle_file:
|
||||
pickle.dump(g2p_dict, pickle_file)
|
||||
|
||||
|
||||
def get_dict():
|
||||
if os.path.exists(CACHE_PATH):
|
||||
with open(CACHE_PATH, "rb") as pickle_file:
|
||||
g2p_dict = pickle.load(pickle_file)
|
||||
else:
|
||||
g2p_dict = read_dict()
|
||||
cache_dict(g2p_dict, CACHE_PATH)
|
||||
|
||||
return g2p_dict
|
||||
|
||||
|
||||
eng_dict = get_dict()
|
||||
|
||||
|
||||
def refine_ph(phn):
|
||||
tone = 0
|
||||
if re.search(r"\d$", phn):
|
||||
tone = int(phn[-1]) + 1
|
||||
phn = phn[:-1]
|
||||
return phn.lower(), tone
|
||||
|
||||
|
||||
def refine_syllables(syllables):
|
||||
tones = []
|
||||
phonemes = []
|
||||
for phn_list in syllables:
|
||||
for i in range(len(phn_list)):
|
||||
phn = phn_list[i]
|
||||
phn, tone = refine_ph(phn)
|
||||
phonemes.append(phn)
|
||||
tones.append(tone)
|
||||
return phonemes, tones
|
||||
|
||||
|
||||
def text_normalize(text):
|
||||
# todo: eng text normalize
|
||||
return text
|
||||
|
||||
|
||||
def g2p(text):
|
||||
phones = []
|
||||
tones = []
|
||||
words = re.split(r"([,;.\-\?\!\s+])", text)
|
||||
for w in words:
|
||||
if w.upper() in eng_dict:
|
||||
phns, tns = refine_syllables(eng_dict[w.upper()])
|
||||
phones += phns
|
||||
tones += tns
|
||||
else:
|
||||
phone_list = list(filter(lambda p: p != " ", _g2p(w)))
|
||||
for ph in phone_list:
|
||||
if ph in arpa:
|
||||
ph, tn = refine_ph(ph)
|
||||
phones.append(ph)
|
||||
tones.append(tn)
|
||||
else:
|
||||
phones.append(ph)
|
||||
tones.append(0)
|
||||
# todo: implement word2ph
|
||||
word2ph = [1 for i in phones]
|
||||
|
||||
phones = [post_replace_ph(i) for i in phones]
|
||||
return phones, tones, word2ph
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# print(get_dict())
|
||||
# print(eng_word_to_phoneme("hello"))
|
||||
print(g2p("In this paper, we propose 1 DSPGAN, a GAN-based universal vocoder."))
|
||||
# all_phones = set()
|
||||
# for k, syllables in eng_dict.items():
|
||||
# for group in syllables:
|
||||
# for ph in group:
|
||||
# all_phones.add(ph)
|
||||
# print(all_phones)
|
||||
5
oldVersion/V101/text/english_bert_mock.py
Normal file
5
oldVersion/V101/text/english_bert_mock.py
Normal file
@@ -0,0 +1,5 @@
|
||||
import torch
|
||||
|
||||
|
||||
def get_bert_feature(norm_text, word2ph):
|
||||
return torch.zeros(1024, sum(word2ph))
|
||||
112
oldVersion/V101/text/japanese.py
Normal file
112
oldVersion/V101/text/japanese.py
Normal file
@@ -0,0 +1,112 @@
|
||||
# modified from https://github.com/CjangCjengh/vits/blob/main/text/japanese.py
|
||||
import re
|
||||
import sys
|
||||
|
||||
import pyopenjtalk
|
||||
|
||||
from . import symbols
|
||||
|
||||
# Regular expression matching Japanese without punctuation marks:
|
||||
_japanese_characters = re.compile(
|
||||
r"[A-Za-z\d\u3005\u3040-\u30ff\u4e00-\u9fff\uff11-\uff19\uff21-\uff3a\uff41-\uff5a\uff66-\uff9d]"
|
||||
)
|
||||
|
||||
# Regular expression matching non-Japanese characters or punctuation marks:
|
||||
_japanese_marks = re.compile(
|
||||
r"[^A-Za-z\d\u3005\u3040-\u30ff\u4e00-\u9fff\uff11-\uff19\uff21-\uff3a\uff41-\uff5a\uff66-\uff9d]"
|
||||
)
|
||||
|
||||
# List of (symbol, Japanese) pairs for marks:
|
||||
_symbols_to_japanese = [(re.compile("%s" % x[0]), x[1]) for x in [("％", "パーセント")]]
|
||||
|
||||
|
||||
# List of (consonant, sokuon) pairs:
|
||||
_real_sokuon = [
|
||||
(re.compile("%s" % x[0]), x[1])
|
||||
for x in [
|
||||
(r"Q([↑↓]*[kg])", r"k#\1"),
|
||||
(r"Q([↑↓]*[tdjʧ])", r"t#\1"),
|
||||
(r"Q([↑↓]*[sʃ])", r"s\1"),
|
||||
(r"Q([↑↓]*[pb])", r"p#\1"),
|
||||
]
|
||||
]
|
||||
|
||||
# List of (consonant, hatsuon) pairs:
|
||||
_real_hatsuon = [
|
||||
(re.compile("%s" % x[0]), x[1])
|
||||
for x in [
|
||||
(r"N([↑↓]*[pbm])", r"m\1"),
|
||||
(r"N([↑↓]*[ʧʥj])", r"n^\1"),
|
||||
(r"N([↑↓]*[tdn])", r"n\1"),
|
||||
(r"N([↑↓]*[kg])", r"ŋ\1"),
|
||||
]
|
||||
]
|
||||
|
||||
|
||||
def post_replace_ph(ph):
|
||||
rep_map = {
|
||||
"：": ",",
|
||||
"；": ",",
|
||||
"，": ",",
|
||||
"。": ".",
|
||||
"！": "!",
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"·": ",",
|
||||
"、": ",",
|
||||
"...": "…",
|
||||
"v": "V",
|
||||
}
|
||||
if ph in rep_map.keys():
|
||||
ph = rep_map[ph]
|
||||
if ph in symbols:
|
||||
return ph
|
||||
if ph not in symbols:
|
||||
ph = "UNK"
|
||||
return ph
|
||||
|
||||
|
||||
def symbols_to_japanese(text):
|
||||
for regex, replacement in _symbols_to_japanese:
|
||||
text = re.sub(regex, replacement, text)
|
||||
return text
|
||||
|
||||
|
||||
def preprocess_jap(text):
|
||||
"""Reference https://r9y9.github.io/ttslearn/latest/notebooks/ch10_Recipe-Tacotron.html"""
|
||||
text = symbols_to_japanese(text)
|
||||
sentences = re.split(_japanese_marks, text)
|
||||
marks = re.findall(_japanese_marks, text)
|
||||
text = []
|
||||
for i, sentence in enumerate(sentences):
|
||||
if re.match(_japanese_characters, sentence):
|
||||
p = pyopenjtalk.g2p(sentence)
|
||||
text += p.split(" ")
|
||||
|
||||
if i < len(marks):
|
||||
text += [marks[i].replace(" ", "")]
|
||||
return text
|
||||
|
||||
|
||||
def text_normalize(text):
|
||||
# todo: jap text normalize
|
||||
return text
|
||||
|
||||
|
||||
def g2p(norm_text):
|
||||
phones = preprocess_jap(norm_text)
|
||||
phones = [post_replace_ph(i) for i in phones]
|
||||
# todo: implement tones and word2ph
|
||||
tones = [0 for i in phones]
|
||||
word2ph = [1 for i in phones]
|
||||
return phones, tones, word2ph
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
for line in open("../../../Downloads/transcript_utf8.txt").readlines():
|
||||
text = line.split(":")[1]
|
||||
phones, tones, word2ph = g2p(text)
|
||||
for p in phones:
|
||||
if p == "z":
|
||||
print(text, phones)
|
||||
sys.exit(0)
|
||||
429
oldVersion/V101/text/opencpop-strict.txt
Normal file
429
oldVersion/V101/text/opencpop-strict.txt
Normal file
@@ -0,0 +1,429 @@
|
||||
a AA a
|
||||
ai AA ai
|
||||
an AA an
|
||||
ang AA ang
|
||||
ao AA ao
|
||||
ba b a
|
||||
bai b ai
|
||||
ban b an
|
||||
bang b ang
|
||||
bao b ao
|
||||
bei b ei
|
||||
ben b en
|
||||
beng b eng
|
||||
bi b i
|
||||
bian b ian
|
||||
biao b iao
|
||||
bie b ie
|
||||
bin b in
|
||||
bing b ing
|
||||
bo b o
|
||||
bu b u
|
||||
ca c a
|
||||
cai c ai
|
||||
can c an
|
||||
cang c ang
|
||||
cao c ao
|
||||
ce c e
|
||||
cei c ei
|
||||
cen c en
|
||||
ceng c eng
|
||||
cha ch a
|
||||
chai ch ai
|
||||
chan ch an
|
||||
chang ch ang
|
||||
chao ch ao
|
||||
che ch e
|
||||
chen ch en
|
||||
cheng ch eng
|
||||
chi ch ir
|
||||
chong ch ong
|
||||
chou ch ou
|
||||
chu ch u
|
||||
chua ch ua
|
||||
chuai ch uai
|
||||
chuan ch uan
|
||||
chuang ch uang
|
||||
chui ch ui
|
||||
chun ch un
|
||||
chuo ch uo
|
||||
ci c i0
|
||||
cong c ong
|
||||
cou c ou
|
||||
cu c u
|
||||
cuan c uan
|
||||
cui c ui
|
||||
cun c un
|
||||
cuo c uo
|
||||
da d a
|
||||
dai d ai
|
||||
dan d an
|
||||
dang d ang
|
||||
dao d ao
|
||||
de d e
|
||||
dei d ei
|
||||
den d en
|
||||
deng d eng
|
||||
di d i
|
||||
dia d ia
|
||||
dian d ian
|
||||
diao d iao
|
||||
die d ie
|
||||
ding d ing
|
||||
diu d iu
|
||||
dong d ong
|
||||
dou d ou
|
||||
du d u
|
||||
duan d uan
|
||||
dui d ui
|
||||
dun d un
|
||||
duo d uo
|
||||
e EE e
|
||||
ei EE ei
|
||||
en EE en
|
||||
eng EE eng
|
||||
er EE er
|
||||
fa f a
|
||||
fan f an
|
||||
fang f ang
|
||||
fei f ei
|
||||
fen f en
|
||||
feng f eng
|
||||
fo f o
|
||||
fou f ou
|
||||
fu f u
|
||||
ga g a
|
||||
gai g ai
|
||||
gan g an
|
||||
gang g ang
|
||||
gao g ao
|
||||
ge g e
|
||||
gei g ei
|
||||
gen g en
|
||||
geng g eng
|
||||
gong g ong
|
||||
gou g ou
|
||||
gu g u
|
||||
gua g ua
|
||||
guai g uai
|
||||
guan g uan
|
||||
guang g uang
|
||||
gui g ui
|
||||
gun g un
|
||||
guo g uo
|
||||
ha h a
|
||||
hai h ai
|
||||
han h an
|
||||
hang h ang
|
||||
hao h ao
|
||||
he h e
|
||||
hei h ei
|
||||
hen h en
|
||||
heng h eng
|
||||
hong h ong
|
||||
hou h ou
|
||||
hu h u
|
||||
hua h ua
|
||||
huai h uai
|
||||
huan h uan
|
||||
huang h uang
|
||||
hui h ui
|
||||
hun h un
|
||||
huo h uo
|
||||
ji j i
|
||||
jia j ia
|
||||
jian j ian
|
||||
jiang j iang
|
||||
jiao j iao
|
||||
jie j ie
|
||||
jin j in
|
||||
jing j ing
|
||||
jiong j iong
|
||||
jiu j iu
|
||||
ju j v
|
||||
jv j v
|
||||
juan j van
|
||||
jvan j van
|
||||
jue j ve
|
||||
jve j ve
|
||||
jun j vn
|
||||
jvn j vn
|
||||
ka k a
|
||||
kai k ai
|
||||
kan k an
|
||||
kang k ang
|
||||
kao k ao
|
||||
ke k e
|
||||
kei k ei
|
||||
ken k en
|
||||
keng k eng
|
||||
kong k ong
|
||||
kou k ou
|
||||
ku k u
|
||||
kua k ua
|
||||
kuai k uai
|
||||
kuan k uan
|
||||
kuang k uang
|
||||
kui k ui
|
||||
kun k un
|
||||
kuo k uo
|
||||
la l a
|
||||
lai l ai
|
||||
lan l an
|
||||
lang l ang
|
||||
lao l ao
|
||||
le l e
|
||||
lei l ei
|
||||
leng l eng
|
||||
li l i
|
||||
lia l ia
|
||||
lian l ian
|
||||
liang l iang
|
||||
liao l iao
|
||||
lie l ie
|
||||
lin l in
|
||||
ling l ing
|
||||
liu l iu
|
||||
lo l o
|
||||
long l ong
|
||||
lou l ou
|
||||
lu l u
|
||||
luan l uan
|
||||
lun l un
|
||||
luo l uo
|
||||
lv l v
|
||||
lve l ve
|
||||
ma m a
|
||||
mai m ai
|
||||
man m an
|
||||
mang m ang
|
||||
mao m ao
|
||||
me m e
|
||||
mei m ei
|
||||
men m en
|
||||
meng m eng
|
||||
mi m i
|
||||
mian m ian
|
||||
miao m iao
|
||||
mie m ie
|
||||
min m in
|
||||
ming m ing
|
||||
miu m iu
|
||||
mo m o
|
||||
mou m ou
|
||||
mu m u
|
||||
na n a
|
||||
nai n ai
|
||||
nan n an
|
||||
nang n ang
|
||||
nao n ao
|
||||
ne n e
|
||||
nei n ei
|
||||
nen n en
|
||||
neng n eng
|
||||
ni n i
|
||||
nian n ian
|
||||
niang n iang
|
||||
niao n iao
|
||||
nie n ie
|
||||
nin n in
|
||||
ning n ing
|
||||
niu n iu
|
||||
nong n ong
|
||||
nou n ou
|
||||
nu n u
|
||||
nuan n uan
|
||||
nun n un
|
||||
nuo n uo
|
||||
nv n v
|
||||
nve n ve
|
||||
o OO o
|
||||
ou OO ou
|
||||
pa p a
|
||||
pai p ai
|
||||
pan p an
|
||||
pang p ang
|
||||
pao p ao
|
||||
pei p ei
|
||||
pen p en
|
||||
peng p eng
|
||||
pi p i
|
||||
pian p ian
|
||||
piao p iao
|
||||
pie p ie
|
||||
pin p in
|
||||
ping p ing
|
||||
po p o
|
||||
pou p ou
|
||||
pu p u
|
||||
qi q i
|
||||
qia q ia
|
||||
qian q ian
|
||||
qiang q iang
|
||||
qiao q iao
|
||||
qie q ie
|
||||
qin q in
|
||||
qing q ing
|
||||
qiong q iong
|
||||
qiu q iu
|
||||
qu q v
|
||||
qv q v
|
||||
quan q van
|
||||
qvan q van
|
||||
que q ve
|
||||
qve q ve
|
||||
qun q vn
|
||||
qvn q vn
|
||||
ran r an
|
||||
rang r ang
|
||||
rao r ao
|
||||
re r e
|
||||
ren r en
|
||||
reng r eng
|
||||
ri r ir
|
||||
rong r ong
|
||||
rou r ou
|
||||
ru r u
|
||||
rua r ua
|
||||
ruan r uan
|
||||
rui r ui
|
||||
run r un
|
||||
ruo r uo
|
||||
sa s a
|
||||
sai s ai
|
||||
san s an
|
||||
sang s ang
|
||||
sao s ao
|
||||
se s e
|
||||
sen s en
|
||||
seng s eng
|
||||
sha sh a
|
||||
shai sh ai
|
||||
shan sh an
|
||||
shang sh ang
|
||||
shao sh ao
|
||||
she sh e
|
||||
shei sh ei
|
||||
shen sh en
|
||||
sheng sh eng
|
||||
shi sh ir
|
||||
shou sh ou
|
||||
shu sh u
|
||||
shua sh ua
|
||||
shuai sh uai
|
||||
shuan sh uan
|
||||
shuang sh uang
|
||||
shui sh ui
|
||||
shun sh un
|
||||
shuo sh uo
|
||||
si s i0
|
||||
song s ong
|
||||
sou s ou
|
||||
su s u
|
||||
suan s uan
|
||||
sui s ui
|
||||
sun s un
|
||||
suo s uo
|
||||
ta t a
|
||||
tai t ai
|
||||
tan t an
|
||||
tang t ang
|
||||
tao t ao
|
||||
te t e
|
||||
tei t ei
|
||||
teng t eng
|
||||
ti t i
|
||||
tian t ian
|
||||
tiao t iao
|
||||
tie t ie
|
||||
ting t ing
|
||||
tong t ong
|
||||
tou t ou
|
||||
tu t u
|
||||
tuan t uan
|
||||
tui t ui
|
||||
tun t un
|
||||
tuo t uo
|
||||
wa w a
|
||||
wai w ai
|
||||
wan w an
|
||||
wang w ang
|
||||
wei w ei
|
||||
wen w en
|
||||
weng w eng
|
||||
wo w o
|
||||
wu w u
|
||||
xi x i
|
||||
xia x ia
|
||||
xian x ian
|
||||
xiang x iang
|
||||
xiao x iao
|
||||
xie x ie
|
||||
xin x in
|
||||
xing x ing
|
||||
xiong x iong
|
||||
xiu x iu
|
||||
xu x v
|
||||
xv x v
|
||||
xuan x van
|
||||
xvan x van
|
||||
xue x ve
|
||||
xve x ve
|
||||
xun x vn
|
||||
xvn x vn
|
||||
ya y a
|
||||
yan y En
|
||||
yang y ang
|
||||
yao y ao
|
||||
ye y E
|
||||
yi y i
|
||||
yin y in
|
||||
ying y ing
|
||||
yo y o
|
||||
yong y ong
|
||||
you y ou
|
||||
yu y v
|
||||
yv y v
|
||||
yuan y van
|
||||
yvan y van
|
||||
yue y ve
|
||||
yve y ve
|
||||
yun y vn
|
||||
yvn y vn
|
||||
za z a
|
||||
zai z ai
|
||||
zan z an
|
||||
zang z ang
|
||||
zao z ao
|
||||
ze z e
|
||||
zei z ei
|
||||
zen z en
|
||||
zeng z eng
|
||||
zha zh a
|
||||
zhai zh ai
|
||||
zhan zh an
|
||||
zhang zh ang
|
||||
zhao zh ao
|
||||
zhe zh e
|
||||
zhei zh ei
|
||||
zhen zh en
|
||||
zheng zh eng
|
||||
zhi zh ir
|
||||
zhong zh ong
|
||||
zhou zh ou
|
||||
zhu zh u
|
||||
zhua zh ua
|
||||
zhuai zh uai
|
||||
zhuan zh uan
|
||||
zhuang zh uang
|
||||
zhui zh ui
|
||||
zhun zh un
|
||||
zhuo zh uo
|
||||
zi z i0
|
||||
zong z ong
|
||||
zou z ou
|
||||
zu z u
|
||||
zuan z uan
|
||||
zui z ui
|
||||
zun z un
|
||||
zuo z uo
|
||||
183
oldVersion/V101/text/symbols.py
Normal file
183
oldVersion/V101/text/symbols.py
Normal file
@@ -0,0 +1,183 @@
|
||||
punctuation = ["!", "?", "…", ",", ".", "'", "-"]
|
||||
pu_symbols = punctuation + ["SP", "UNK"]
|
||||
pad = "_"
|
||||
|
||||
# chinese
|
||||
zh_symbols = [
|
||||
"E",
|
||||
"En",
|
||||
"a",
|
||||
"ai",
|
||||
"an",
|
||||
"ang",
|
||||
"ao",
|
||||
"b",
|
||||
"c",
|
||||
"ch",
|
||||
"d",
|
||||
"e",
|
||||
"ei",
|
||||
"en",
|
||||
"eng",
|
||||
"er",
|
||||
"f",
|
||||
"g",
|
||||
"h",
|
||||
"i",
|
||||
"i0",
|
||||
"ia",
|
||||
"ian",
|
||||
"iang",
|
||||
"iao",
|
||||
"ie",
|
||||
"in",
|
||||
"ing",
|
||||
"iong",
|
||||
"ir",
|
||||
"iu",
|
||||
"j",
|
||||
"k",
|
||||
"l",
|
||||
"m",
|
||||
"n",
|
||||
"o",
|
||||
"ong",
|
||||
"ou",
|
||||
"p",
|
||||
"q",
|
||||
"r",
|
||||
"s",
|
||||
"sh",
|
||||
"t",
|
||||
"u",
|
||||
"ua",
|
||||
"uai",
|
||||
"uan",
|
||||
"uang",
|
||||
"ui",
|
||||
"un",
|
||||
"uo",
|
||||
"v",
|
||||
"van",
|
||||
"ve",
|
||||
"vn",
|
||||
"w",
|
||||
"x",
|
||||
"y",
|
||||
"z",
|
||||
"zh",
|
||||
"AA",
|
||||
"EE",
|
||||
"OO",
|
||||
]
|
||||
num_zh_tones = 6
|
||||
|
||||
# japanese
|
||||
ja_symbols = [
|
||||
"I",
|
||||
"N",
|
||||
"U",
|
||||
"a",
|
||||
"b",
|
||||
"by",
|
||||
"ch",
|
||||
"cl",
|
||||
"d",
|
||||
"dy",
|
||||
"e",
|
||||
"f",
|
||||
"g",
|
||||
"gy",
|
||||
"h",
|
||||
"hy",
|
||||
"i",
|
||||
"j",
|
||||
"k",
|
||||
"ky",
|
||||
"m",
|
||||
"my",
|
||||
"n",
|
||||
"ny",
|
||||
"o",
|
||||
"p",
|
||||
"py",
|
||||
"r",
|
||||
"ry",
|
||||
"s",
|
||||
"sh",
|
||||
"t",
|
||||
"ts",
|
||||
"u",
|
||||
"V",
|
||||
"w",
|
||||
"y",
|
||||
"z",
|
||||
]
|
||||
num_ja_tones = 1
|
||||
|
||||
# English
|
||||
en_symbols = [
|
||||
"aa",
|
||||
"ae",
|
||||
"ah",
|
||||
"ao",
|
||||
"aw",
|
||||
"ay",
|
||||
"b",
|
||||
"ch",
|
||||
"d",
|
||||
"dh",
|
||||
"eh",
|
||||
"er",
|
||||
"ey",
|
||||
"f",
|
||||
"g",
|
||||
"hh",
|
||||
"ih",
|
||||
"iy",
|
||||
"jh",
|
||||
"k",
|
||||
"l",
|
||||
"m",
|
||||
"n",
|
||||
"ng",
|
||||
"ow",
|
||||
"oy",
|
||||
"p",
|
||||
"r",
|
||||
"s",
|
||||
"sh",
|
||||
"t",
|
||||
"th",
|
||||
"uh",
|
||||
"uw",
|
||||
"V",
|
||||
"w",
|
||||
"y",
|
||||
"z",
|
||||
"zh",
|
||||
]
|
||||
num_en_tones = 4
|
||||
|
||||
# combine all symbols
|
||||
normal_symbols = sorted(set(zh_symbols + ja_symbols + en_symbols))
|
||||
symbols = [pad] + normal_symbols + pu_symbols
|
||||
sil_phonemes_ids = [symbols.index(i) for i in pu_symbols]
|
||||
|
||||
# combine all tones
|
||||
num_tones = num_zh_tones + num_ja_tones + num_en_tones
|
||||
|
||||
# language maps
|
||||
language_id_map = {"ZH": 0, "JA": 1, "EN": 2}
|
||||
num_languages = len(language_id_map.keys())
|
||||
|
||||
language_tone_start_map = {
|
||||
"ZH": 0,
|
||||
"JA": num_zh_tones,
|
||||
"EN": num_zh_tones + num_ja_tones,
|
||||
}
|
||||
|
||||
if __name__ == "__main__":
|
||||
a = set(zh_symbols)
|
||||
b = set(en_symbols)
|
||||
print(sorted(a & b))
|
||||
769
oldVersion/V101/text/tone_sandhi.py
Normal file
769
oldVersion/V101/text/tone_sandhi.py
Normal file
@@ -0,0 +1,769 @@
|
||||
# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
from typing import List
|
||||
from typing import Tuple
|
||||
|
||||
import jieba
|
||||
from pypinyin import lazy_pinyin
|
||||
from pypinyin import Style
|
||||
|
||||
|
||||
class ToneSandhi:
|
||||
def __init__(self):
|
||||
self.must_neural_tone_words = {
|
||||
"麻烦",
|
||||
"麻利",
|
||||
"鸳鸯",
|
||||
"高粱",
|
||||
"骨头",
|
||||
"骆驼",
|
||||
"马虎",
|
||||
"首饰",
|
||||
"馒头",
|
||||
"馄饨",
|
||||
"风筝",
|
||||
"难为",
|
||||
"队伍",
|
||||
"阔气",
|
||||
"闺女",
|
||||
"门道",
|
||||
"锄头",
|
||||
"铺盖",
|
||||
"铃铛",
|
||||
"铁匠",
|
||||
"钥匙",
|
||||
"里脊",
|
||||
"里头",
|
||||
"部分",
|
||||
"那么",
|
||||
"道士",
|
||||
"造化",
|
||||
"迷糊",
|
||||
"连累",
|
||||
"这么",
|
||||
"这个",
|
||||
"运气",
|
||||
"过去",
|
||||
"软和",
|
||||
"转悠",
|
||||
"踏实",
|
||||
"跳蚤",
|
||||
"跟头",
|
||||
"趔趄",
|
||||
"财主",
|
||||
"豆腐",
|
||||
"讲究",
|
||||
"记性",
|
||||
"记号",
|
||||
"认识",
|
||||
"规矩",
|
||||
"见识",
|
||||
"裁缝",
|
||||
"补丁",
|
||||
"衣裳",
|
||||
"衣服",
|
||||
"衙门",
|
||||
"街坊",
|
||||
"行李",
|
||||
"行当",
|
||||
"蛤蟆",
|
||||
"蘑菇",
|
||||
"薄荷",
|
||||
"葫芦",
|
||||
"葡萄",
|
||||
"萝卜",
|
||||
"荸荠",
|
||||
"苗条",
|
||||
"苗头",
|
||||
"苍蝇",
|
||||
"芝麻",
|
||||
"舒服",
|
||||
"舒坦",
|
||||
"舌头",
|
||||
"自在",
|
||||
"膏药",
|
||||
"脾气",
|
||||
"脑袋",
|
||||
"脊梁",
|
||||
"能耐",
|
||||
"胳膊",
|
||||
"胭脂",
|
||||
"胡萝",
|
||||
"胡琴",
|
||||
"胡同",
|
||||
"聪明",
|
||||
"耽误",
|
||||
"耽搁",
|
||||
"耷拉",
|
||||
"耳朵",
|
||||
"老爷",
|
||||
"老实",
|
||||
"老婆",
|
||||
"老头",
|
||||
"老太",
|
||||
"翻腾",
|
||||
"罗嗦",
|
||||
"罐头",
|
||||
"编辑",
|
||||
"结实",
|
||||
"红火",
|
||||
"累赘",
|
||||
"糨糊",
|
||||
"糊涂",
|
||||
"精神",
|
||||
"粮食",
|
||||
"簸箕",
|
||||
"篱笆",
|
||||
"算计",
|
||||
"算盘",
|
||||
"答应",
|
||||
"笤帚",
|
||||
"笑语",
|
||||
"笑话",
|
||||
"窟窿",
|
||||
"窝囊",
|
||||
"窗户",
|
||||
"稳当",
|
||||
"稀罕",
|
||||
"称呼",
|
||||
"秧歌",
|
||||
"秀气",
|
||||
"秀才",
|
||||
"福气",
|
||||
"祖宗",
|
||||
"砚台",
|
||||
"码头",
|
||||
"石榴",
|
||||
"石头",
|
||||
"石匠",
|
||||
"知识",
|
||||
"眼睛",
|
||||
"眯缝",
|
||||
"眨巴",
|
||||
"眉毛",
|
||||
"相声",
|
||||
"盘算",
|
||||
"白净",
|
||||
"痢疾",
|
||||
"痛快",
|
||||
"疟疾",
|
||||
"疙瘩",
|
||||
"疏忽",
|
||||
"畜生",
|
||||
"生意",
|
||||
"甘蔗",
|
||||
"琵琶",
|
||||
"琢磨",
|
||||
"琉璃",
|
||||
"玻璃",
|
||||
"玫瑰",
|
||||
"玄乎",
|
||||
"狐狸",
|
||||
"状元",
|
||||
"特务",
|
||||
"牲口",
|
||||
"牙碜",
|
||||
"牌楼",
|
||||
"爽快",
|
||||
"爱人",
|
||||
"热闹",
|
||||
"烧饼",
|
||||
"烟筒",
|
||||
"烂糊",
|
||||
"点心",
|
||||
"炊帚",
|
||||
"灯笼",
|
||||
"火候",
|
||||
"漂亮",
|
||||
"滑溜",
|
||||
"溜达",
|
||||
"温和",
|
||||
"清楚",
|
||||
"消息",
|
||||
"浪头",
|
||||
"活泼",
|
||||
"比方",
|
||||
"正经",
|
||||
"欺负",
|
||||
"模糊",
|
||||
"槟榔",
|
||||
"棺材",
|
||||
"棒槌",
|
||||
"棉花",
|
||||
"核桃",
|
||||
"栅栏",
|
||||
"柴火",
|
||||
"架势",
|
||||
"枕头",
|
||||
"枇杷",
|
||||
"机灵",
|
||||
"本事",
|
||||
"木头",
|
||||
"木匠",
|
||||
"朋友",
|
||||
"月饼",
|
||||
"月亮",
|
||||
"暖和",
|
||||
"明白",
|
||||
"时候",
|
||||
"新鲜",
|
||||
"故事",
|
||||
"收拾",
|
||||
"收成",
|
||||
"提防",
|
||||
"挖苦",
|
||||
"挑剔",
|
||||
"指甲",
|
||||
"指头",
|
||||
"拾掇",
|
||||
"拳头",
|
||||
"拨弄",
|
||||
"招牌",
|
||||
"招呼",
|
||||
"抬举",
|
||||
"护士",
|
||||
"折腾",
|
||||
"扫帚",
|
||||
"打量",
|
||||
"打算",
|
||||
"打点",
|
||||
"打扮",
|
||||
"打听",
|
||||
"打发",
|
||||
"扎实",
|
||||
"扁担",
|
||||
"戒指",
|
||||
"懒得",
|
||||
"意识",
|
||||
"意思",
|
||||
"情形",
|
||||
"悟性",
|
||||
"怪物",
|
||||
"思量",
|
||||
"怎么",
|
||||
"念头",
|
||||
"念叨",
|
||||
"快活",
|
||||
"忙活",
|
||||
"志气",
|
||||
"心思",
|
||||
"得罪",
|
||||
"张罗",
|
||||
"弟兄",
|
||||
"开通",
|
||||
"应酬",
|
||||
"庄稼",
|
||||
"干事",
|
||||
"帮手",
|
||||
"帐篷",
|
||||
"希罕",
|
||||
"师父",
|
||||
"师傅",
|
||||
"巴结",
|
||||
"巴掌",
|
||||
"差事",
|
||||
"工夫",
|
||||
"岁数",
|
||||
"屁股",
|
||||
"尾巴",
|
||||
"少爷",
|
||||
"小气",
|
||||
"小伙",
|
||||
"将就",
|
||||
"对头",
|
||||
"对付",
|
||||
"寡妇",
|
||||
"家伙",
|
||||
"客气",
|
||||
"实在",
|
||||
"官司",
|
||||
"学问",
|
||||
"学生",
|
||||
"字号",
|
||||
"嫁妆",
|
||||
"媳妇",
|
||||
"媒人",
|
||||
"婆家",
|
||||
"娘家",
|
||||
"委屈",
|
||||
"姑娘",
|
||||
"姐夫",
|
||||
"妯娌",
|
||||
"妥当",
|
||||
"妖精",
|
||||
"奴才",
|
||||
"女婿",
|
||||
"头发",
|
||||
"太阳",
|
||||
"大爷",
|
||||
"大方",
|
||||
"大意",
|
||||
"大夫",
|
||||
"多少",
|
||||
"多么",
|
||||
"外甥",
|
||||
"壮实",
|
||||
"地道",
|
||||
"地方",
|
||||
"在乎",
|
||||
"困难",
|
||||
"嘴巴",
|
||||
"嘱咐",
|
||||
"嘟囔",
|
||||
"嘀咕",
|
||||
"喜欢",
|
||||
"喇嘛",
|
||||
"喇叭",
|
||||
"商量",
|
||||
"唾沫",
|
||||
"哑巴",
|
||||
"哈欠",
|
||||
"哆嗦",
|
||||
"咳嗽",
|
||||
"和尚",
|
||||
"告诉",
|
||||
"告示",
|
||||
"含糊",
|
||||
"吓唬",
|
||||
"后头",
|
||||
"名字",
|
||||
"名堂",
|
||||
"合同",
|
||||
"吆喝",
|
||||
"叫唤",
|
||||
"口袋",
|
||||
"厚道",
|
||||
"厉害",
|
||||
"千斤",
|
||||
"包袱",
|
||||
"包涵",
|
||||
"匀称",
|
||||
"勤快",
|
||||
"动静",
|
||||
"动弹",
|
||||
"功夫",
|
||||
"力气",
|
||||
"前头",
|
||||
"刺猬",
|
||||
"刺激",
|
||||
"别扭",
|
||||
"利落",
|
||||
"利索",
|
||||
"利害",
|
||||
"分析",
|
||||
"出息",
|
||||
"凑合",
|
||||
"凉快",
|
||||
"冷战",
|
||||
"冤枉",
|
||||
"冒失",
|
||||
"养活",
|
||||
"关系",
|
||||
"先生",
|
||||
"兄弟",
|
||||
"便宜",
|
||||
"使唤",
|
||||
"佩服",
|
||||
"作坊",
|
||||
"体面",
|
||||
"位置",
|
||||
"似的",
|
||||
"伙计",
|
||||
"休息",
|
||||
"什么",
|
||||
"人家",
|
||||
"亲戚",
|
||||
"亲家",
|
||||
"交情",
|
||||
"云彩",
|
||||
"事情",
|
||||
"买卖",
|
||||
"主意",
|
||||
"丫头",
|
||||
"丧气",
|
||||
"两口",
|
||||
"东西",
|
||||
"东家",
|
||||
"世故",
|
||||
"不由",
|
||||
"不在",
|
||||
"下水",
|
||||
"下巴",
|
||||
"上头",
|
||||
"上司",
|
||||
"丈夫",
|
||||
"丈人",
|
||||
"一辈",
|
||||
"那个",
|
||||
"菩萨",
|
||||
"父亲",
|
||||
"母亲",
|
||||
"咕噜",
|
||||
"邋遢",
|
||||
"费用",
|
||||
"冤家",
|
||||
"甜头",
|
||||
"介绍",
|
||||
"荒唐",
|
||||
"大人",
|
||||
"泥鳅",
|
||||
"幸福",
|
||||
"熟悉",
|
||||
"计划",
|
||||
"扑腾",
|
||||
"蜡烛",
|
||||
"姥爷",
|
||||
"照顾",
|
||||
"喉咙",
|
||||
"吉他",
|
||||
"弄堂",
|
||||
"蚂蚱",
|
||||
"凤凰",
|
||||
"拖沓",
|
||||
"寒碜",
|
||||
"糟蹋",
|
||||
"倒腾",
|
||||
"报复",
|
||||
"逻辑",
|
||||
"盘缠",
|
||||
"喽啰",
|
||||
"牢骚",
|
||||
"咖喱",
|
||||
"扫把",
|
||||
"惦记",
|
||||
}
|
||||
self.must_not_neural_tone_words = {
|
||||
"男子",
|
||||
"女子",
|
||||
"分子",
|
||||
"原子",
|
||||
"量子",
|
||||
"莲子",
|
||||
"石子",
|
||||
"瓜子",
|
||||
"电子",
|
||||
"人人",
|
||||
"虎虎",
|
||||
}
|
||||
self.punc = "：，；。？！“”‘’':,;.?!"
|
||||
|
||||
# the meaning of jieba pos tag: https://blog.csdn.net/weixin_44174352/article/details/113731041
|
||||
# e.g.
|
||||
# word: "家里"
|
||||
# pos: "s"
|
||||
# finals: ['ia1', 'i3']
|
||||
def _neural_sandhi(self, word: str, pos: str, finals: List[str]) -> List[str]:
|
||||
# reduplication words for n. and v. e.g. 奶奶, 试试, 旺旺
|
||||
for j, item in enumerate(word):
|
||||
if (
|
||||
j - 1 >= 0
|
||||
and item == word[j - 1]
|
||||
and pos[0] in {"n", "v", "a"}
|
||||
and word not in self.must_not_neural_tone_words
|
||||
):
|
||||
finals[j] = finals[j][:-1] + "5"
|
||||
ge_idx = word.find("个")
|
||||
if len(word) >= 1 and word[-1] in "吧呢啊呐噻嘛吖嗨呐哦哒额滴哩哟喽啰耶喔诶":
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
elif len(word) >= 1 and word[-1] in "的地得":
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
# e.g. 走了, 看着, 去过
|
||||
# elif len(word) == 1 and word in "了着过" and pos in {"ul", "uz", "ug"}:
|
||||
# finals[-1] = finals[-1][:-1] + "5"
|
||||
elif (
|
||||
len(word) > 1
|
||||
and word[-1] in "们子"
|
||||
and pos in {"r", "n"}
|
||||
and word not in self.must_not_neural_tone_words
|
||||
):
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
# e.g. 桌上, 地下, 家里
|
||||
elif len(word) > 1 and word[-1] in "上下里" and pos in {"s", "l", "f"}:
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
# e.g. 上来, 下去
|
||||
elif len(word) > 1 and word[-1] in "来去" and word[-2] in "上下进出回过起开":
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
# 个做量词
|
||||
elif (
|
||||
ge_idx >= 1
|
||||
and (word[ge_idx - 1].isnumeric() or word[ge_idx - 1] in "几有两半多各整每做是")
|
||||
) or word == "个":
|
||||
finals[ge_idx] = finals[ge_idx][:-1] + "5"
|
||||
else:
|
||||
if (
|
||||
word in self.must_neural_tone_words
|
||||
or word[-2:] in self.must_neural_tone_words
|
||||
):
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
|
||||
word_list = self._split_word(word)
|
||||
finals_list = [finals[: len(word_list[0])], finals[len(word_list[0]) :]]
|
||||
for i, word in enumerate(word_list):
|
||||
# conventional neural in Chinese
|
||||
if (
|
||||
word in self.must_neural_tone_words
|
||||
or word[-2:] in self.must_neural_tone_words
|
||||
):
|
||||
finals_list[i][-1] = finals_list[i][-1][:-1] + "5"
|
||||
finals = sum(finals_list, [])
|
||||
return finals
|
||||
|
||||
def _bu_sandhi(self, word: str, finals: List[str]) -> List[str]:
|
||||
# e.g. 看不懂
|
||||
if len(word) == 3 and word[1] == "不":
|
||||
finals[1] = finals[1][:-1] + "5"
|
||||
else:
|
||||
for i, char in enumerate(word):
|
||||
# "不" before tone4 should be bu2, e.g. 不怕
|
||||
if char == "不" and i + 1 < len(word) and finals[i + 1][-1] == "4":
|
||||
finals[i] = finals[i][:-1] + "2"
|
||||
return finals
|
||||
|
||||
def _yi_sandhi(self, word: str, finals: List[str]) -> List[str]:
|
||||
# "一" in number sequences, e.g. 一零零, 二一零
|
||||
if word.find("一") != -1 and all(
|
||||
[item.isnumeric() for item in word if item != "一"]
|
||||
):
|
||||
return finals
|
||||
# "一" between reduplication words shold be yi5, e.g. 看一看
|
||||
elif len(word) == 3 and word[1] == "一" and word[0] == word[-1]:
|
||||
finals[1] = finals[1][:-1] + "5"
|
||||
# when "一" is ordinal word, it should be yi1
|
||||
elif word.startswith("第一"):
|
||||
finals[1] = finals[1][:-1] + "1"
|
||||
else:
|
||||
for i, char in enumerate(word):
|
||||
if char == "一" and i + 1 < len(word):
|
||||
# "一" before tone4 should be yi2, e.g. 一段
|
||||
if finals[i + 1][-1] == "4":
|
||||
finals[i] = finals[i][:-1] + "2"
|
||||
# "一" before non-tone4 should be yi4, e.g. 一天
|
||||
else:
|
||||
# "一" 后面如果是标点，还读一声
|
||||
if word[i + 1] not in self.punc:
|
||||
finals[i] = finals[i][:-1] + "4"
|
||||
return finals
|
||||
|
||||
def _split_word(self, word: str) -> List[str]:
|
||||
word_list = jieba.cut_for_search(word)
|
||||
word_list = sorted(word_list, key=lambda i: len(i), reverse=False)
|
||||
first_subword = word_list[0]
|
||||
first_begin_idx = word.find(first_subword)
|
||||
if first_begin_idx == 0:
|
||||
second_subword = word[len(first_subword) :]
|
||||
new_word_list = [first_subword, second_subword]
|
||||
else:
|
||||
second_subword = word[: -len(first_subword)]
|
||||
new_word_list = [second_subword, first_subword]
|
||||
return new_word_list
|
||||
|
||||
def _three_sandhi(self, word: str, finals: List[str]) -> List[str]:
|
||||
if len(word) == 2 and self._all_tone_three(finals):
|
||||
finals[0] = finals[0][:-1] + "2"
|
||||
elif len(word) == 3:
|
||||
word_list = self._split_word(word)
|
||||
if self._all_tone_three(finals):
|
||||
# disyllabic + monosyllabic, e.g. 蒙古/包
|
||||
if len(word_list[0]) == 2:
|
||||
finals[0] = finals[0][:-1] + "2"
|
||||
finals[1] = finals[1][:-1] + "2"
|
||||
# monosyllabic + disyllabic, e.g. 纸/老虎
|
||||
elif len(word_list[0]) == 1:
|
||||
finals[1] = finals[1][:-1] + "2"
|
||||
else:
|
||||
finals_list = [finals[: len(word_list[0])], finals[len(word_list[0]) :]]
|
||||
if len(finals_list) == 2:
|
||||
for i, sub in enumerate(finals_list):
|
||||
# e.g. 所有/人
|
||||
if self._all_tone_three(sub) and len(sub) == 2:
|
||||
finals_list[i][0] = finals_list[i][0][:-1] + "2"
|
||||
# e.g. 好/喜欢
|
||||
elif (
|
||||
i == 1
|
||||
and not self._all_tone_three(sub)
|
||||
and finals_list[i][0][-1] == "3"
|
||||
and finals_list[0][-1][-1] == "3"
|
||||
):
|
||||
finals_list[0][-1] = finals_list[0][-1][:-1] + "2"
|
||||
finals = sum(finals_list, [])
|
||||
# split idiom into two words who's length is 2
|
||||
elif len(word) == 4:
|
||||
finals_list = [finals[:2], finals[2:]]
|
||||
finals = []
|
||||
for sub in finals_list:
|
||||
if self._all_tone_three(sub):
|
||||
sub[0] = sub[0][:-1] + "2"
|
||||
finals += sub
|
||||
|
||||
return finals
|
||||
|
||||
def _all_tone_three(self, finals: List[str]) -> bool:
|
||||
return all(x[-1] == "3" for x in finals)
|
||||
|
||||
# merge "不" and the word behind it
|
||||
# if don't merge, "不" sometimes appears alone according to jieba, which may occur sandhi error
|
||||
def _merge_bu(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
last_word = ""
|
||||
for word, pos in seg:
|
||||
if last_word == "不":
|
||||
word = last_word + word
|
||||
if word != "不":
|
||||
new_seg.append((word, pos))
|
||||
last_word = word[:]
|
||||
if last_word == "不":
|
||||
new_seg.append((last_word, "d"))
|
||||
last_word = ""
|
||||
return new_seg
|
||||
|
||||
# function 1: merge "一" and reduplication words in it's left and right, e.g. "听","一","听" ->"听一听"
|
||||
# function 2: merge single "一" and the word behind it
|
||||
# if don't merge, "一" sometimes appears alone according to jieba, which may occur sandhi error
|
||||
# e.g.
|
||||
# input seg: [('听', 'v'), ('一', 'm'), ('听', 'v')]
|
||||
# output seg: [['听一听', 'v']]
|
||||
def _merge_yi(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
# function 1
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if (
|
||||
i - 1 >= 0
|
||||
and word == "一"
|
||||
and i + 1 < len(seg)
|
||||
and seg[i - 1][0] == seg[i + 1][0]
|
||||
and seg[i - 1][1] == "v"
|
||||
):
|
||||
new_seg[i - 1][0] = new_seg[i - 1][0] + "一" + new_seg[i - 1][0]
|
||||
else:
|
||||
if (
|
||||
i - 2 >= 0
|
||||
and seg[i - 1][0] == "一"
|
||||
and seg[i - 2][0] == word
|
||||
and pos == "v"
|
||||
):
|
||||
continue
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
seg = new_seg
|
||||
new_seg = []
|
||||
# function 2
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if new_seg and new_seg[-1][0] == "一":
|
||||
new_seg[-1][0] = new_seg[-1][0] + word
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
return new_seg
|
||||
|
||||
# the first and the second words are all_tone_three
|
||||
def _merge_continuous_three_tones(
|
||||
self, seg: List[Tuple[str, str]]
|
||||
) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
sub_finals_list = [
|
||||
lazy_pinyin(word, neutral_tone_with_five=True, style=Style.FINALS_TONE3)
|
||||
for (word, pos) in seg
|
||||
]
|
||||
assert len(sub_finals_list) == len(seg)
|
||||
merge_last = [False] * len(seg)
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if (
|
||||
i - 1 >= 0
|
||||
and self._all_tone_three(sub_finals_list[i - 1])
|
||||
and self._all_tone_three(sub_finals_list[i])
|
||||
and not merge_last[i - 1]
|
||||
):
|
||||
# if the last word is reduplication, not merge, because reduplication need to be _neural_sandhi
|
||||
if (
|
||||
not self._is_reduplication(seg[i - 1][0])
|
||||
and len(seg[i - 1][0]) + len(seg[i][0]) <= 3
|
||||
):
|
||||
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
|
||||
merge_last[i] = True
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
|
||||
return new_seg
|
||||
|
||||
def _is_reduplication(self, word: str) -> bool:
|
||||
return len(word) == 2 and word[0] == word[1]
|
||||
|
||||
# the last char of first word and the first char of second word is tone_three
|
||||
def _merge_continuous_three_tones_2(
|
||||
self, seg: List[Tuple[str, str]]
|
||||
) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
sub_finals_list = [
|
||||
lazy_pinyin(word, neutral_tone_with_five=True, style=Style.FINALS_TONE3)
|
||||
for (word, pos) in seg
|
||||
]
|
||||
assert len(sub_finals_list) == len(seg)
|
||||
merge_last = [False] * len(seg)
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if (
|
||||
i - 1 >= 0
|
||||
and sub_finals_list[i - 1][-1][-1] == "3"
|
||||
and sub_finals_list[i][0][-1] == "3"
|
||||
and not merge_last[i - 1]
|
||||
):
|
||||
# if the last word is reduplication, not merge, because reduplication need to be _neural_sandhi
|
||||
if (
|
||||
not self._is_reduplication(seg[i - 1][0])
|
||||
and len(seg[i - 1][0]) + len(seg[i][0]) <= 3
|
||||
):
|
||||
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
|
||||
merge_last[i] = True
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
return new_seg
|
||||
|
||||
def _merge_er(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if i - 1 >= 0 and word == "儿" and seg[i - 1][0] != "#":
|
||||
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
return new_seg
|
||||
|
||||
def _merge_reduplication(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if new_seg and word == new_seg[-1][0]:
|
||||
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
return new_seg
|
||||
|
||||
def pre_merge_for_modify(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
seg = self._merge_bu(seg)
|
||||
try:
|
||||
seg = self._merge_yi(seg)
|
||||
except:
|
||||
print("_merge_yi failed")
|
||||
seg = self._merge_reduplication(seg)
|
||||
seg = self._merge_continuous_three_tones(seg)
|
||||
seg = self._merge_continuous_three_tones_2(seg)
|
||||
seg = self._merge_er(seg)
|
||||
return seg
|
||||
|
||||
def modified_tone(self, word: str, pos: str, finals: List[str]) -> List[str]:
|
||||
finals = self._bu_sandhi(word, finals)
|
||||
finals = self._yi_sandhi(word, finals)
|
||||
finals = self._neural_sandhi(word, pos, finals)
|
||||
finals = self._three_sandhi(word, finals)
|
||||
return finals
|
||||
90
oldVersion/V110/__init__.py
Normal file
90
oldVersion/V110/__init__.py
Normal file
@@ -0,0 +1,90 @@
|
||||
"""
|
||||
1.1 版本兼容
|
||||
https://github.com/fishaudio/Bert-VITS2/releases/tag/1.1
|
||||
"""
|
||||
import torch
|
||||
import commons
|
||||
from .text.cleaner import clean_text
|
||||
from .text import cleaned_text_to_sequence
|
||||
from oldVersion.V111.text import get_bert
|
||||
|
||||
|
||||
def get_text(text, language_str, hps, device):
|
||||
norm_text, phone, tone, word2ph = clean_text(text, language_str)
|
||||
phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
|
||||
|
||||
if hps.data.add_blank:
|
||||
phone = commons.intersperse(phone, 0)
|
||||
tone = commons.intersperse(tone, 0)
|
||||
language = commons.intersperse(language, 0)
|
||||
for i in range(len(word2ph)):
|
||||
word2ph[i] = word2ph[i] * 2
|
||||
word2ph[0] += 1
|
||||
bert = get_bert(norm_text, word2ph, language_str, device)
|
||||
del word2ph
|
||||
assert bert.shape[-1] == len(phone), phone
|
||||
|
||||
if language_str == "ZH":
|
||||
bert = bert
|
||||
ja_bert = torch.zeros(768, len(phone))
|
||||
elif language_str == "JP":
|
||||
ja_bert = bert
|
||||
bert = torch.zeros(1024, len(phone))
|
||||
else:
|
||||
bert = torch.zeros(1024, len(phone))
|
||||
ja_bert = torch.zeros(768, len(phone))
|
||||
|
||||
assert bert.shape[-1] == len(
|
||||
phone
|
||||
), f"Bert seq len {bert.shape[-1]} != {len(phone)}"
|
||||
|
||||
phone = torch.LongTensor(phone)
|
||||
tone = torch.LongTensor(tone)
|
||||
language = torch.LongTensor(language)
|
||||
return bert, ja_bert, phone, tone, language
|
||||
|
||||
|
||||
def infer(
|
||||
text,
|
||||
sdp_ratio,
|
||||
noise_scale,
|
||||
noise_scale_w,
|
||||
length_scale,
|
||||
sid,
|
||||
language,
|
||||
hps,
|
||||
net_g,
|
||||
device,
|
||||
):
|
||||
bert, ja_bert, phones, tones, lang_ids = get_text(text, language, hps, device)
|
||||
with torch.no_grad():
|
||||
x_tst = phones.to(device).unsqueeze(0)
|
||||
tones = tones.to(device).unsqueeze(0)
|
||||
lang_ids = lang_ids.to(device).unsqueeze(0)
|
||||
bert = bert.to(device).unsqueeze(0)
|
||||
ja_bert = ja_bert.to(device).unsqueeze(0)
|
||||
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
|
||||
del phones
|
||||
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
|
||||
audio = (
|
||||
net_g.infer(
|
||||
x_tst,
|
||||
x_tst_lengths,
|
||||
speakers,
|
||||
tones,
|
||||
lang_ids,
|
||||
bert,
|
||||
ja_bert,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise_scale,
|
||||
noise_scale_w=noise_scale_w,
|
||||
length_scale=length_scale,
|
||||
)[0][0, 0]
|
||||
.data.cpu()
|
||||
.float()
|
||||
.numpy()
|
||||
)
|
||||
del x_tst, x_tst_lengths, speakers, tones, lang_ids, bert, ja_bert
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
return audio
|
||||
986
oldVersion/V110/models.py
Normal file
986
oldVersion/V110/models.py
Normal file
@@ -0,0 +1,986 @@
|
||||
import math
|
||||
import torch
|
||||
from torch import nn
|
||||
from torch.nn import functional as F
|
||||
|
||||
import commons
|
||||
import modules
|
||||
import attentions
|
||||
import monotonic_align
|
||||
|
||||
from torch.nn import Conv1d, ConvTranspose1d, Conv2d
|
||||
from torch.nn.utils import weight_norm, remove_weight_norm, spectral_norm
|
||||
|
||||
from commons import init_weights, get_padding
|
||||
from .text import symbols, num_tones, num_languages
|
||||
|
||||
|
||||
class DurationDiscriminator(nn.Module): # vits2
|
||||
def __init__(
|
||||
self, in_channels, filter_channels, kernel_size, p_dropout, gin_channels=0
|
||||
):
|
||||
super().__init__()
|
||||
|
||||
self.in_channels = in_channels
|
||||
self.filter_channels = filter_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.p_dropout = p_dropout
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.drop = nn.Dropout(p_dropout)
|
||||
self.conv_1 = nn.Conv1d(
|
||||
in_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.norm_1 = modules.LayerNorm(filter_channels)
|
||||
self.conv_2 = nn.Conv1d(
|
||||
filter_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.norm_2 = modules.LayerNorm(filter_channels)
|
||||
self.dur_proj = nn.Conv1d(1, filter_channels, 1)
|
||||
|
||||
self.pre_out_conv_1 = nn.Conv1d(
|
||||
2 * filter_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.pre_out_norm_1 = modules.LayerNorm(filter_channels)
|
||||
self.pre_out_conv_2 = nn.Conv1d(
|
||||
filter_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.pre_out_norm_2 = modules.LayerNorm(filter_channels)
|
||||
|
||||
if gin_channels != 0:
|
||||
self.cond = nn.Conv1d(gin_channels, in_channels, 1)
|
||||
|
||||
self.output_layer = nn.Sequential(nn.Linear(filter_channels, 1), nn.Sigmoid())
|
||||
|
||||
def forward_probability(self, x, x_mask, dur, g=None):
|
||||
dur = self.dur_proj(dur)
|
||||
x = torch.cat([x, dur], dim=1)
|
||||
x = self.pre_out_conv_1(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.pre_out_norm_1(x)
|
||||
x = self.drop(x)
|
||||
x = self.pre_out_conv_2(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.pre_out_norm_2(x)
|
||||
x = self.drop(x)
|
||||
x = x * x_mask
|
||||
x = x.transpose(1, 2)
|
||||
output_prob = self.output_layer(x)
|
||||
return output_prob
|
||||
|
||||
def forward(self, x, x_mask, dur_r, dur_hat, g=None):
|
||||
x = torch.detach(x)
|
||||
if g is not None:
|
||||
g = torch.detach(g)
|
||||
x = x + self.cond(g)
|
||||
x = self.conv_1(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.norm_1(x)
|
||||
x = self.drop(x)
|
||||
x = self.conv_2(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.norm_2(x)
|
||||
x = self.drop(x)
|
||||
|
||||
output_probs = []
|
||||
for dur in [dur_r, dur_hat]:
|
||||
output_prob = self.forward_probability(x, x_mask, dur, g)
|
||||
output_probs.append(output_prob)
|
||||
|
||||
return output_probs
|
||||
|
||||
|
||||
class TransformerCouplingBlock(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
channels,
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
n_flows=4,
|
||||
gin_channels=0,
|
||||
share_parameter=False,
|
||||
):
|
||||
super().__init__()
|
||||
self.channels = channels
|
||||
self.hidden_channels = hidden_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.n_layers = n_layers
|
||||
self.n_flows = n_flows
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.flows = nn.ModuleList()
|
||||
|
||||
self.wn = (
|
||||
attentions.FFT(
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
isflow=True,
|
||||
gin_channels=self.gin_channels,
|
||||
)
|
||||
if share_parameter
|
||||
else None
|
||||
)
|
||||
|
||||
for i in range(n_flows):
|
||||
self.flows.append(
|
||||
modules.TransformerCouplingLayer(
|
||||
channels,
|
||||
hidden_channels,
|
||||
kernel_size,
|
||||
n_layers,
|
||||
n_heads,
|
||||
p_dropout,
|
||||
filter_channels,
|
||||
mean_only=True,
|
||||
wn_sharing_parameter=self.wn,
|
||||
gin_channels=self.gin_channels,
|
||||
)
|
||||
)
|
||||
self.flows.append(modules.Flip())
|
||||
|
||||
def forward(self, x, x_mask, g=None, reverse=False):
|
||||
if not reverse:
|
||||
for flow in self.flows:
|
||||
x, _ = flow(x, x_mask, g=g, reverse=reverse)
|
||||
else:
|
||||
for flow in reversed(self.flows):
|
||||
x = flow(x, x_mask, g=g, reverse=reverse)
|
||||
return x
|
||||
|
||||
|
||||
class StochasticDurationPredictor(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
in_channels,
|
||||
filter_channels,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
n_flows=4,
|
||||
gin_channels=0,
|
||||
):
|
||||
super().__init__()
|
||||
filter_channels = in_channels # it needs to be removed from future version.
|
||||
self.in_channels = in_channels
|
||||
self.filter_channels = filter_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.p_dropout = p_dropout
|
||||
self.n_flows = n_flows
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.log_flow = modules.Log()
|
||||
self.flows = nn.ModuleList()
|
||||
self.flows.append(modules.ElementwiseAffine(2))
|
||||
for i in range(n_flows):
|
||||
self.flows.append(
|
||||
modules.ConvFlow(2, filter_channels, kernel_size, n_layers=3)
|
||||
)
|
||||
self.flows.append(modules.Flip())
|
||||
|
||||
self.post_pre = nn.Conv1d(1, filter_channels, 1)
|
||||
self.post_proj = nn.Conv1d(filter_channels, filter_channels, 1)
|
||||
self.post_convs = modules.DDSConv(
|
||||
filter_channels, kernel_size, n_layers=3, p_dropout=p_dropout
|
||||
)
|
||||
self.post_flows = nn.ModuleList()
|
||||
self.post_flows.append(modules.ElementwiseAffine(2))
|
||||
for i in range(4):
|
||||
self.post_flows.append(
|
||||
modules.ConvFlow(2, filter_channels, kernel_size, n_layers=3)
|
||||
)
|
||||
self.post_flows.append(modules.Flip())
|
||||
|
||||
self.pre = nn.Conv1d(in_channels, filter_channels, 1)
|
||||
self.proj = nn.Conv1d(filter_channels, filter_channels, 1)
|
||||
self.convs = modules.DDSConv(
|
||||
filter_channels, kernel_size, n_layers=3, p_dropout=p_dropout
|
||||
)
|
||||
if gin_channels != 0:
|
||||
self.cond = nn.Conv1d(gin_channels, filter_channels, 1)
|
||||
|
||||
def forward(self, x, x_mask, w=None, g=None, reverse=False, noise_scale=1.0):
|
||||
x = torch.detach(x)
|
||||
x = self.pre(x)
|
||||
if g is not None:
|
||||
g = torch.detach(g)
|
||||
x = x + self.cond(g)
|
||||
x = self.convs(x, x_mask)
|
||||
x = self.proj(x) * x_mask
|
||||
|
||||
if not reverse:
|
||||
flows = self.flows
|
||||
assert w is not None
|
||||
|
||||
logdet_tot_q = 0
|
||||
h_w = self.post_pre(w)
|
||||
h_w = self.post_convs(h_w, x_mask)
|
||||
h_w = self.post_proj(h_w) * x_mask
|
||||
e_q = (
|
||||
torch.randn(w.size(0), 2, w.size(2)).to(device=x.device, dtype=x.dtype)
|
||||
* x_mask
|
||||
)
|
||||
z_q = e_q
|
||||
for flow in self.post_flows:
|
||||
z_q, logdet_q = flow(z_q, x_mask, g=(x + h_w))
|
||||
logdet_tot_q += logdet_q
|
||||
z_u, z1 = torch.split(z_q, [1, 1], 1)
|
||||
u = torch.sigmoid(z_u) * x_mask
|
||||
z0 = (w - u) * x_mask
|
||||
logdet_tot_q += torch.sum(
|
||||
(F.logsigmoid(z_u) + F.logsigmoid(-z_u)) * x_mask, [1, 2]
|
||||
)
|
||||
logq = (
|
||||
torch.sum(-0.5 * (math.log(2 * math.pi) + (e_q**2)) * x_mask, [1, 2])
|
||||
- logdet_tot_q
|
||||
)
|
||||
|
||||
logdet_tot = 0
|
||||
z0, logdet = self.log_flow(z0, x_mask)
|
||||
logdet_tot += logdet
|
||||
z = torch.cat([z0, z1], 1)
|
||||
for flow in flows:
|
||||
z, logdet = flow(z, x_mask, g=x, reverse=reverse)
|
||||
logdet_tot = logdet_tot + logdet
|
||||
nll = (
|
||||
torch.sum(0.5 * (math.log(2 * math.pi) + (z**2)) * x_mask, [1, 2])
|
||||
- logdet_tot
|
||||
)
|
||||
return nll + logq # [b]
|
||||
else:
|
||||
flows = list(reversed(self.flows))
|
||||
flows = flows[:-2] + [flows[-1]] # remove a useless vflow
|
||||
z = (
|
||||
torch.randn(x.size(0), 2, x.size(2)).to(device=x.device, dtype=x.dtype)
|
||||
* noise_scale
|
||||
)
|
||||
for flow in flows:
|
||||
z = flow(z, x_mask, g=x, reverse=reverse)
|
||||
z0, z1 = torch.split(z, [1, 1], 1)
|
||||
logw = z0
|
||||
return logw
|
||||
|
||||
|
||||
class DurationPredictor(nn.Module):
|
||||
def __init__(
|
||||
self, in_channels, filter_channels, kernel_size, p_dropout, gin_channels=0
|
||||
):
|
||||
super().__init__()
|
||||
|
||||
self.in_channels = in_channels
|
||||
self.filter_channels = filter_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.p_dropout = p_dropout
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.drop = nn.Dropout(p_dropout)
|
||||
self.conv_1 = nn.Conv1d(
|
||||
in_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.norm_1 = modules.LayerNorm(filter_channels)
|
||||
self.conv_2 = nn.Conv1d(
|
||||
filter_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.norm_2 = modules.LayerNorm(filter_channels)
|
||||
self.proj = nn.Conv1d(filter_channels, 1, 1)
|
||||
|
||||
if gin_channels != 0:
|
||||
self.cond = nn.Conv1d(gin_channels, in_channels, 1)
|
||||
|
||||
def forward(self, x, x_mask, g=None):
|
||||
x = torch.detach(x)
|
||||
if g is not None:
|
||||
g = torch.detach(g)
|
||||
x = x + self.cond(g)
|
||||
x = self.conv_1(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.norm_1(x)
|
||||
x = self.drop(x)
|
||||
x = self.conv_2(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.norm_2(x)
|
||||
x = self.drop(x)
|
||||
x = self.proj(x * x_mask)
|
||||
return x * x_mask
|
||||
|
||||
|
||||
class TextEncoder(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
n_vocab,
|
||||
out_channels,
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
gin_channels=0,
|
||||
):
|
||||
super().__init__()
|
||||
self.n_vocab = n_vocab
|
||||
self.out_channels = out_channels
|
||||
self.hidden_channels = hidden_channels
|
||||
self.filter_channels = filter_channels
|
||||
self.n_heads = n_heads
|
||||
self.n_layers = n_layers
|
||||
self.kernel_size = kernel_size
|
||||
self.p_dropout = p_dropout
|
||||
self.gin_channels = gin_channels
|
||||
self.emb = nn.Embedding(len(symbols), hidden_channels)
|
||||
nn.init.normal_(self.emb.weight, 0.0, hidden_channels**-0.5)
|
||||
self.tone_emb = nn.Embedding(num_tones, hidden_channels)
|
||||
nn.init.normal_(self.tone_emb.weight, 0.0, hidden_channels**-0.5)
|
||||
self.language_emb = nn.Embedding(num_languages, hidden_channels)
|
||||
nn.init.normal_(self.language_emb.weight, 0.0, hidden_channels**-0.5)
|
||||
self.bert_proj = nn.Conv1d(1024, hidden_channels, 1)
|
||||
self.ja_bert_proj = nn.Conv1d(768, hidden_channels, 1)
|
||||
|
||||
self.encoder = attentions.Encoder(
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
gin_channels=self.gin_channels,
|
||||
)
|
||||
self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
|
||||
|
||||
def forward(self, x, x_lengths, tone, language, bert, ja_bert, g=None):
|
||||
bert_emb = self.bert_proj(bert).transpose(1, 2)
|
||||
ja_bert_emb = self.ja_bert_proj(ja_bert).transpose(1, 2)
|
||||
x = (
|
||||
self.emb(x)
|
||||
+ self.tone_emb(tone)
|
||||
+ self.language_emb(language)
|
||||
+ bert_emb
|
||||
+ ja_bert_emb
|
||||
) * math.sqrt(
|
||||
self.hidden_channels
|
||||
) # [b, t, h]
|
||||
x = torch.transpose(x, 1, -1) # [b, h, t]
|
||||
x_mask = torch.unsqueeze(commons.sequence_mask(x_lengths, x.size(2)), 1).to(
|
||||
x.dtype
|
||||
)
|
||||
|
||||
x = self.encoder(x * x_mask, x_mask, g=g)
|
||||
stats = self.proj(x) * x_mask
|
||||
|
||||
m, logs = torch.split(stats, self.out_channels, dim=1)
|
||||
return x, m, logs, x_mask
|
||||
|
||||
|
||||
class ResidualCouplingBlock(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
channels,
|
||||
hidden_channels,
|
||||
kernel_size,
|
||||
dilation_rate,
|
||||
n_layers,
|
||||
n_flows=4,
|
||||
gin_channels=0,
|
||||
):
|
||||
super().__init__()
|
||||
self.channels = channels
|
||||
self.hidden_channels = hidden_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.dilation_rate = dilation_rate
|
||||
self.n_layers = n_layers
|
||||
self.n_flows = n_flows
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.flows = nn.ModuleList()
|
||||
for i in range(n_flows):
|
||||
self.flows.append(
|
||||
modules.ResidualCouplingLayer(
|
||||
channels,
|
||||
hidden_channels,
|
||||
kernel_size,
|
||||
dilation_rate,
|
||||
n_layers,
|
||||
gin_channels=gin_channels,
|
||||
mean_only=True,
|
||||
)
|
||||
)
|
||||
self.flows.append(modules.Flip())
|
||||
|
||||
def forward(self, x, x_mask, g=None, reverse=False):
|
||||
if not reverse:
|
||||
for flow in self.flows:
|
||||
x, _ = flow(x, x_mask, g=g, reverse=reverse)
|
||||
else:
|
||||
for flow in reversed(self.flows):
|
||||
x = flow(x, x_mask, g=g, reverse=reverse)
|
||||
return x
|
||||
|
||||
|
||||
class PosteriorEncoder(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
in_channels,
|
||||
out_channels,
|
||||
hidden_channels,
|
||||
kernel_size,
|
||||
dilation_rate,
|
||||
n_layers,
|
||||
gin_channels=0,
|
||||
):
|
||||
super().__init__()
|
||||
self.in_channels = in_channels
|
||||
self.out_channels = out_channels
|
||||
self.hidden_channels = hidden_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.dilation_rate = dilation_rate
|
||||
self.n_layers = n_layers
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.pre = nn.Conv1d(in_channels, hidden_channels, 1)
|
||||
self.enc = modules.WN(
|
||||
hidden_channels,
|
||||
kernel_size,
|
||||
dilation_rate,
|
||||
n_layers,
|
||||
gin_channels=gin_channels,
|
||||
)
|
||||
self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
|
||||
|
||||
def forward(self, x, x_lengths, g=None):
|
||||
x_mask = torch.unsqueeze(commons.sequence_mask(x_lengths, x.size(2)), 1).to(
|
||||
x.dtype
|
||||
)
|
||||
x = self.pre(x) * x_mask
|
||||
x = self.enc(x, x_mask, g=g)
|
||||
stats = self.proj(x) * x_mask
|
||||
m, logs = torch.split(stats, self.out_channels, dim=1)
|
||||
z = (m + torch.randn_like(m) * torch.exp(logs)) * x_mask
|
||||
return z, m, logs, x_mask
|
||||
|
||||
|
||||
class Generator(torch.nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
initial_channel,
|
||||
resblock,
|
||||
resblock_kernel_sizes,
|
||||
resblock_dilation_sizes,
|
||||
upsample_rates,
|
||||
upsample_initial_channel,
|
||||
upsample_kernel_sizes,
|
||||
gin_channels=0,
|
||||
):
|
||||
super(Generator, self).__init__()
|
||||
self.num_kernels = len(resblock_kernel_sizes)
|
||||
self.num_upsamples = len(upsample_rates)
|
||||
self.conv_pre = Conv1d(
|
||||
initial_channel, upsample_initial_channel, 7, 1, padding=3
|
||||
)
|
||||
resblock = modules.ResBlock1 if resblock == "1" else modules.ResBlock2
|
||||
|
||||
self.ups = nn.ModuleList()
|
||||
for i, (u, k) in enumerate(zip(upsample_rates, upsample_kernel_sizes)):
|
||||
self.ups.append(
|
||||
weight_norm(
|
||||
ConvTranspose1d(
|
||||
upsample_initial_channel // (2**i),
|
||||
upsample_initial_channel // (2 ** (i + 1)),
|
||||
k,
|
||||
u,
|
||||
padding=(k - u) // 2,
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
self.resblocks = nn.ModuleList()
|
||||
for i in range(len(self.ups)):
|
||||
ch = upsample_initial_channel // (2 ** (i + 1))
|
||||
for j, (k, d) in enumerate(
|
||||
zip(resblock_kernel_sizes, resblock_dilation_sizes)
|
||||
):
|
||||
self.resblocks.append(resblock(ch, k, d))
|
||||
|
||||
self.conv_post = Conv1d(ch, 1, 7, 1, padding=3, bias=False)
|
||||
self.ups.apply(init_weights)
|
||||
|
||||
if gin_channels != 0:
|
||||
self.cond = nn.Conv1d(gin_channels, upsample_initial_channel, 1)
|
||||
|
||||
def forward(self, x, g=None):
|
||||
x = self.conv_pre(x)
|
||||
if g is not None:
|
||||
x = x + self.cond(g)
|
||||
|
||||
for i in range(self.num_upsamples):
|
||||
x = F.leaky_relu(x, modules.LRELU_SLOPE)
|
||||
x = self.ups[i](x)
|
||||
xs = None
|
||||
for j in range(self.num_kernels):
|
||||
if xs is None:
|
||||
xs = self.resblocks[i * self.num_kernels + j](x)
|
||||
else:
|
||||
xs += self.resblocks[i * self.num_kernels + j](x)
|
||||
x = xs / self.num_kernels
|
||||
x = F.leaky_relu(x)
|
||||
x = self.conv_post(x)
|
||||
x = torch.tanh(x)
|
||||
|
||||
return x
|
||||
|
||||
def remove_weight_norm(self):
|
||||
print("Removing weight norm...")
|
||||
for layer in self.ups:
|
||||
remove_weight_norm(layer)
|
||||
for layer in self.resblocks:
|
||||
layer.remove_weight_norm()
|
||||
|
||||
|
||||
class DiscriminatorP(torch.nn.Module):
|
||||
def __init__(self, period, kernel_size=5, stride=3, use_spectral_norm=False):
|
||||
super(DiscriminatorP, self).__init__()
|
||||
self.period = period
|
||||
self.use_spectral_norm = use_spectral_norm
|
||||
norm_f = weight_norm if use_spectral_norm is False else spectral_norm
|
||||
self.convs = nn.ModuleList(
|
||||
[
|
||||
norm_f(
|
||||
Conv2d(
|
||||
1,
|
||||
32,
|
||||
(kernel_size, 1),
|
||||
(stride, 1),
|
||||
padding=(get_padding(kernel_size, 1), 0),
|
||||
)
|
||||
),
|
||||
norm_f(
|
||||
Conv2d(
|
||||
32,
|
||||
128,
|
||||
(kernel_size, 1),
|
||||
(stride, 1),
|
||||
padding=(get_padding(kernel_size, 1), 0),
|
||||
)
|
||||
),
|
||||
norm_f(
|
||||
Conv2d(
|
||||
128,
|
||||
512,
|
||||
(kernel_size, 1),
|
||||
(stride, 1),
|
||||
padding=(get_padding(kernel_size, 1), 0),
|
||||
)
|
||||
),
|
||||
norm_f(
|
||||
Conv2d(
|
||||
512,
|
||||
1024,
|
||||
(kernel_size, 1),
|
||||
(stride, 1),
|
||||
padding=(get_padding(kernel_size, 1), 0),
|
||||
)
|
||||
),
|
||||
norm_f(
|
||||
Conv2d(
|
||||
1024,
|
||||
1024,
|
||||
(kernel_size, 1),
|
||||
1,
|
||||
padding=(get_padding(kernel_size, 1), 0),
|
||||
)
|
||||
),
|
||||
]
|
||||
)
|
||||
self.conv_post = norm_f(Conv2d(1024, 1, (3, 1), 1, padding=(1, 0)))
|
||||
|
||||
def forward(self, x):
|
||||
fmap = []
|
||||
|
||||
# 1d to 2d
|
||||
b, c, t = x.shape
|
||||
if t % self.period != 0: # pad first
|
||||
n_pad = self.period - (t % self.period)
|
||||
x = F.pad(x, (0, n_pad), "reflect")
|
||||
t = t + n_pad
|
||||
x = x.view(b, c, t // self.period, self.period)
|
||||
|
||||
for layer in self.convs:
|
||||
x = layer(x)
|
||||
x = F.leaky_relu(x, modules.LRELU_SLOPE)
|
||||
fmap.append(x)
|
||||
x = self.conv_post(x)
|
||||
fmap.append(x)
|
||||
x = torch.flatten(x, 1, -1)
|
||||
|
||||
return x, fmap
|
||||
|
||||
|
||||
class DiscriminatorS(torch.nn.Module):
|
||||
def __init__(self, use_spectral_norm=False):
|
||||
super(DiscriminatorS, self).__init__()
|
||||
norm_f = weight_norm if use_spectral_norm is False else spectral_norm
|
||||
self.convs = nn.ModuleList(
|
||||
[
|
||||
norm_f(Conv1d(1, 16, 15, 1, padding=7)),
|
||||
norm_f(Conv1d(16, 64, 41, 4, groups=4, padding=20)),
|
||||
norm_f(Conv1d(64, 256, 41, 4, groups=16, padding=20)),
|
||||
norm_f(Conv1d(256, 1024, 41, 4, groups=64, padding=20)),
|
||||
norm_f(Conv1d(1024, 1024, 41, 4, groups=256, padding=20)),
|
||||
norm_f(Conv1d(1024, 1024, 5, 1, padding=2)),
|
||||
]
|
||||
)
|
||||
self.conv_post = norm_f(Conv1d(1024, 1, 3, 1, padding=1))
|
||||
|
||||
def forward(self, x):
|
||||
fmap = []
|
||||
|
||||
for layer in self.convs:
|
||||
x = layer(x)
|
||||
x = F.leaky_relu(x, modules.LRELU_SLOPE)
|
||||
fmap.append(x)
|
||||
x = self.conv_post(x)
|
||||
fmap.append(x)
|
||||
x = torch.flatten(x, 1, -1)
|
||||
|
||||
return x, fmap
|
||||
|
||||
|
||||
class MultiPeriodDiscriminator(torch.nn.Module):
|
||||
def __init__(self, use_spectral_norm=False):
|
||||
super(MultiPeriodDiscriminator, self).__init__()
|
||||
periods = [2, 3, 5, 7, 11]
|
||||
|
||||
discs = [DiscriminatorS(use_spectral_norm=use_spectral_norm)]
|
||||
discs = discs + [
|
||||
DiscriminatorP(i, use_spectral_norm=use_spectral_norm) for i in periods
|
||||
]
|
||||
self.discriminators = nn.ModuleList(discs)
|
||||
|
||||
def forward(self, y, y_hat):
|
||||
y_d_rs = []
|
||||
y_d_gs = []
|
||||
fmap_rs = []
|
||||
fmap_gs = []
|
||||
for i, d in enumerate(self.discriminators):
|
||||
y_d_r, fmap_r = d(y)
|
||||
y_d_g, fmap_g = d(y_hat)
|
||||
y_d_rs.append(y_d_r)
|
||||
y_d_gs.append(y_d_g)
|
||||
fmap_rs.append(fmap_r)
|
||||
fmap_gs.append(fmap_g)
|
||||
|
||||
return y_d_rs, y_d_gs, fmap_rs, fmap_gs
|
||||
|
||||
|
||||
class ReferenceEncoder(nn.Module):
|
||||
"""
|
||||
inputs --- [N, Ty/r, n_mels*r] mels
|
||||
outputs --- [N, ref_enc_gru_size]
|
||||
"""
|
||||
|
||||
def __init__(self, spec_channels, gin_channels=0):
|
||||
super().__init__()
|
||||
self.spec_channels = spec_channels
|
||||
ref_enc_filters = [32, 32, 64, 64, 128, 128]
|
||||
K = len(ref_enc_filters)
|
||||
filters = [1] + ref_enc_filters
|
||||
convs = [
|
||||
weight_norm(
|
||||
nn.Conv2d(
|
||||
in_channels=filters[i],
|
||||
out_channels=filters[i + 1],
|
||||
kernel_size=(3, 3),
|
||||
stride=(2, 2),
|
||||
padding=(1, 1),
|
||||
)
|
||||
)
|
||||
for i in range(K)
|
||||
]
|
||||
self.convs = nn.ModuleList(convs)
|
||||
# self.wns = nn.ModuleList([weight_norm(num_features=ref_enc_filters[i]) for i in range(K)]) # noqa: E501
|
||||
|
||||
out_channels = self.calculate_channels(spec_channels, 3, 2, 1, K)
|
||||
self.gru = nn.GRU(
|
||||
input_size=ref_enc_filters[-1] * out_channels,
|
||||
hidden_size=256 // 2,
|
||||
batch_first=True,
|
||||
)
|
||||
self.proj = nn.Linear(128, gin_channels)
|
||||
|
||||
def forward(self, inputs, mask=None):
|
||||
N = inputs.size(0)
|
||||
out = inputs.view(N, 1, -1, self.spec_channels) # [N, 1, Ty, n_freqs]
|
||||
for conv in self.convs:
|
||||
out = conv(out)
|
||||
# out = wn(out)
|
||||
out = F.relu(out) # [N, 128, Ty//2^K, n_mels//2^K]
|
||||
|
||||
out = out.transpose(1, 2) # [N, Ty//2^K, 128, n_mels//2^K]
|
||||
T = out.size(1)
|
||||
N = out.size(0)
|
||||
out = out.contiguous().view(N, T, -1) # [N, Ty//2^K, 128*n_mels//2^K]
|
||||
|
||||
self.gru.flatten_parameters()
|
||||
memory, out = self.gru(out) # out --- [1, N, 128]
|
||||
|
||||
return self.proj(out.squeeze(0))
|
||||
|
||||
def calculate_channels(self, L, kernel_size, stride, pad, n_convs):
|
||||
for i in range(n_convs):
|
||||
L = (L - kernel_size + 2 * pad) // stride + 1
|
||||
return L
|
||||
|
||||
|
||||
class SynthesizerTrn(nn.Module):
|
||||
"""
|
||||
Synthesizer for Training
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
n_vocab,
|
||||
spec_channels,
|
||||
segment_size,
|
||||
inter_channels,
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
resblock,
|
||||
resblock_kernel_sizes,
|
||||
resblock_dilation_sizes,
|
||||
upsample_rates,
|
||||
upsample_initial_channel,
|
||||
upsample_kernel_sizes,
|
||||
n_speakers=256,
|
||||
gin_channels=256,
|
||||
use_sdp=True,
|
||||
n_flow_layer=4,
|
||||
n_layers_trans_flow=6,
|
||||
flow_share_parameter=False,
|
||||
use_transformer_flow=True,
|
||||
**kwargs
|
||||
):
|
||||
super().__init__()
|
||||
self.n_vocab = n_vocab
|
||||
self.spec_channels = spec_channels
|
||||
self.inter_channels = inter_channels
|
||||
self.hidden_channels = hidden_channels
|
||||
self.filter_channels = filter_channels
|
||||
self.n_heads = n_heads
|
||||
self.n_layers = n_layers
|
||||
self.kernel_size = kernel_size
|
||||
self.p_dropout = p_dropout
|
||||
self.resblock = resblock
|
||||
self.resblock_kernel_sizes = resblock_kernel_sizes
|
||||
self.resblock_dilation_sizes = resblock_dilation_sizes
|
||||
self.upsample_rates = upsample_rates
|
||||
self.upsample_initial_channel = upsample_initial_channel
|
||||
self.upsample_kernel_sizes = upsample_kernel_sizes
|
||||
self.segment_size = segment_size
|
||||
self.n_speakers = n_speakers
|
||||
self.gin_channels = gin_channels
|
||||
self.n_layers_trans_flow = n_layers_trans_flow
|
||||
self.use_spk_conditioned_encoder = kwargs.get(
|
||||
"use_spk_conditioned_encoder", True
|
||||
)
|
||||
self.use_sdp = use_sdp
|
||||
self.use_noise_scaled_mas = kwargs.get("use_noise_scaled_mas", False)
|
||||
self.mas_noise_scale_initial = kwargs.get("mas_noise_scale_initial", 0.01)
|
||||
self.noise_scale_delta = kwargs.get("noise_scale_delta", 2e-6)
|
||||
self.current_mas_noise_scale = self.mas_noise_scale_initial
|
||||
if self.use_spk_conditioned_encoder and gin_channels > 0:
|
||||
self.enc_gin_channels = gin_channels
|
||||
self.enc_p = TextEncoder(
|
||||
n_vocab,
|
||||
inter_channels,
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
gin_channels=self.enc_gin_channels,
|
||||
)
|
||||
self.dec = Generator(
|
||||
inter_channels,
|
||||
resblock,
|
||||
resblock_kernel_sizes,
|
||||
resblock_dilation_sizes,
|
||||
upsample_rates,
|
||||
upsample_initial_channel,
|
||||
upsample_kernel_sizes,
|
||||
gin_channels=gin_channels,
|
||||
)
|
||||
self.enc_q = PosteriorEncoder(
|
||||
spec_channels,
|
||||
inter_channels,
|
||||
hidden_channels,
|
||||
5,
|
||||
1,
|
||||
16,
|
||||
gin_channels=gin_channels,
|
||||
)
|
||||
if use_transformer_flow:
|
||||
self.flow = TransformerCouplingBlock(
|
||||
inter_channels,
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers_trans_flow,
|
||||
5,
|
||||
p_dropout,
|
||||
n_flow_layer,
|
||||
gin_channels=gin_channels,
|
||||
share_parameter=flow_share_parameter,
|
||||
)
|
||||
else:
|
||||
self.flow = ResidualCouplingBlock(
|
||||
inter_channels,
|
||||
hidden_channels,
|
||||
5,
|
||||
1,
|
||||
n_flow_layer,
|
||||
gin_channels=gin_channels,
|
||||
)
|
||||
self.sdp = StochasticDurationPredictor(
|
||||
hidden_channels, 192, 3, 0.5, 4, gin_channels=gin_channels
|
||||
)
|
||||
self.dp = DurationPredictor(
|
||||
hidden_channels, 256, 3, 0.5, gin_channels=gin_channels
|
||||
)
|
||||
|
||||
if n_speakers > 0:
|
||||
self.emb_g = nn.Embedding(n_speakers, gin_channels)
|
||||
else:
|
||||
self.ref_enc = ReferenceEncoder(spec_channels, gin_channels)
|
||||
|
||||
def forward(self, x, x_lengths, y, y_lengths, sid, tone, language, bert, ja_bert):
|
||||
if self.n_speakers > 0:
|
||||
g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
|
||||
else:
|
||||
g = self.ref_enc(y.transpose(1, 2)).unsqueeze(-1)
|
||||
x, m_p, logs_p, x_mask = self.enc_p(
|
||||
x, x_lengths, tone, language, bert, ja_bert, g=g
|
||||
)
|
||||
z, m_q, logs_q, y_mask = self.enc_q(y, y_lengths, g=g)
|
||||
z_p = self.flow(z, y_mask, g=g)
|
||||
|
||||
with torch.no_grad():
|
||||
# negative cross-entropy
|
||||
s_p_sq_r = torch.exp(-2 * logs_p) # [b, d, t]
|
||||
neg_cent1 = torch.sum(
|
||||
-0.5 * math.log(2 * math.pi) - logs_p, [1], keepdim=True
|
||||
) # [b, 1, t_s]
|
||||
neg_cent2 = torch.matmul(
|
||||
-0.5 * (z_p**2).transpose(1, 2), s_p_sq_r
|
||||
) # [b, t_t, d] x [b, d, t_s] = [b, t_t, t_s]
|
||||
neg_cent3 = torch.matmul(
|
||||
z_p.transpose(1, 2), (m_p * s_p_sq_r)
|
||||
) # [b, t_t, d] x [b, d, t_s] = [b, t_t, t_s]
|
||||
neg_cent4 = torch.sum(
|
||||
-0.5 * (m_p**2) * s_p_sq_r, [1], keepdim=True
|
||||
) # [b, 1, t_s]
|
||||
neg_cent = neg_cent1 + neg_cent2 + neg_cent3 + neg_cent4
|
||||
if self.use_noise_scaled_mas:
|
||||
epsilon = (
|
||||
torch.std(neg_cent)
|
||||
* torch.randn_like(neg_cent)
|
||||
* self.current_mas_noise_scale
|
||||
)
|
||||
neg_cent = neg_cent + epsilon
|
||||
|
||||
attn_mask = torch.unsqueeze(x_mask, 2) * torch.unsqueeze(y_mask, -1)
|
||||
attn = (
|
||||
monotonic_align.maximum_path(neg_cent, attn_mask.squeeze(1))
|
||||
.unsqueeze(1)
|
||||
.detach()
|
||||
)
|
||||
|
||||
w = attn.sum(2)
|
||||
|
||||
l_length_sdp = self.sdp(x, x_mask, w, g=g)
|
||||
l_length_sdp = l_length_sdp / torch.sum(x_mask)
|
||||
|
||||
logw_ = torch.log(w + 1e-6) * x_mask
|
||||
logw = self.dp(x, x_mask, g=g)
|
||||
l_length_dp = torch.sum((logw - logw_) ** 2, [1, 2]) / torch.sum(
|
||||
x_mask
|
||||
) # for averaging
|
||||
|
||||
l_length = l_length_dp + l_length_sdp
|
||||
|
||||
# expand prior
|
||||
m_p = torch.matmul(attn.squeeze(1), m_p.transpose(1, 2)).transpose(1, 2)
|
||||
logs_p = torch.matmul(attn.squeeze(1), logs_p.transpose(1, 2)).transpose(1, 2)
|
||||
|
||||
z_slice, ids_slice = commons.rand_slice_segments(
|
||||
z, y_lengths, self.segment_size
|
||||
)
|
||||
o = self.dec(z_slice, g=g)
|
||||
return (
|
||||
o,
|
||||
l_length,
|
||||
attn,
|
||||
ids_slice,
|
||||
x_mask,
|
||||
y_mask,
|
||||
(z, z_p, m_p, logs_p, m_q, logs_q),
|
||||
(x, logw, logw_),
|
||||
)
|
||||
|
||||
def infer(
|
||||
self,
|
||||
x,
|
||||
x_lengths,
|
||||
sid,
|
||||
tone,
|
||||
language,
|
||||
bert,
|
||||
ja_bert,
|
||||
noise_scale=0.667,
|
||||
length_scale=1,
|
||||
noise_scale_w=0.8,
|
||||
max_len=None,
|
||||
sdp_ratio=0,
|
||||
y=None,
|
||||
):
|
||||
# x, m_p, logs_p, x_mask = self.enc_p(x, x_lengths, tone, language, bert)
|
||||
# g = self.gst(y)
|
||||
if self.n_speakers > 0:
|
||||
g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
|
||||
else:
|
||||
g = self.ref_enc(y.transpose(1, 2)).unsqueeze(-1)
|
||||
x, m_p, logs_p, x_mask = self.enc_p(
|
||||
x, x_lengths, tone, language, bert, ja_bert, g=g
|
||||
)
|
||||
logw = self.sdp(x, x_mask, g=g, reverse=True, noise_scale=noise_scale_w) * (
|
||||
sdp_ratio
|
||||
) + self.dp(x, x_mask, g=g) * (1 - sdp_ratio)
|
||||
w = torch.exp(logw) * x_mask * length_scale
|
||||
w_ceil = torch.ceil(w)
|
||||
y_lengths = torch.clamp_min(torch.sum(w_ceil, [1, 2]), 1).long()
|
||||
y_mask = torch.unsqueeze(commons.sequence_mask(y_lengths, None), 1).to(
|
||||
x_mask.dtype
|
||||
)
|
||||
attn_mask = torch.unsqueeze(x_mask, 2) * torch.unsqueeze(y_mask, -1)
|
||||
attn = commons.generate_path(w_ceil, attn_mask)
|
||||
|
||||
m_p = torch.matmul(attn.squeeze(1), m_p.transpose(1, 2)).transpose(
|
||||
1, 2
|
||||
) # [b, t', t], [b, t, d] -> [b, d, t']
|
||||
logs_p = torch.matmul(attn.squeeze(1), logs_p.transpose(1, 2)).transpose(
|
||||
1, 2
|
||||
) # [b, t', t], [b, t, d] -> [b, d, t']
|
||||
|
||||
z_p = m_p + torch.randn_like(m_p) * torch.exp(logs_p) * noise_scale
|
||||
z = self.flow(z_p, y_mask, g=g, reverse=True)
|
||||
o = self.dec((z * y_mask)[:, :, :max_len], g=g)
|
||||
return o, attn, y_mask, (z, z_p, m_p, logs_p)
|
||||
29
oldVersion/V110/text/__init__.py
Normal file
29
oldVersion/V110/text/__init__.py
Normal file
@@ -0,0 +1,29 @@
|
||||
from .symbols import *
|
||||
|
||||
|
||||
_symbol_to_id = {s: i for i, s in enumerate(symbols)}
|
||||
|
||||
|
||||
def cleaned_text_to_sequence(cleaned_text, tones, language):
|
||||
"""Converts a string of text to a sequence of IDs corresponding to the symbols in the text.
|
||||
Args:
|
||||
text: string to convert to a sequence
|
||||
Returns:
|
||||
List of integers corresponding to the symbols in the text
|
||||
"""
|
||||
phones = [_symbol_to_id[symbol] for symbol in cleaned_text]
|
||||
tone_start = language_tone_start_map[language]
|
||||
tones = [i + tone_start for i in tones]
|
||||
lang_id = language_id_map[language]
|
||||
lang_ids = [lang_id for i in phones]
|
||||
return phones, tones, lang_ids
|
||||
|
||||
|
||||
def get_bert(norm_text, word2ph, language, device):
|
||||
from .chinese_bert import get_bert_feature as zh_bert
|
||||
from .english_bert_mock import get_bert_feature as en_bert
|
||||
from .japanese_bert import get_bert_feature as jp_bert
|
||||
|
||||
lang_bert_func_map = {"ZH": zh_bert, "EN": en_bert, "JP": jp_bert}
|
||||
bert = lang_bert_func_map[language](norm_text, word2ph, device)
|
||||
return bert
|
||||
198
oldVersion/V110/text/chinese.py
Normal file
198
oldVersion/V110/text/chinese.py
Normal file
@@ -0,0 +1,198 @@
|
||||
import os
|
||||
import re
|
||||
|
||||
import cn2an
|
||||
from pypinyin import lazy_pinyin, Style
|
||||
|
||||
from .symbols import punctuation
|
||||
from .tone_sandhi import ToneSandhi
|
||||
|
||||
current_file_path = os.path.dirname(__file__)
|
||||
pinyin_to_symbol_map = {
|
||||
line.split("\t")[0]: line.strip().split("\t")[1]
|
||||
for line in open(os.path.join(current_file_path, "opencpop-strict.txt")).readlines()
|
||||
}
|
||||
|
||||
import jieba.posseg as psg
|
||||
|
||||
|
||||
rep_map = {
|
||||
"：": ",",
|
||||
"；": ",",
|
||||
"，": ",",
|
||||
"。": ".",
|
||||
"！": "!",
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"·": ",",
|
||||
"、": ",",
|
||||
"...": "…",
|
||||
"$": ".",
|
||||
"“": "'",
|
||||
"”": "'",
|
||||
"‘": "'",
|
||||
"’": "'",
|
||||
"（": "'",
|
||||
"）": "'",
|
||||
"(": "'",
|
||||
")": "'",
|
||||
"《": "'",
|
||||
"》": "'",
|
||||
"【": "'",
|
||||
"】": "'",
|
||||
"[": "'",
|
||||
"]": "'",
|
||||
"—": "-",
|
||||
"～": "-",
|
||||
"~": "-",
|
||||
"「": "'",
|
||||
"」": "'",
|
||||
}
|
||||
|
||||
tone_modifier = ToneSandhi()
|
||||
|
||||
|
||||
def replace_punctuation(text):
|
||||
text = text.replace("嗯", "恩").replace("呣", "母")
|
||||
pattern = re.compile("|".join(re.escape(p) for p in rep_map.keys()))
|
||||
|
||||
replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
|
||||
|
||||
replaced_text = re.sub(
|
||||
r"[^\u4e00-\u9fa5" + "".join(punctuation) + r"]+", "", replaced_text
|
||||
)
|
||||
|
||||
return replaced_text
|
||||
|
||||
|
||||
def g2p(text):
|
||||
pattern = r"(?<=[{0}])\s*".format("".join(punctuation))
|
||||
sentences = [i for i in re.split(pattern, text) if i.strip() != ""]
|
||||
phones, tones, word2ph = _g2p(sentences)
|
||||
assert sum(word2ph) == len(phones)
|
||||
assert len(word2ph) == len(text) # Sometimes it will crash,you can add a try-catch.
|
||||
phones = ["_"] + phones + ["_"]
|
||||
tones = [0] + tones + [0]
|
||||
word2ph = [1] + word2ph + [1]
|
||||
return phones, tones, word2ph
|
||||
|
||||
|
||||
def _get_initials_finals(word):
|
||||
initials = []
|
||||
finals = []
|
||||
orig_initials = lazy_pinyin(word, neutral_tone_with_five=True, style=Style.INITIALS)
|
||||
orig_finals = lazy_pinyin(
|
||||
word, neutral_tone_with_five=True, style=Style.FINALS_TONE3
|
||||
)
|
||||
for c, v in zip(orig_initials, orig_finals):
|
||||
initials.append(c)
|
||||
finals.append(v)
|
||||
return initials, finals
|
||||
|
||||
|
||||
def _g2p(segments):
|
||||
phones_list = []
|
||||
tones_list = []
|
||||
word2ph = []
|
||||
for seg in segments:
|
||||
# Replace all English words in the sentence
|
||||
seg = re.sub("[a-zA-Z]+", "", seg)
|
||||
seg_cut = psg.lcut(seg)
|
||||
initials = []
|
||||
finals = []
|
||||
seg_cut = tone_modifier.pre_merge_for_modify(seg_cut)
|
||||
for word, pos in seg_cut:
|
||||
if pos == "eng":
|
||||
continue
|
||||
sub_initials, sub_finals = _get_initials_finals(word)
|
||||
sub_finals = tone_modifier.modified_tone(word, pos, sub_finals)
|
||||
initials.append(sub_initials)
|
||||
finals.append(sub_finals)
|
||||
|
||||
# assert len(sub_initials) == len(sub_finals) == len(word)
|
||||
initials = sum(initials, [])
|
||||
finals = sum(finals, [])
|
||||
#
|
||||
for c, v in zip(initials, finals):
|
||||
raw_pinyin = c + v
|
||||
# NOTE: post process for pypinyin outputs
|
||||
# we discriminate i, ii and iii
|
||||
if c == v:
|
||||
assert c in punctuation
|
||||
phone = [c]
|
||||
tone = "0"
|
||||
word2ph.append(1)
|
||||
else:
|
||||
v_without_tone = v[:-1]
|
||||
tone = v[-1]
|
||||
|
||||
pinyin = c + v_without_tone
|
||||
assert tone in "12345"
|
||||
|
||||
if c:
|
||||
# 多音节
|
||||
v_rep_map = {
|
||||
"uei": "ui",
|
||||
"iou": "iu",
|
||||
"uen": "un",
|
||||
}
|
||||
if v_without_tone in v_rep_map.keys():
|
||||
pinyin = c + v_rep_map[v_without_tone]
|
||||
else:
|
||||
# 单音节
|
||||
pinyin_rep_map = {
|
||||
"ing": "ying",
|
||||
"i": "yi",
|
||||
"in": "yin",
|
||||
"u": "wu",
|
||||
}
|
||||
if pinyin in pinyin_rep_map.keys():
|
||||
pinyin = pinyin_rep_map[pinyin]
|
||||
else:
|
||||
single_rep_map = {
|
||||
"v": "yu",
|
||||
"e": "e",
|
||||
"i": "y",
|
||||
"u": "w",
|
||||
}
|
||||
if pinyin[0] in single_rep_map.keys():
|
||||
pinyin = single_rep_map[pinyin[0]] + pinyin[1:]
|
||||
|
||||
assert pinyin in pinyin_to_symbol_map.keys(), (pinyin, seg, raw_pinyin)
|
||||
phone = pinyin_to_symbol_map[pinyin].split(" ")
|
||||
word2ph.append(len(phone))
|
||||
|
||||
phones_list += phone
|
||||
tones_list += [int(tone)] * len(phone)
|
||||
return phones_list, tones_list, word2ph
|
||||
|
||||
|
||||
def text_normalize(text):
|
||||
numbers = re.findall(r"\d+(?:\.?\d+)?", text)
|
||||
for number in numbers:
|
||||
text = text.replace(number, cn2an.an2cn(number), 1)
|
||||
text = replace_punctuation(text)
|
||||
return text
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph):
|
||||
from text import chinese_bert
|
||||
|
||||
return chinese_bert.get_bert_feature(text, word2ph)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from text.chinese_bert import get_bert_feature
|
||||
|
||||
text = "啊！但是《原神》是由,米哈\游自主， [研发]的一款全.新开放世界.冒险游戏"
|
||||
text = text_normalize(text)
|
||||
print(text)
|
||||
phones, tones, word2ph = g2p(text)
|
||||
bert = get_bert_feature(text, word2ph)
|
||||
|
||||
print(phones, tones, word2ph, bert.shape)
|
||||
|
||||
|
||||
# # 示例用法
|
||||
# text = "这是一个示例文本：,你好！这是一个测试...."
|
||||
# print(g2p_paddle(text)) # 输出: 这是一个示例文本你好这是一个测试
|
||||
97
oldVersion/V110/text/chinese_bert.py
Normal file
97
oldVersion/V110/text/chinese_bert.py
Normal file
@@ -0,0 +1,97 @@
|
||||
import torch
|
||||
import sys
|
||||
from transformers import AutoTokenizer, AutoModelForMaskedLM
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/chinese-roberta-wwm-ext-large")
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph, device=None):
|
||||
if (
|
||||
sys.platform == "darwin"
|
||||
and torch.backends.mps.is_available()
|
||||
and device == "cpu"
|
||||
):
|
||||
device = "mps"
|
||||
if not device:
|
||||
device = "cuda"
|
||||
model = AutoModelForMaskedLM.from_pretrained(
|
||||
"./bert/chinese-roberta-wwm-ext-large"
|
||||
).to(device)
|
||||
with torch.no_grad():
|
||||
inputs = tokenizer(text, return_tensors="pt")
|
||||
for i in inputs:
|
||||
inputs[i] = inputs[i].to(device)
|
||||
res = model(**inputs, output_hidden_states=True)
|
||||
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
|
||||
|
||||
assert len(word2ph) == len(text) + 2
|
||||
word2phone = word2ph
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
repeat_feature = res[i].repeat(word2phone[i], 1)
|
||||
phone_level_feature.append(repeat_feature)
|
||||
|
||||
phone_level_feature = torch.cat(phone_level_feature, dim=0)
|
||||
|
||||
return phone_level_feature.T
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import torch
|
||||
|
||||
word_level_feature = torch.rand(38, 1024) # 12个词,每个词1024维特征
|
||||
word2phone = [
|
||||
1,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
]
|
||||
|
||||
# 计算总帧数
|
||||
total_frames = sum(word2phone)
|
||||
print(word_level_feature.shape)
|
||||
print(word2phone)
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
print(word_level_feature[i].shape)
|
||||
|
||||
# 对每个词重复word2phone[i]次
|
||||
repeat_feature = word_level_feature[i].repeat(word2phone[i], 1)
|
||||
phone_level_feature.append(repeat_feature)
|
||||
|
||||
phone_level_feature = torch.cat(phone_level_feature, dim=0)
|
||||
print(phone_level_feature.shape) # torch.Size([36, 1024])
|
||||
28
oldVersion/V110/text/cleaner.py
Normal file
28
oldVersion/V110/text/cleaner.py
Normal file
@@ -0,0 +1,28 @@
|
||||
from . import chinese, japanese, cleaned_text_to_sequence
|
||||
|
||||
|
||||
language_module_map = {"ZH": chinese, "JP": japanese}
|
||||
|
||||
|
||||
def clean_text(text, language):
|
||||
language_module = language_module_map[language]
|
||||
norm_text = language_module.text_normalize(text)
|
||||
phones, tones, word2ph = language_module.g2p(norm_text)
|
||||
return norm_text, phones, tones, word2ph
|
||||
|
||||
|
||||
def clean_text_bert(text, language):
|
||||
language_module = language_module_map[language]
|
||||
norm_text = language_module.text_normalize(text)
|
||||
phones, tones, word2ph = language_module.g2p(norm_text)
|
||||
bert = language_module.get_bert_feature(norm_text, word2ph)
|
||||
return phones, tones, bert
|
||||
|
||||
|
||||
def text_to_sequence(text, language):
|
||||
norm_text, phones, tones, word2ph = clean_text(text, language)
|
||||
return cleaned_text_to_sequence(phones, tones, language)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pass
|
||||
214
oldVersion/V110/text/english.py
Normal file
214
oldVersion/V110/text/english.py
Normal file
@@ -0,0 +1,214 @@
|
||||
import pickle
|
||||
import os
|
||||
import re
|
||||
from g2p_en import G2p
|
||||
|
||||
from . import symbols
|
||||
|
||||
current_file_path = os.path.dirname(__file__)
|
||||
CMU_DICT_PATH = os.path.join(current_file_path, "cmudict.rep")
|
||||
CACHE_PATH = os.path.join(current_file_path, "cmudict_cache.pickle")
|
||||
_g2p = G2p()
|
||||
|
||||
arpa = {
|
||||
"AH0",
|
||||
"S",
|
||||
"AH1",
|
||||
"EY2",
|
||||
"AE2",
|
||||
"EH0",
|
||||
"OW2",
|
||||
"UH0",
|
||||
"NG",
|
||||
"B",
|
||||
"G",
|
||||
"AY0",
|
||||
"M",
|
||||
"AA0",
|
||||
"F",
|
||||
"AO0",
|
||||
"ER2",
|
||||
"UH1",
|
||||
"IY1",
|
||||
"AH2",
|
||||
"DH",
|
||||
"IY0",
|
||||
"EY1",
|
||||
"IH0",
|
||||
"K",
|
||||
"N",
|
||||
"W",
|
||||
"IY2",
|
||||
"T",
|
||||
"AA1",
|
||||
"ER1",
|
||||
"EH2",
|
||||
"OY0",
|
||||
"UH2",
|
||||
"UW1",
|
||||
"Z",
|
||||
"AW2",
|
||||
"AW1",
|
||||
"V",
|
||||
"UW2",
|
||||
"AA2",
|
||||
"ER",
|
||||
"AW0",
|
||||
"UW0",
|
||||
"R",
|
||||
"OW1",
|
||||
"EH1",
|
||||
"ZH",
|
||||
"AE0",
|
||||
"IH2",
|
||||
"IH",
|
||||
"Y",
|
||||
"JH",
|
||||
"P",
|
||||
"AY1",
|
||||
"EY0",
|
||||
"OY2",
|
||||
"TH",
|
||||
"HH",
|
||||
"D",
|
||||
"ER0",
|
||||
"CH",
|
||||
"AO1",
|
||||
"AE1",
|
||||
"AO2",
|
||||
"OY1",
|
||||
"AY2",
|
||||
"IH1",
|
||||
"OW0",
|
||||
"L",
|
||||
"SH",
|
||||
}
|
||||
|
||||
|
||||
def post_replace_ph(ph):
|
||||
rep_map = {
|
||||
"：": ",",
|
||||
"；": ",",
|
||||
"，": ",",
|
||||
"。": ".",
|
||||
"！": "!",
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"·": ",",
|
||||
"、": ",",
|
||||
"...": "…",
|
||||
"v": "V",
|
||||
}
|
||||
if ph in rep_map.keys():
|
||||
ph = rep_map[ph]
|
||||
if ph in symbols:
|
||||
return ph
|
||||
if ph not in symbols:
|
||||
ph = "UNK"
|
||||
return ph
|
||||
|
||||
|
||||
def read_dict():
|
||||
g2p_dict = {}
|
||||
start_line = 49
|
||||
with open(CMU_DICT_PATH) as f:
|
||||
line = f.readline()
|
||||
line_index = 1
|
||||
while line:
|
||||
if line_index >= start_line:
|
||||
line = line.strip()
|
||||
word_split = line.split(" ")
|
||||
word = word_split[0]
|
||||
|
||||
syllable_split = word_split[1].split(" - ")
|
||||
g2p_dict[word] = []
|
||||
for syllable in syllable_split:
|
||||
phone_split = syllable.split(" ")
|
||||
g2p_dict[word].append(phone_split)
|
||||
|
||||
line_index = line_index + 1
|
||||
line = f.readline()
|
||||
|
||||
return g2p_dict
|
||||
|
||||
|
||||
def cache_dict(g2p_dict, file_path):
|
||||
with open(file_path, "wb") as pickle_file:
|
||||
pickle.dump(g2p_dict, pickle_file)
|
||||
|
||||
|
||||
def get_dict():
|
||||
if os.path.exists(CACHE_PATH):
|
||||
with open(CACHE_PATH, "rb") as pickle_file:
|
||||
g2p_dict = pickle.load(pickle_file)
|
||||
else:
|
||||
g2p_dict = read_dict()
|
||||
cache_dict(g2p_dict, CACHE_PATH)
|
||||
|
||||
return g2p_dict
|
||||
|
||||
|
||||
eng_dict = get_dict()
|
||||
|
||||
|
||||
def refine_ph(phn):
|
||||
tone = 0
|
||||
if re.search(r"\d$", phn):
|
||||
tone = int(phn[-1]) + 1
|
||||
phn = phn[:-1]
|
||||
return phn.lower(), tone
|
||||
|
||||
|
||||
def refine_syllables(syllables):
|
||||
tones = []
|
||||
phonemes = []
|
||||
for phn_list in syllables:
|
||||
for i in range(len(phn_list)):
|
||||
phn = phn_list[i]
|
||||
phn, tone = refine_ph(phn)
|
||||
phonemes.append(phn)
|
||||
tones.append(tone)
|
||||
return phonemes, tones
|
||||
|
||||
|
||||
def text_normalize(text):
|
||||
# todo: eng text normalize
|
||||
return text
|
||||
|
||||
|
||||
def g2p(text):
|
||||
phones = []
|
||||
tones = []
|
||||
words = re.split(r"([,;.\-\?\!\s+])", text)
|
||||
for w in words:
|
||||
if w.upper() in eng_dict:
|
||||
phns, tns = refine_syllables(eng_dict[w.upper()])
|
||||
phones += phns
|
||||
tones += tns
|
||||
else:
|
||||
phone_list = list(filter(lambda p: p != " ", _g2p(w)))
|
||||
for ph in phone_list:
|
||||
if ph in arpa:
|
||||
ph, tn = refine_ph(ph)
|
||||
phones.append(ph)
|
||||
tones.append(tn)
|
||||
else:
|
||||
phones.append(ph)
|
||||
tones.append(0)
|
||||
# todo: implement word2ph
|
||||
word2ph = [1 for i in phones]
|
||||
|
||||
phones = [post_replace_ph(i) for i in phones]
|
||||
return phones, tones, word2ph
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# print(get_dict())
|
||||
# print(eng_word_to_phoneme("hello"))
|
||||
print(g2p("In this paper, we propose 1 DSPGAN, a GAN-based universal vocoder."))
|
||||
# all_phones = set()
|
||||
# for k, syllables in eng_dict.items():
|
||||
# for group in syllables:
|
||||
# for ph in group:
|
||||
# all_phones.add(ph)
|
||||
# print(all_phones)
|
||||
5
oldVersion/V110/text/english_bert_mock.py
Normal file
5
oldVersion/V110/text/english_bert_mock.py
Normal file
@@ -0,0 +1,5 @@
|
||||
import torch
|
||||
|
||||
|
||||
def get_bert_feature(norm_text, word2ph):
|
||||
return torch.zeros(1024, sum(word2ph))
|
||||
586
oldVersion/V110/text/japanese.py
Normal file
586
oldVersion/V110/text/japanese.py
Normal file
@@ -0,0 +1,586 @@
|
||||
# Convert Japanese text to phonemes which is
|
||||
# compatible with Julius https://github.com/julius-speech/segmentation-kit
|
||||
import re
|
||||
import unicodedata
|
||||
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
from . import punctuation, symbols
|
||||
|
||||
try:
|
||||
import MeCab
|
||||
except ImportError as e:
|
||||
raise ImportError("Japanese requires mecab-python3 and unidic-lite.") from e
|
||||
from num2words import num2words
|
||||
|
||||
_CONVRULES = [
|
||||
# Conversion of 2 letters
|
||||
"アァ/ a a",
|
||||
"イィ/ i i",
|
||||
"イェ/ i e",
|
||||
"イャ/ y a",
|
||||
"ウゥ/ u:",
|
||||
"エェ/ e e",
|
||||
"オォ/ o:",
|
||||
"カァ/ k a:",
|
||||
"キィ/ k i:",
|
||||
"クゥ/ k u:",
|
||||
"クャ/ ky a",
|
||||
"クュ/ ky u",
|
||||
"クョ/ ky o",
|
||||
"ケェ/ k e:",
|
||||
"コォ/ k o:",
|
||||
"ガァ/ g a:",
|
||||
"ギィ/ g i:",
|
||||
"グゥ/ g u:",
|
||||
"グャ/ gy a",
|
||||
"グュ/ gy u",
|
||||
"グョ/ gy o",
|
||||
"ゲェ/ g e:",
|
||||
"ゴォ/ g o:",
|
||||
"サァ/ s a:",
|
||||
"シィ/ sh i:",
|
||||
"スゥ/ s u:",
|
||||
"スャ/ sh a",
|
||||
"スュ/ sh u",
|
||||
"スョ/ sh o",
|
||||
"セェ/ s e:",
|
||||
"ソォ/ s o:",
|
||||
"ザァ/ z a:",
|
||||
"ジィ/ j i:",
|
||||
"ズゥ/ z u:",
|
||||
"ズャ/ zy a",
|
||||
"ズュ/ zy u",
|
||||
"ズョ/ zy o",
|
||||
"ゼェ/ z e:",
|
||||
"ゾォ/ z o:",
|
||||
"タァ/ t a:",
|
||||
"チィ/ ch i:",
|
||||
"ツァ/ ts a",
|
||||
"ツィ/ ts i",
|
||||
"ツゥ/ ts u:",
|
||||
"ツャ/ ch a",
|
||||
"ツュ/ ch u",
|
||||
"ツョ/ ch o",
|
||||
"ツェ/ ts e",
|
||||
"ツォ/ ts o",
|
||||
"テェ/ t e:",
|
||||
"トォ/ t o:",
|
||||
"ダァ/ d a:",
|
||||
"ヂィ/ j i:",
|
||||
"ヅゥ/ d u:",
|
||||
"ヅャ/ zy a",
|
||||
"ヅュ/ zy u",
|
||||
"ヅョ/ zy o",
|
||||
"デェ/ d e:",
|
||||
"ドォ/ d o:",
|
||||
"ナァ/ n a:",
|
||||
"ニィ/ n i:",
|
||||
"ヌゥ/ n u:",
|
||||
"ヌャ/ ny a",
|
||||
"ヌュ/ ny u",
|
||||
"ヌョ/ ny o",
|
||||
"ネェ/ n e:",
|
||||
"ノォ/ n o:",
|
||||
"ハァ/ h a:",
|
||||
"ヒィ/ h i:",
|
||||
"フゥ/ f u:",
|
||||
"フャ/ hy a",
|
||||
"フュ/ hy u",
|
||||
"フョ/ hy o",
|
||||
"ヘェ/ h e:",
|
||||
"ホォ/ h o:",
|
||||
"バァ/ b a:",
|
||||
"ビィ/ b i:",
|
||||
"ブゥ/ b u:",
|
||||
"フャ/ hy a",
|
||||
"ブュ/ by u",
|
||||
"フョ/ hy o",
|
||||
"ベェ/ b e:",
|
||||
"ボォ/ b o:",
|
||||
"パァ/ p a:",
|
||||
"ピィ/ p i:",
|
||||
"プゥ/ p u:",
|
||||
"プャ/ py a",
|
||||
"プュ/ py u",
|
||||
"プョ/ py o",
|
||||
"ペェ/ p e:",
|
||||
"ポォ/ p o:",
|
||||
"マァ/ m a:",
|
||||
"ミィ/ m i:",
|
||||
"ムゥ/ m u:",
|
||||
"ムャ/ my a",
|
||||
"ムュ/ my u",
|
||||
"ムョ/ my o",
|
||||
"メェ/ m e:",
|
||||
"モォ/ m o:",
|
||||
"ヤァ/ y a:",
|
||||
"ユゥ/ y u:",
|
||||
"ユャ/ y a:",
|
||||
"ユュ/ y u:",
|
||||
"ユョ/ y o:",
|
||||
"ヨォ/ y o:",
|
||||
"ラァ/ r a:",
|
||||
"リィ/ r i:",
|
||||
"ルゥ/ r u:",
|
||||
"ルャ/ ry a",
|
||||
"ルュ/ ry u",
|
||||
"ルョ/ ry o",
|
||||
"レェ/ r e:",
|
||||
"ロォ/ r o:",
|
||||
"ワァ/ w a:",
|
||||
"ヲォ/ o:",
|
||||
"ディ/ d i",
|
||||
"デェ/ d e:",
|
||||
"デャ/ dy a",
|
||||
"デュ/ dy u",
|
||||
"デョ/ dy o",
|
||||
"ティ/ t i",
|
||||
"テェ/ t e:",
|
||||
"テャ/ ty a",
|
||||
"テュ/ ty u",
|
||||
"テョ/ ty o",
|
||||
"スィ/ s i",
|
||||
"ズァ/ z u a",
|
||||
"ズィ/ z i",
|
||||
"ズゥ/ z u",
|
||||
"ズャ/ zy a",
|
||||
"ズュ/ zy u",
|
||||
"ズョ/ zy o",
|
||||
"ズェ/ z e",
|
||||
"ズォ/ z o",
|
||||
"キャ/ ky a",
|
||||
"キュ/ ky u",
|
||||
"キョ/ ky o",
|
||||
"シャ/ sh a",
|
||||
"シュ/ sh u",
|
||||
"シェ/ sh e",
|
||||
"ショ/ sh o",
|
||||
"チャ/ ch a",
|
||||
"チュ/ ch u",
|
||||
"チェ/ ch e",
|
||||
"チョ/ ch o",
|
||||
"トゥ/ t u",
|
||||
"トャ/ ty a",
|
||||
"トュ/ ty u",
|
||||
"トョ/ ty o",
|
||||
"ドァ/ d o a",
|
||||
"ドゥ/ d u",
|
||||
"ドャ/ dy a",
|
||||
"ドュ/ dy u",
|
||||
"ドョ/ dy o",
|
||||
"ドォ/ d o:",
|
||||
"ニャ/ ny a",
|
||||
"ニュ/ ny u",
|
||||
"ニョ/ ny o",
|
||||
"ヒャ/ hy a",
|
||||
"ヒュ/ hy u",
|
||||
"ヒョ/ hy o",
|
||||
"ミャ/ my a",
|
||||
"ミュ/ my u",
|
||||
"ミョ/ my o",
|
||||
"リャ/ ry a",
|
||||
"リュ/ ry u",
|
||||
"リョ/ ry o",
|
||||
"ギャ/ gy a",
|
||||
"ギュ/ gy u",
|
||||
"ギョ/ gy o",
|
||||
"ヂェ/ j e",
|
||||
"ヂャ/ j a",
|
||||
"ヂュ/ j u",
|
||||
"ヂョ/ j o",
|
||||
"ジェ/ j e",
|
||||
"ジャ/ j a",
|
||||
"ジュ/ j u",
|
||||
"ジョ/ j o",
|
||||
"ビャ/ by a",
|
||||
"ビュ/ by u",
|
||||
"ビョ/ by o",
|
||||
"ピャ/ py a",
|
||||
"ピュ/ py u",
|
||||
"ピョ/ py o",
|
||||
"ウァ/ u a",
|
||||
"ウィ/ w i",
|
||||
"ウェ/ w e",
|
||||
"ウォ/ w o",
|
||||
"ファ/ f a",
|
||||
"フィ/ f i",
|
||||
"フゥ/ f u",
|
||||
"フャ/ hy a",
|
||||
"フュ/ hy u",
|
||||
"フョ/ hy o",
|
||||
"フェ/ f e",
|
||||
"フォ/ f o",
|
||||
"ヴァ/ b a",
|
||||
"ヴィ/ b i",
|
||||
"ヴェ/ b e",
|
||||
"ヴォ/ b o",
|
||||
"ヴュ/ by u",
|
||||
# Conversion of 1 letter
|
||||
"ア/ a",
|
||||
"イ/ i",
|
||||
"ウ/ u",
|
||||
"エ/ e",
|
||||
"オ/ o",
|
||||
"カ/ k a",
|
||||
"キ/ k i",
|
||||
"ク/ k u",
|
||||
"ケ/ k e",
|
||||
"コ/ k o",
|
||||
"サ/ s a",
|
||||
"シ/ sh i",
|
||||
"ス/ s u",
|
||||
"セ/ s e",
|
||||
"ソ/ s o",
|
||||
"タ/ t a",
|
||||
"チ/ ch i",
|
||||
"ツ/ ts u",
|
||||
"テ/ t e",
|
||||
"ト/ t o",
|
||||
"ナ/ n a",
|
||||
"ニ/ n i",
|
||||
"ヌ/ n u",
|
||||
"ネ/ n e",
|
||||
"ノ/ n o",
|
||||
"ハ/ h a",
|
||||
"ヒ/ h i",
|
||||
"フ/ f u",
|
||||
"ヘ/ h e",
|
||||
"ホ/ h o",
|
||||
"マ/ m a",
|
||||
"ミ/ m i",
|
||||
"ム/ m u",
|
||||
"メ/ m e",
|
||||
"モ/ m o",
|
||||
"ラ/ r a",
|
||||
"リ/ r i",
|
||||
"ル/ r u",
|
||||
"レ/ r e",
|
||||
"ロ/ r o",
|
||||
"ガ/ g a",
|
||||
"ギ/ g i",
|
||||
"グ/ g u",
|
||||
"ゲ/ g e",
|
||||
"ゴ/ g o",
|
||||
"ザ/ z a",
|
||||
"ジ/ j i",
|
||||
"ズ/ z u",
|
||||
"ゼ/ z e",
|
||||
"ゾ/ z o",
|
||||
"ダ/ d a",
|
||||
"ヂ/ j i",
|
||||
"ヅ/ z u",
|
||||
"デ/ d e",
|
||||
"ド/ d o",
|
||||
"バ/ b a",
|
||||
"ビ/ b i",
|
||||
"ブ/ b u",
|
||||
"ベ/ b e",
|
||||
"ボ/ b o",
|
||||
"パ/ p a",
|
||||
"ピ/ p i",
|
||||
"プ/ p u",
|
||||
"ペ/ p e",
|
||||
"ポ/ p o",
|
||||
"ヤ/ y a",
|
||||
"ユ/ y u",
|
||||
"ヨ/ y o",
|
||||
"ワ/ w a",
|
||||
"ヰ/ i",
|
||||
"ヱ/ e",
|
||||
"ヲ/ o",
|
||||
"ン/ N",
|
||||
"ッ/ q",
|
||||
"ヴ/ b u",
|
||||
"ー/:",
|
||||
# Try converting broken text
|
||||
"ァ/ a",
|
||||
"ィ/ i",
|
||||
"ゥ/ u",
|
||||
"ェ/ e",
|
||||
"ォ/ o",
|
||||
"ヮ/ w a",
|
||||
"ォ/ o",
|
||||
# Symbols
|
||||
"、/ ,",
|
||||
"。/ .",
|
||||
"！/ !",
|
||||
"？/ ?",
|
||||
"・/ ,",
|
||||
]
|
||||
|
||||
_COLON_RX = re.compile(":+")
|
||||
_REJECT_RX = re.compile("[^ a-zA-Z:,.?]")
|
||||
|
||||
|
||||
def _makerulemap():
|
||||
l = [tuple(x.split("/")) for x in _CONVRULES]
|
||||
return tuple({k: v for k, v in l if len(k) == i} for i in (1, 2))
|
||||
|
||||
|
||||
_RULEMAP1, _RULEMAP2 = _makerulemap()
|
||||
|
||||
|
||||
def kata2phoneme(text: str) -> str:
|
||||
"""Convert katakana text to phonemes."""
|
||||
text = text.strip()
|
||||
res = []
|
||||
while text:
|
||||
if len(text) >= 2:
|
||||
x = _RULEMAP2.get(text[:2])
|
||||
if x is not None:
|
||||
text = text[2:]
|
||||
res += x.split(" ")[1:]
|
||||
continue
|
||||
x = _RULEMAP1.get(text[0])
|
||||
if x is not None:
|
||||
text = text[1:]
|
||||
res += x.split(" ")[1:]
|
||||
continue
|
||||
res.append(text[0])
|
||||
text = text[1:]
|
||||
# res = _COLON_RX.sub(":", res)
|
||||
return res
|
||||
|
||||
|
||||
_KATAKANA = "".join(chr(ch) for ch in range(ord("ァ"), ord("ン") + 1))
|
||||
_HIRAGANA = "".join(chr(ch) for ch in range(ord("ぁ"), ord("ん") + 1))
|
||||
_HIRA2KATATRANS = str.maketrans(_HIRAGANA, _KATAKANA)
|
||||
|
||||
|
||||
def hira2kata(text: str) -> str:
|
||||
text = text.translate(_HIRA2KATATRANS)
|
||||
return text.replace("う゛", "ヴ")
|
||||
|
||||
|
||||
_SYMBOL_TOKENS = set(list("・、。？！"))
|
||||
_NO_YOMI_TOKENS = set(list("「」『』―（）［］[]"))
|
||||
_TAGGER = MeCab.Tagger()
|
||||
|
||||
|
||||
def text2kata(text: str) -> str:
|
||||
parsed = _TAGGER.parse(text)
|
||||
res = []
|
||||
for line in parsed.split("\n"):
|
||||
if line == "EOS":
|
||||
break
|
||||
parts = line.split("\t")
|
||||
|
||||
word, yomi = parts[0], parts[1]
|
||||
if yomi:
|
||||
res.append(yomi)
|
||||
else:
|
||||
if word in _SYMBOL_TOKENS:
|
||||
res.append(word)
|
||||
elif word in ("っ", "ッ"):
|
||||
res.append("ッ")
|
||||
elif word in _NO_YOMI_TOKENS:
|
||||
pass
|
||||
else:
|
||||
res.append(word)
|
||||
return hira2kata("".join(res))
|
||||
|
||||
|
||||
_ALPHASYMBOL_YOMI = {
|
||||
"#": "シャープ",
|
||||
"%": "パーセント",
|
||||
"&": "アンド",
|
||||
"+": "プラス",
|
||||
"-": "マイナス",
|
||||
":": "コロン",
|
||||
";": "セミコロン",
|
||||
"<": "小なり",
|
||||
"=": "イコール",
|
||||
">": "大なり",
|
||||
"@": "アット",
|
||||
"a": "エー",
|
||||
"b": "ビー",
|
||||
"c": "シー",
|
||||
"d": "ディー",
|
||||
"e": "イー",
|
||||
"f": "エフ",
|
||||
"g": "ジー",
|
||||
"h": "エイチ",
|
||||
"i": "アイ",
|
||||
"j": "ジェー",
|
||||
"k": "ケー",
|
||||
"l": "エル",
|
||||
"m": "エム",
|
||||
"n": "エヌ",
|
||||
"o": "オー",
|
||||
"p": "ピー",
|
||||
"q": "キュー",
|
||||
"r": "アール",
|
||||
"s": "エス",
|
||||
"t": "ティー",
|
||||
"u": "ユー",
|
||||
"v": "ブイ",
|
||||
"w": "ダブリュー",
|
||||
"x": "エックス",
|
||||
"y": "ワイ",
|
||||
"z": "ゼット",
|
||||
"α": "アルファ",
|
||||
"β": "ベータ",
|
||||
"γ": "ガンマ",
|
||||
"δ": "デルタ",
|
||||
"ε": "イプシロン",
|
||||
"ζ": "ゼータ",
|
||||
"η": "イータ",
|
||||
"θ": "シータ",
|
||||
"ι": "イオタ",
|
||||
"κ": "カッパ",
|
||||
"λ": "ラムダ",
|
||||
"μ": "ミュー",
|
||||
"ν": "ニュー",
|
||||
"ξ": "クサイ",
|
||||
"ο": "オミクロン",
|
||||
"π": "パイ",
|
||||
"ρ": "ロー",
|
||||
"σ": "シグマ",
|
||||
"τ": "タウ",
|
||||
"υ": "ウプシロン",
|
||||
"φ": "ファイ",
|
||||
"χ": "カイ",
|
||||
"ψ": "プサイ",
|
||||
"ω": "オメガ",
|
||||
}
|
||||
|
||||
|
||||
_NUMBER_WITH_SEPARATOR_RX = re.compile("[0-9]{1,3}(,[0-9]{3})+")
|
||||
_CURRENCY_MAP = {"$": "ドル", "¥": "円", "£": "ポンド", "€": "ユーロ"}
|
||||
_CURRENCY_RX = re.compile(r"([$¥£€])([0-9.]*[0-9])")
|
||||
_NUMBER_RX = re.compile(r"[0-9]+(\.[0-9]+)?")
|
||||
|
||||
|
||||
def japanese_convert_numbers_to_words(text: str) -> str:
|
||||
res = _NUMBER_WITH_SEPARATOR_RX.sub(lambda m: m[0].replace(",", ""), text)
|
||||
res = _CURRENCY_RX.sub(lambda m: m[2] + _CURRENCY_MAP.get(m[1], m[1]), res)
|
||||
res = _NUMBER_RX.sub(lambda m: num2words(m[0], lang="ja"), res)
|
||||
return res
|
||||
|
||||
|
||||
def japanese_convert_alpha_symbols_to_words(text: str) -> str:
|
||||
return "".join([_ALPHASYMBOL_YOMI.get(ch, ch) for ch in text.lower()])
|
||||
|
||||
|
||||
def japanese_text_to_phonemes(text: str) -> str:
|
||||
"""Convert Japanese text to phonemes."""
|
||||
res = unicodedata.normalize("NFKC", text)
|
||||
res = japanese_convert_numbers_to_words(res)
|
||||
# res = japanese_convert_alpha_symbols_to_words(res)
|
||||
res = text2kata(res)
|
||||
res = kata2phoneme(res)
|
||||
return res
|
||||
|
||||
|
||||
def is_japanese_character(char):
|
||||
# 定义日语文字系统的 Unicode 范围
|
||||
japanese_ranges = [
|
||||
(0x3040, 0x309F), # 平假名
|
||||
(0x30A0, 0x30FF), # 片假名
|
||||
(0x4E00, 0x9FFF), # 汉字 (CJK Unified Ideographs)
|
||||
(0x3400, 0x4DBF), # 汉字扩展 A
|
||||
(0x20000, 0x2A6DF), # 汉字扩展 B
|
||||
# 可以根据需要添加其他汉字扩展范围
|
||||
]
|
||||
|
||||
# 将字符的 Unicode 编码转换为整数
|
||||
char_code = ord(char)
|
||||
|
||||
# 检查字符是否在任何一个日语范围内
|
||||
for start, end in japanese_ranges:
|
||||
if start <= char_code <= end:
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
rep_map = {
|
||||
"：": ",",
|
||||
"；": ",",
|
||||
"，": ",",
|
||||
"。": ".",
|
||||
"！": "!",
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"·": ",",
|
||||
"、": ",",
|
||||
"...": "…",
|
||||
}
|
||||
|
||||
|
||||
def replace_punctuation(text):
|
||||
pattern = re.compile("|".join(re.escape(p) for p in rep_map.keys()))
|
||||
|
||||
replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
|
||||
|
||||
replaced_text = re.sub(
|
||||
r"[^\u3040-\u309F\u30A0-\u30FF\u4E00-\u9FFF\u3400-\u4DBF"
|
||||
+ "".join(punctuation)
|
||||
+ r"]+",
|
||||
"",
|
||||
replaced_text,
|
||||
)
|
||||
|
||||
return replaced_text
|
||||
|
||||
|
||||
def text_normalize(text):
|
||||
res = unicodedata.normalize("NFKC", text)
|
||||
res = japanese_convert_numbers_to_words(res)
|
||||
# res = "".join([i for i in res if is_japanese_character(i)])
|
||||
res = replace_punctuation(res)
|
||||
return res
|
||||
|
||||
|
||||
def distribute_phone(n_phone, n_word):
|
||||
phones_per_word = [0] * n_word
|
||||
for task in range(n_phone):
|
||||
min_tasks = min(phones_per_word)
|
||||
min_index = phones_per_word.index(min_tasks)
|
||||
phones_per_word[min_index] += 1
|
||||
return phones_per_word
|
||||
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/bert-base-japanese-v3")
|
||||
|
||||
|
||||
def g2p(norm_text):
|
||||
tokenized = tokenizer.tokenize(norm_text)
|
||||
phs = []
|
||||
ph_groups = []
|
||||
for t in tokenized:
|
||||
if not t.startswith("#"):
|
||||
ph_groups.append([t])
|
||||
else:
|
||||
ph_groups[-1].append(t.replace("#", ""))
|
||||
word2ph = []
|
||||
for group in ph_groups:
|
||||
phonemes = kata2phoneme(text2kata("".join(group)))
|
||||
# phonemes = [i for i in phonemes if i in symbols]
|
||||
for i in phonemes:
|
||||
assert i in symbols, (group, norm_text, tokenized)
|
||||
phone_len = len(phonemes)
|
||||
word_len = len(group)
|
||||
|
||||
aaa = distribute_phone(phone_len, word_len)
|
||||
word2ph += aaa
|
||||
|
||||
phs += phonemes
|
||||
phones = ["_"] + phs + ["_"]
|
||||
tones = [0 for i in phones]
|
||||
word2ph = [1] + word2ph + [1]
|
||||
return phones, tones, word2ph
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/bert-base-japanese-v3")
|
||||
text = "hello,こんにちは、世界！……"
|
||||
from text.japanese_bert import get_bert_feature
|
||||
|
||||
text = text_normalize(text)
|
||||
print(text)
|
||||
phones, tones, word2ph = g2p(text)
|
||||
bert = get_bert_feature(text, word2ph)
|
||||
|
||||
print(phones, tones, word2ph, bert.shape)
|
||||
35
oldVersion/V110/text/japanese_bert.py
Normal file
35
oldVersion/V110/text/japanese_bert.py
Normal file
@@ -0,0 +1,35 @@
|
||||
import torch
|
||||
from transformers import AutoTokenizer, AutoModelForMaskedLM
|
||||
import sys
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/bert-base-japanese-v3")
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph, device=None):
|
||||
if (
|
||||
sys.platform == "darwin"
|
||||
and torch.backends.mps.is_available()
|
||||
and device == "cpu"
|
||||
):
|
||||
device = "mps"
|
||||
if not device:
|
||||
device = "cuda"
|
||||
model = AutoModelForMaskedLM.from_pretrained("./bert/bert-base-japanese-v3").to(
|
||||
device
|
||||
)
|
||||
with torch.no_grad():
|
||||
inputs = tokenizer(text, return_tensors="pt")
|
||||
for i in inputs:
|
||||
inputs[i] = inputs[i].to(device)
|
||||
res = model(**inputs, output_hidden_states=True)
|
||||
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
|
||||
assert inputs["input_ids"].shape[-1] == len(word2ph)
|
||||
word2phone = word2ph
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
repeat_feature = res[i].repeat(word2phone[i], 1)
|
||||
phone_level_feature.append(repeat_feature)
|
||||
|
||||
phone_level_feature = torch.cat(phone_level_feature, dim=0)
|
||||
|
||||
return phone_level_feature.T
|
||||
429
oldVersion/V110/text/opencpop-strict.txt
Normal file
429
oldVersion/V110/text/opencpop-strict.txt
Normal file
@@ -0,0 +1,429 @@
|
||||
a AA a
|
||||
ai AA ai
|
||||
an AA an
|
||||
ang AA ang
|
||||
ao AA ao
|
||||
ba b a
|
||||
bai b ai
|
||||
ban b an
|
||||
bang b ang
|
||||
bao b ao
|
||||
bei b ei
|
||||
ben b en
|
||||
beng b eng
|
||||
bi b i
|
||||
bian b ian
|
||||
biao b iao
|
||||
bie b ie
|
||||
bin b in
|
||||
bing b ing
|
||||
bo b o
|
||||
bu b u
|
||||
ca c a
|
||||
cai c ai
|
||||
can c an
|
||||
cang c ang
|
||||
cao c ao
|
||||
ce c e
|
||||
cei c ei
|
||||
cen c en
|
||||
ceng c eng
|
||||
cha ch a
|
||||
chai ch ai
|
||||
chan ch an
|
||||
chang ch ang
|
||||
chao ch ao
|
||||
che ch e
|
||||
chen ch en
|
||||
cheng ch eng
|
||||
chi ch ir
|
||||
chong ch ong
|
||||
chou ch ou
|
||||
chu ch u
|
||||
chua ch ua
|
||||
chuai ch uai
|
||||
chuan ch uan
|
||||
chuang ch uang
|
||||
chui ch ui
|
||||
chun ch un
|
||||
chuo ch uo
|
||||
ci c i0
|
||||
cong c ong
|
||||
cou c ou
|
||||
cu c u
|
||||
cuan c uan
|
||||
cui c ui
|
||||
cun c un
|
||||
cuo c uo
|
||||
da d a
|
||||
dai d ai
|
||||
dan d an
|
||||
dang d ang
|
||||
dao d ao
|
||||
de d e
|
||||
dei d ei
|
||||
den d en
|
||||
deng d eng
|
||||
di d i
|
||||
dia d ia
|
||||
dian d ian
|
||||
diao d iao
|
||||
die d ie
|
||||
ding d ing
|
||||
diu d iu
|
||||
dong d ong
|
||||
dou d ou
|
||||
du d u
|
||||
duan d uan
|
||||
dui d ui
|
||||
dun d un
|
||||
duo d uo
|
||||
e EE e
|
||||
ei EE ei
|
||||
en EE en
|
||||
eng EE eng
|
||||
er EE er
|
||||
fa f a
|
||||
fan f an
|
||||
fang f ang
|
||||
fei f ei
|
||||
fen f en
|
||||
feng f eng
|
||||
fo f o
|
||||
fou f ou
|
||||
fu f u
|
||||
ga g a
|
||||
gai g ai
|
||||
gan g an
|
||||
gang g ang
|
||||
gao g ao
|
||||
ge g e
|
||||
gei g ei
|
||||
gen g en
|
||||
geng g eng
|
||||
gong g ong
|
||||
gou g ou
|
||||
gu g u
|
||||
gua g ua
|
||||
guai g uai
|
||||
guan g uan
|
||||
guang g uang
|
||||
gui g ui
|
||||
gun g un
|
||||
guo g uo
|
||||
ha h a
|
||||
hai h ai
|
||||
han h an
|
||||
hang h ang
|
||||
hao h ao
|
||||
he h e
|
||||
hei h ei
|
||||
hen h en
|
||||
heng h eng
|
||||
hong h ong
|
||||
hou h ou
|
||||
hu h u
|
||||
hua h ua
|
||||
huai h uai
|
||||
huan h uan
|
||||
huang h uang
|
||||
hui h ui
|
||||
hun h un
|
||||
huo h uo
|
||||
ji j i
|
||||
jia j ia
|
||||
jian j ian
|
||||
jiang j iang
|
||||
jiao j iao
|
||||
jie j ie
|
||||
jin j in
|
||||
jing j ing
|
||||
jiong j iong
|
||||
jiu j iu
|
||||
ju j v
|
||||
jv j v
|
||||
juan j van
|
||||
jvan j van
|
||||
jue j ve
|
||||
jve j ve
|
||||
jun j vn
|
||||
jvn j vn
|
||||
ka k a
|
||||
kai k ai
|
||||
kan k an
|
||||
kang k ang
|
||||
kao k ao
|
||||
ke k e
|
||||
kei k ei
|
||||
ken k en
|
||||
keng k eng
|
||||
kong k ong
|
||||
kou k ou
|
||||
ku k u
|
||||
kua k ua
|
||||
kuai k uai
|
||||
kuan k uan
|
||||
kuang k uang
|
||||
kui k ui
|
||||
kun k un
|
||||
kuo k uo
|
||||
la l a
|
||||
lai l ai
|
||||
lan l an
|
||||
lang l ang
|
||||
lao l ao
|
||||
le l e
|
||||
lei l ei
|
||||
leng l eng
|
||||
li l i
|
||||
lia l ia
|
||||
lian l ian
|
||||
liang l iang
|
||||
liao l iao
|
||||
lie l ie
|
||||
lin l in
|
||||
ling l ing
|
||||
liu l iu
|
||||
lo l o
|
||||
long l ong
|
||||
lou l ou
|
||||
lu l u
|
||||
luan l uan
|
||||
lun l un
|
||||
luo l uo
|
||||
lv l v
|
||||
lve l ve
|
||||
ma m a
|
||||
mai m ai
|
||||
man m an
|
||||
mang m ang
|
||||
mao m ao
|
||||
me m e
|
||||
mei m ei
|
||||
men m en
|
||||
meng m eng
|
||||
mi m i
|
||||
mian m ian
|
||||
miao m iao
|
||||
mie m ie
|
||||
min m in
|
||||
ming m ing
|
||||
miu m iu
|
||||
mo m o
|
||||
mou m ou
|
||||
mu m u
|
||||
na n a
|
||||
nai n ai
|
||||
nan n an
|
||||
nang n ang
|
||||
nao n ao
|
||||
ne n e
|
||||
nei n ei
|
||||
nen n en
|
||||
neng n eng
|
||||
ni n i
|
||||
nian n ian
|
||||
niang n iang
|
||||
niao n iao
|
||||
nie n ie
|
||||
nin n in
|
||||
ning n ing
|
||||
niu n iu
|
||||
nong n ong
|
||||
nou n ou
|
||||
nu n u
|
||||
nuan n uan
|
||||
nun n un
|
||||
nuo n uo
|
||||
nv n v
|
||||
nve n ve
|
||||
o OO o
|
||||
ou OO ou
|
||||
pa p a
|
||||
pai p ai
|
||||
pan p an
|
||||
pang p ang
|
||||
pao p ao
|
||||
pei p ei
|
||||
pen p en
|
||||
peng p eng
|
||||
pi p i
|
||||
pian p ian
|
||||
piao p iao
|
||||
pie p ie
|
||||
pin p in
|
||||
ping p ing
|
||||
po p o
|
||||
pou p ou
|
||||
pu p u
|
||||
qi q i
|
||||
qia q ia
|
||||
qian q ian
|
||||
qiang q iang
|
||||
qiao q iao
|
||||
qie q ie
|
||||
qin q in
|
||||
qing q ing
|
||||
qiong q iong
|
||||
qiu q iu
|
||||
qu q v
|
||||
qv q v
|
||||
quan q van
|
||||
qvan q van
|
||||
que q ve
|
||||
qve q ve
|
||||
qun q vn
|
||||
qvn q vn
|
||||
ran r an
|
||||
rang r ang
|
||||
rao r ao
|
||||
re r e
|
||||
ren r en
|
||||
reng r eng
|
||||
ri r ir
|
||||
rong r ong
|
||||
rou r ou
|
||||
ru r u
|
||||
rua r ua
|
||||
ruan r uan
|
||||
rui r ui
|
||||
run r un
|
||||
ruo r uo
|
||||
sa s a
|
||||
sai s ai
|
||||
san s an
|
||||
sang s ang
|
||||
sao s ao
|
||||
se s e
|
||||
sen s en
|
||||
seng s eng
|
||||
sha sh a
|
||||
shai sh ai
|
||||
shan sh an
|
||||
shang sh ang
|
||||
shao sh ao
|
||||
she sh e
|
||||
shei sh ei
|
||||
shen sh en
|
||||
sheng sh eng
|
||||
shi sh ir
|
||||
shou sh ou
|
||||
shu sh u
|
||||
shua sh ua
|
||||
shuai sh uai
|
||||
shuan sh uan
|
||||
shuang sh uang
|
||||
shui sh ui
|
||||
shun sh un
|
||||
shuo sh uo
|
||||
si s i0
|
||||
song s ong
|
||||
sou s ou
|
||||
su s u
|
||||
suan s uan
|
||||
sui s ui
|
||||
sun s un
|
||||
suo s uo
|
||||
ta t a
|
||||
tai t ai
|
||||
tan t an
|
||||
tang t ang
|
||||
tao t ao
|
||||
te t e
|
||||
tei t ei
|
||||
teng t eng
|
||||
ti t i
|
||||
tian t ian
|
||||
tiao t iao
|
||||
tie t ie
|
||||
ting t ing
|
||||
tong t ong
|
||||
tou t ou
|
||||
tu t u
|
||||
tuan t uan
|
||||
tui t ui
|
||||
tun t un
|
||||
tuo t uo
|
||||
wa w a
|
||||
wai w ai
|
||||
wan w an
|
||||
wang w ang
|
||||
wei w ei
|
||||
wen w en
|
||||
weng w eng
|
||||
wo w o
|
||||
wu w u
|
||||
xi x i
|
||||
xia x ia
|
||||
xian x ian
|
||||
xiang x iang
|
||||
xiao x iao
|
||||
xie x ie
|
||||
xin x in
|
||||
xing x ing
|
||||
xiong x iong
|
||||
xiu x iu
|
||||
xu x v
|
||||
xv x v
|
||||
xuan x van
|
||||
xvan x van
|
||||
xue x ve
|
||||
xve x ve
|
||||
xun x vn
|
||||
xvn x vn
|
||||
ya y a
|
||||
yan y En
|
||||
yang y ang
|
||||
yao y ao
|
||||
ye y E
|
||||
yi y i
|
||||
yin y in
|
||||
ying y ing
|
||||
yo y o
|
||||
yong y ong
|
||||
you y ou
|
||||
yu y v
|
||||
yv y v
|
||||
yuan y van
|
||||
yvan y van
|
||||
yue y ve
|
||||
yve y ve
|
||||
yun y vn
|
||||
yvn y vn
|
||||
za z a
|
||||
zai z ai
|
||||
zan z an
|
||||
zang z ang
|
||||
zao z ao
|
||||
ze z e
|
||||
zei z ei
|
||||
zen z en
|
||||
zeng z eng
|
||||
zha zh a
|
||||
zhai zh ai
|
||||
zhan zh an
|
||||
zhang zh ang
|
||||
zhao zh ao
|
||||
zhe zh e
|
||||
zhei zh ei
|
||||
zhen zh en
|
||||
zheng zh eng
|
||||
zhi zh ir
|
||||
zhong zh ong
|
||||
zhou zh ou
|
||||
zhu zh u
|
||||
zhua zh ua
|
||||
zhuai zh uai
|
||||
zhuan zh uan
|
||||
zhuang zh uang
|
||||
zhui zh ui
|
||||
zhun zh un
|
||||
zhuo zh uo
|
||||
zi z i0
|
||||
zong z ong
|
||||
zou z ou
|
||||
zu z u
|
||||
zuan z uan
|
||||
zui z ui
|
||||
zun z un
|
||||
zuo z uo
|
||||
187
oldVersion/V110/text/symbols.py
Normal file
187
oldVersion/V110/text/symbols.py
Normal file
@@ -0,0 +1,187 @@
|
||||
punctuation = ["!", "?", "…", ",", ".", "'", "-"]
|
||||
pu_symbols = punctuation + ["SP", "UNK"]
|
||||
pad = "_"
|
||||
|
||||
# chinese
|
||||
zh_symbols = [
|
||||
"E",
|
||||
"En",
|
||||
"a",
|
||||
"ai",
|
||||
"an",
|
||||
"ang",
|
||||
"ao",
|
||||
"b",
|
||||
"c",
|
||||
"ch",
|
||||
"d",
|
||||
"e",
|
||||
"ei",
|
||||
"en",
|
||||
"eng",
|
||||
"er",
|
||||
"f",
|
||||
"g",
|
||||
"h",
|
||||
"i",
|
||||
"i0",
|
||||
"ia",
|
||||
"ian",
|
||||
"iang",
|
||||
"iao",
|
||||
"ie",
|
||||
"in",
|
||||
"ing",
|
||||
"iong",
|
||||
"ir",
|
||||
"iu",
|
||||
"j",
|
||||
"k",
|
||||
"l",
|
||||
"m",
|
||||
"n",
|
||||
"o",
|
||||
"ong",
|
||||
"ou",
|
||||
"p",
|
||||
"q",
|
||||
"r",
|
||||
"s",
|
||||
"sh",
|
||||
"t",
|
||||
"u",
|
||||
"ua",
|
||||
"uai",
|
||||
"uan",
|
||||
"uang",
|
||||
"ui",
|
||||
"un",
|
||||
"uo",
|
||||
"v",
|
||||
"van",
|
||||
"ve",
|
||||
"vn",
|
||||
"w",
|
||||
"x",
|
||||
"y",
|
||||
"z",
|
||||
"zh",
|
||||
"AA",
|
||||
"EE",
|
||||
"OO",
|
||||
]
|
||||
num_zh_tones = 6
|
||||
|
||||
# japanese
|
||||
ja_symbols = [
|
||||
"N",
|
||||
"a",
|
||||
"a:",
|
||||
"b",
|
||||
"by",
|
||||
"ch",
|
||||
"d",
|
||||
"dy",
|
||||
"e",
|
||||
"e:",
|
||||
"f",
|
||||
"g",
|
||||
"gy",
|
||||
"h",
|
||||
"hy",
|
||||
"i",
|
||||
"i:",
|
||||
"j",
|
||||
"k",
|
||||
"ky",
|
||||
"m",
|
||||
"my",
|
||||
"n",
|
||||
"ny",
|
||||
"o",
|
||||
"o:",
|
||||
"p",
|
||||
"py",
|
||||
"q",
|
||||
"r",
|
||||
"ry",
|
||||
"s",
|
||||
"sh",
|
||||
"t",
|
||||
"ts",
|
||||
"ty",
|
||||
"u",
|
||||
"u:",
|
||||
"w",
|
||||
"y",
|
||||
"z",
|
||||
"zy",
|
||||
]
|
||||
num_ja_tones = 1
|
||||
|
||||
# English
|
||||
en_symbols = [
|
||||
"aa",
|
||||
"ae",
|
||||
"ah",
|
||||
"ao",
|
||||
"aw",
|
||||
"ay",
|
||||
"b",
|
||||
"ch",
|
||||
"d",
|
||||
"dh",
|
||||
"eh",
|
||||
"er",
|
||||
"ey",
|
||||
"f",
|
||||
"g",
|
||||
"hh",
|
||||
"ih",
|
||||
"iy",
|
||||
"jh",
|
||||
"k",
|
||||
"l",
|
||||
"m",
|
||||
"n",
|
||||
"ng",
|
||||
"ow",
|
||||
"oy",
|
||||
"p",
|
||||
"r",
|
||||
"s",
|
||||
"sh",
|
||||
"t",
|
||||
"th",
|
||||
"uh",
|
||||
"uw",
|
||||
"V",
|
||||
"w",
|
||||
"y",
|
||||
"z",
|
||||
"zh",
|
||||
]
|
||||
num_en_tones = 4
|
||||
|
||||
# combine all symbols
|
||||
normal_symbols = sorted(set(zh_symbols + ja_symbols + en_symbols))
|
||||
symbols = [pad] + normal_symbols + pu_symbols
|
||||
sil_phonemes_ids = [symbols.index(i) for i in pu_symbols]
|
||||
|
||||
# combine all tones
|
||||
num_tones = num_zh_tones + num_ja_tones + num_en_tones
|
||||
|
||||
# language maps
|
||||
language_id_map = {"ZH": 0, "JP": 1, "EN": 2}
|
||||
num_languages = len(language_id_map.keys())
|
||||
|
||||
language_tone_start_map = {
|
||||
"ZH": 0,
|
||||
"JP": num_zh_tones,
|
||||
"EN": num_zh_tones + num_ja_tones,
|
||||
}
|
||||
|
||||
if __name__ == "__main__":
|
||||
a = set(zh_symbols)
|
||||
b = set(en_symbols)
|
||||
print(sorted(a & b))
|
||||
769
oldVersion/V110/text/tone_sandhi.py
Normal file
769
oldVersion/V110/text/tone_sandhi.py
Normal file
@@ -0,0 +1,769 @@
|
||||
# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
from typing import List
|
||||
from typing import Tuple
|
||||
|
||||
import jieba
|
||||
from pypinyin import lazy_pinyin
|
||||
from pypinyin import Style
|
||||
|
||||
|
||||
class ToneSandhi:
|
||||
def __init__(self):
|
||||
self.must_neural_tone_words = {
|
||||
"麻烦",
|
||||
"麻利",
|
||||
"鸳鸯",
|
||||
"高粱",
|
||||
"骨头",
|
||||
"骆驼",
|
||||
"马虎",
|
||||
"首饰",
|
||||
"馒头",
|
||||
"馄饨",
|
||||
"风筝",
|
||||
"难为",
|
||||
"队伍",
|
||||
"阔气",
|
||||
"闺女",
|
||||
"门道",
|
||||
"锄头",
|
||||
"铺盖",
|
||||
"铃铛",
|
||||
"铁匠",
|
||||
"钥匙",
|
||||
"里脊",
|
||||
"里头",
|
||||
"部分",
|
||||
"那么",
|
||||
"道士",
|
||||
"造化",
|
||||
"迷糊",
|
||||
"连累",
|
||||
"这么",
|
||||
"这个",
|
||||
"运气",
|
||||
"过去",
|
||||
"软和",
|
||||
"转悠",
|
||||
"踏实",
|
||||
"跳蚤",
|
||||
"跟头",
|
||||
"趔趄",
|
||||
"财主",
|
||||
"豆腐",
|
||||
"讲究",
|
||||
"记性",
|
||||
"记号",
|
||||
"认识",
|
||||
"规矩",
|
||||
"见识",
|
||||
"裁缝",
|
||||
"补丁",
|
||||
"衣裳",
|
||||
"衣服",
|
||||
"衙门",
|
||||
"街坊",
|
||||
"行李",
|
||||
"行当",
|
||||
"蛤蟆",
|
||||
"蘑菇",
|
||||
"薄荷",
|
||||
"葫芦",
|
||||
"葡萄",
|
||||
"萝卜",
|
||||
"荸荠",
|
||||
"苗条",
|
||||
"苗头",
|
||||
"苍蝇",
|
||||
"芝麻",
|
||||
"舒服",
|
||||
"舒坦",
|
||||
"舌头",
|
||||
"自在",
|
||||
"膏药",
|
||||
"脾气",
|
||||
"脑袋",
|
||||
"脊梁",
|
||||
"能耐",
|
||||
"胳膊",
|
||||
"胭脂",
|
||||
"胡萝",
|
||||
"胡琴",
|
||||
"胡同",
|
||||
"聪明",
|
||||
"耽误",
|
||||
"耽搁",
|
||||
"耷拉",
|
||||
"耳朵",
|
||||
"老爷",
|
||||
"老实",
|
||||
"老婆",
|
||||
"老头",
|
||||
"老太",
|
||||
"翻腾",
|
||||
"罗嗦",
|
||||
"罐头",
|
||||
"编辑",
|
||||
"结实",
|
||||
"红火",
|
||||
"累赘",
|
||||
"糨糊",
|
||||
"糊涂",
|
||||
"精神",
|
||||
"粮食",
|
||||
"簸箕",
|
||||
"篱笆",
|
||||
"算计",
|
||||
"算盘",
|
||||
"答应",
|
||||
"笤帚",
|
||||
"笑语",
|
||||
"笑话",
|
||||
"窟窿",
|
||||
"窝囊",
|
||||
"窗户",
|
||||
"稳当",
|
||||
"稀罕",
|
||||
"称呼",
|
||||
"秧歌",
|
||||
"秀气",
|
||||
"秀才",
|
||||
"福气",
|
||||
"祖宗",
|
||||
"砚台",
|
||||
"码头",
|
||||
"石榴",
|
||||
"石头",
|
||||
"石匠",
|
||||
"知识",
|
||||
"眼睛",
|
||||
"眯缝",
|
||||
"眨巴",
|
||||
"眉毛",
|
||||
"相声",
|
||||
"盘算",
|
||||
"白净",
|
||||
"痢疾",
|
||||
"痛快",
|
||||
"疟疾",
|
||||
"疙瘩",
|
||||
"疏忽",
|
||||
"畜生",
|
||||
"生意",
|
||||
"甘蔗",
|
||||
"琵琶",
|
||||
"琢磨",
|
||||
"琉璃",
|
||||
"玻璃",
|
||||
"玫瑰",
|
||||
"玄乎",
|
||||
"狐狸",
|
||||
"状元",
|
||||
"特务",
|
||||
"牲口",
|
||||
"牙碜",
|
||||
"牌楼",
|
||||
"爽快",
|
||||
"爱人",
|
||||
"热闹",
|
||||
"烧饼",
|
||||
"烟筒",
|
||||
"烂糊",
|
||||
"点心",
|
||||
"炊帚",
|
||||
"灯笼",
|
||||
"火候",
|
||||
"漂亮",
|
||||
"滑溜",
|
||||
"溜达",
|
||||
"温和",
|
||||
"清楚",
|
||||
"消息",
|
||||
"浪头",
|
||||
"活泼",
|
||||
"比方",
|
||||
"正经",
|
||||
"欺负",
|
||||
"模糊",
|
||||
"槟榔",
|
||||
"棺材",
|
||||
"棒槌",
|
||||
"棉花",
|
||||
"核桃",
|
||||
"栅栏",
|
||||
"柴火",
|
||||
"架势",
|
||||
"枕头",
|
||||
"枇杷",
|
||||
"机灵",
|
||||
"本事",
|
||||
"木头",
|
||||
"木匠",
|
||||
"朋友",
|
||||
"月饼",
|
||||
"月亮",
|
||||
"暖和",
|
||||
"明白",
|
||||
"时候",
|
||||
"新鲜",
|
||||
"故事",
|
||||
"收拾",
|
||||
"收成",
|
||||
"提防",
|
||||
"挖苦",
|
||||
"挑剔",
|
||||
"指甲",
|
||||
"指头",
|
||||
"拾掇",
|
||||
"拳头",
|
||||
"拨弄",
|
||||
"招牌",
|
||||
"招呼",
|
||||
"抬举",
|
||||
"护士",
|
||||
"折腾",
|
||||
"扫帚",
|
||||
"打量",
|
||||
"打算",
|
||||
"打点",
|
||||
"打扮",
|
||||
"打听",
|
||||
"打发",
|
||||
"扎实",
|
||||
"扁担",
|
||||
"戒指",
|
||||
"懒得",
|
||||
"意识",
|
||||
"意思",
|
||||
"情形",
|
||||
"悟性",
|
||||
"怪物",
|
||||
"思量",
|
||||
"怎么",
|
||||
"念头",
|
||||
"念叨",
|
||||
"快活",
|
||||
"忙活",
|
||||
"志气",
|
||||
"心思",
|
||||
"得罪",
|
||||
"张罗",
|
||||
"弟兄",
|
||||
"开通",
|
||||
"应酬",
|
||||
"庄稼",
|
||||
"干事",
|
||||
"帮手",
|
||||
"帐篷",
|
||||
"希罕",
|
||||
"师父",
|
||||
"师傅",
|
||||
"巴结",
|
||||
"巴掌",
|
||||
"差事",
|
||||
"工夫",
|
||||
"岁数",
|
||||
"屁股",
|
||||
"尾巴",
|
||||
"少爷",
|
||||
"小气",
|
||||
"小伙",
|
||||
"将就",
|
||||
"对头",
|
||||
"对付",
|
||||
"寡妇",
|
||||
"家伙",
|
||||
"客气",
|
||||
"实在",
|
||||
"官司",
|
||||
"学问",
|
||||
"学生",
|
||||
"字号",
|
||||
"嫁妆",
|
||||
"媳妇",
|
||||
"媒人",
|
||||
"婆家",
|
||||
"娘家",
|
||||
"委屈",
|
||||
"姑娘",
|
||||
"姐夫",
|
||||
"妯娌",
|
||||
"妥当",
|
||||
"妖精",
|
||||
"奴才",
|
||||
"女婿",
|
||||
"头发",
|
||||
"太阳",
|
||||
"大爷",
|
||||
"大方",
|
||||
"大意",
|
||||
"大夫",
|
||||
"多少",
|
||||
"多么",
|
||||
"外甥",
|
||||
"壮实",
|
||||
"地道",
|
||||
"地方",
|
||||
"在乎",
|
||||
"困难",
|
||||
"嘴巴",
|
||||
"嘱咐",
|
||||
"嘟囔",
|
||||
"嘀咕",
|
||||
"喜欢",
|
||||
"喇嘛",
|
||||
"喇叭",
|
||||
"商量",
|
||||
"唾沫",
|
||||
"哑巴",
|
||||
"哈欠",
|
||||
"哆嗦",
|
||||
"咳嗽",
|
||||
"和尚",
|
||||
"告诉",
|
||||
"告示",
|
||||
"含糊",
|
||||
"吓唬",
|
||||
"后头",
|
||||
"名字",
|
||||
"名堂",
|
||||
"合同",
|
||||
"吆喝",
|
||||
"叫唤",
|
||||
"口袋",
|
||||
"厚道",
|
||||
"厉害",
|
||||
"千斤",
|
||||
"包袱",
|
||||
"包涵",
|
||||
"匀称",
|
||||
"勤快",
|
||||
"动静",
|
||||
"动弹",
|
||||
"功夫",
|
||||
"力气",
|
||||
"前头",
|
||||
"刺猬",
|
||||
"刺激",
|
||||
"别扭",
|
||||
"利落",
|
||||
"利索",
|
||||
"利害",
|
||||
"分析",
|
||||
"出息",
|
||||
"凑合",
|
||||
"凉快",
|
||||
"冷战",
|
||||
"冤枉",
|
||||
"冒失",
|
||||
"养活",
|
||||
"关系",
|
||||
"先生",
|
||||
"兄弟",
|
||||
"便宜",
|
||||
"使唤",
|
||||
"佩服",
|
||||
"作坊",
|
||||
"体面",
|
||||
"位置",
|
||||
"似的",
|
||||
"伙计",
|
||||
"休息",
|
||||
"什么",
|
||||
"人家",
|
||||
"亲戚",
|
||||
"亲家",
|
||||
"交情",
|
||||
"云彩",
|
||||
"事情",
|
||||
"买卖",
|
||||
"主意",
|
||||
"丫头",
|
||||
"丧气",
|
||||
"两口",
|
||||
"东西",
|
||||
"东家",
|
||||
"世故",
|
||||
"不由",
|
||||
"不在",
|
||||
"下水",
|
||||
"下巴",
|
||||
"上头",
|
||||
"上司",
|
||||
"丈夫",
|
||||
"丈人",
|
||||
"一辈",
|
||||
"那个",
|
||||
"菩萨",
|
||||
"父亲",
|
||||
"母亲",
|
||||
"咕噜",
|
||||
"邋遢",
|
||||
"费用",
|
||||
"冤家",
|
||||
"甜头",
|
||||
"介绍",
|
||||
"荒唐",
|
||||
"大人",
|
||||
"泥鳅",
|
||||
"幸福",
|
||||
"熟悉",
|
||||
"计划",
|
||||
"扑腾",
|
||||
"蜡烛",
|
||||
"姥爷",
|
||||
"照顾",
|
||||
"喉咙",
|
||||
"吉他",
|
||||
"弄堂",
|
||||
"蚂蚱",
|
||||
"凤凰",
|
||||
"拖沓",
|
||||
"寒碜",
|
||||
"糟蹋",
|
||||
"倒腾",
|
||||
"报复",
|
||||
"逻辑",
|
||||
"盘缠",
|
||||
"喽啰",
|
||||
"牢骚",
|
||||
"咖喱",
|
||||
"扫把",
|
||||
"惦记",
|
||||
}
|
||||
self.must_not_neural_tone_words = {
|
||||
"男子",
|
||||
"女子",
|
||||
"分子",
|
||||
"原子",
|
||||
"量子",
|
||||
"莲子",
|
||||
"石子",
|
||||
"瓜子",
|
||||
"电子",
|
||||
"人人",
|
||||
"虎虎",
|
||||
}
|
||||
self.punc = "：，；。？！“”‘’':,;.?!"
|
||||
|
||||
# the meaning of jieba pos tag: https://blog.csdn.net/weixin_44174352/article/details/113731041
|
||||
# e.g.
|
||||
# word: "家里"
|
||||
# pos: "s"
|
||||
# finals: ['ia1', 'i3']
|
||||
def _neural_sandhi(self, word: str, pos: str, finals: List[str]) -> List[str]:
|
||||
# reduplication words for n. and v. e.g. 奶奶, 试试, 旺旺
|
||||
for j, item in enumerate(word):
|
||||
if (
|
||||
j - 1 >= 0
|
||||
and item == word[j - 1]
|
||||
and pos[0] in {"n", "v", "a"}
|
||||
and word not in self.must_not_neural_tone_words
|
||||
):
|
||||
finals[j] = finals[j][:-1] + "5"
|
||||
ge_idx = word.find("个")
|
||||
if len(word) >= 1 and word[-1] in "吧呢啊呐噻嘛吖嗨呐哦哒额滴哩哟喽啰耶喔诶":
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
elif len(word) >= 1 and word[-1] in "的地得":
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
# e.g. 走了, 看着, 去过
|
||||
# elif len(word) == 1 and word in "了着过" and pos in {"ul", "uz", "ug"}:
|
||||
# finals[-1] = finals[-1][:-1] + "5"
|
||||
elif (
|
||||
len(word) > 1
|
||||
and word[-1] in "们子"
|
||||
and pos in {"r", "n"}
|
||||
and word not in self.must_not_neural_tone_words
|
||||
):
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
# e.g. 桌上, 地下, 家里
|
||||
elif len(word) > 1 and word[-1] in "上下里" and pos in {"s", "l", "f"}:
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
# e.g. 上来, 下去
|
||||
elif len(word) > 1 and word[-1] in "来去" and word[-2] in "上下进出回过起开":
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
# 个做量词
|
||||
elif (
|
||||
ge_idx >= 1
|
||||
and (word[ge_idx - 1].isnumeric() or word[ge_idx - 1] in "几有两半多各整每做是")
|
||||
) or word == "个":
|
||||
finals[ge_idx] = finals[ge_idx][:-1] + "5"
|
||||
else:
|
||||
if (
|
||||
word in self.must_neural_tone_words
|
||||
or word[-2:] in self.must_neural_tone_words
|
||||
):
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
|
||||
word_list = self._split_word(word)
|
||||
finals_list = [finals[: len(word_list[0])], finals[len(word_list[0]) :]]
|
||||
for i, word in enumerate(word_list):
|
||||
# conventional neural in Chinese
|
||||
if (
|
||||
word in self.must_neural_tone_words
|
||||
or word[-2:] in self.must_neural_tone_words
|
||||
):
|
||||
finals_list[i][-1] = finals_list[i][-1][:-1] + "5"
|
||||
finals = sum(finals_list, [])
|
||||
return finals
|
||||
|
||||
def _bu_sandhi(self, word: str, finals: List[str]) -> List[str]:
|
||||
# e.g. 看不懂
|
||||
if len(word) == 3 and word[1] == "不":
|
||||
finals[1] = finals[1][:-1] + "5"
|
||||
else:
|
||||
for i, char in enumerate(word):
|
||||
# "不" before tone4 should be bu2, e.g. 不怕
|
||||
if char == "不" and i + 1 < len(word) and finals[i + 1][-1] == "4":
|
||||
finals[i] = finals[i][:-1] + "2"
|
||||
return finals
|
||||
|
||||
def _yi_sandhi(self, word: str, finals: List[str]) -> List[str]:
|
||||
# "一" in number sequences, e.g. 一零零, 二一零
|
||||
if word.find("一") != -1 and all(
|
||||
[item.isnumeric() for item in word if item != "一"]
|
||||
):
|
||||
return finals
|
||||
# "一" between reduplication words should be yi5, e.g. 看一看
|
||||
elif len(word) == 3 and word[1] == "一" and word[0] == word[-1]:
|
||||
finals[1] = finals[1][:-1] + "5"
|
||||
# when "一" is ordinal word, it should be yi1
|
||||
elif word.startswith("第一"):
|
||||
finals[1] = finals[1][:-1] + "1"
|
||||
else:
|
||||
for i, char in enumerate(word):
|
||||
if char == "一" and i + 1 < len(word):
|
||||
# "一" before tone4 should be yi2, e.g. 一段
|
||||
if finals[i + 1][-1] == "4":
|
||||
finals[i] = finals[i][:-1] + "2"
|
||||
# "一" before non-tone4 should be yi4, e.g. 一天
|
||||
else:
|
||||
# "一" 后面如果是标点，还读一声
|
||||
if word[i + 1] not in self.punc:
|
||||
finals[i] = finals[i][:-1] + "4"
|
||||
return finals
|
||||
|
||||
def _split_word(self, word: str) -> List[str]:
|
||||
word_list = jieba.cut_for_search(word)
|
||||
word_list = sorted(word_list, key=lambda i: len(i), reverse=False)
|
||||
first_subword = word_list[0]
|
||||
first_begin_idx = word.find(first_subword)
|
||||
if first_begin_idx == 0:
|
||||
second_subword = word[len(first_subword) :]
|
||||
new_word_list = [first_subword, second_subword]
|
||||
else:
|
||||
second_subword = word[: -len(first_subword)]
|
||||
new_word_list = [second_subword, first_subword]
|
||||
return new_word_list
|
||||
|
||||
def _three_sandhi(self, word: str, finals: List[str]) -> List[str]:
|
||||
if len(word) == 2 and self._all_tone_three(finals):
|
||||
finals[0] = finals[0][:-1] + "2"
|
||||
elif len(word) == 3:
|
||||
word_list = self._split_word(word)
|
||||
if self._all_tone_three(finals):
|
||||
# disyllabic + monosyllabic, e.g. 蒙古/包
|
||||
if len(word_list[0]) == 2:
|
||||
finals[0] = finals[0][:-1] + "2"
|
||||
finals[1] = finals[1][:-1] + "2"
|
||||
# monosyllabic + disyllabic, e.g. 纸/老虎
|
||||
elif len(word_list[0]) == 1:
|
||||
finals[1] = finals[1][:-1] + "2"
|
||||
else:
|
||||
finals_list = [finals[: len(word_list[0])], finals[len(word_list[0]) :]]
|
||||
if len(finals_list) == 2:
|
||||
for i, sub in enumerate(finals_list):
|
||||
# e.g. 所有/人
|
||||
if self._all_tone_three(sub) and len(sub) == 2:
|
||||
finals_list[i][0] = finals_list[i][0][:-1] + "2"
|
||||
# e.g. 好/喜欢
|
||||
elif (
|
||||
i == 1
|
||||
and not self._all_tone_three(sub)
|
||||
and finals_list[i][0][-1] == "3"
|
||||
and finals_list[0][-1][-1] == "3"
|
||||
):
|
||||
finals_list[0][-1] = finals_list[0][-1][:-1] + "2"
|
||||
finals = sum(finals_list, [])
|
||||
# split idiom into two words who's length is 2
|
||||
elif len(word) == 4:
|
||||
finals_list = [finals[:2], finals[2:]]
|
||||
finals = []
|
||||
for sub in finals_list:
|
||||
if self._all_tone_three(sub):
|
||||
sub[0] = sub[0][:-1] + "2"
|
||||
finals += sub
|
||||
|
||||
return finals
|
||||
|
||||
def _all_tone_three(self, finals: List[str]) -> bool:
|
||||
return all(x[-1] == "3" for x in finals)
|
||||
|
||||
# merge "不" and the word behind it
|
||||
# if don't merge, "不" sometimes appears alone according to jieba, which may occur sandhi error
|
||||
def _merge_bu(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
last_word = ""
|
||||
for word, pos in seg:
|
||||
if last_word == "不":
|
||||
word = last_word + word
|
||||
if word != "不":
|
||||
new_seg.append((word, pos))
|
||||
last_word = word[:]
|
||||
if last_word == "不":
|
||||
new_seg.append((last_word, "d"))
|
||||
last_word = ""
|
||||
return new_seg
|
||||
|
||||
# function 1: merge "一" and reduplication words in it's left and right, e.g. "听","一","听" ->"听一听"
|
||||
# function 2: merge single "一" and the word behind it
|
||||
# if don't merge, "一" sometimes appears alone according to jieba, which may occur sandhi error
|
||||
# e.g.
|
||||
# input seg: [('听', 'v'), ('一', 'm'), ('听', 'v')]
|
||||
# output seg: [['听一听', 'v']]
|
||||
def _merge_yi(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
# function 1
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if (
|
||||
i - 1 >= 0
|
||||
and word == "一"
|
||||
and i + 1 < len(seg)
|
||||
and seg[i - 1][0] == seg[i + 1][0]
|
||||
and seg[i - 1][1] == "v"
|
||||
):
|
||||
new_seg[i - 1][0] = new_seg[i - 1][0] + "一" + new_seg[i - 1][0]
|
||||
else:
|
||||
if (
|
||||
i - 2 >= 0
|
||||
and seg[i - 1][0] == "一"
|
||||
and seg[i - 2][0] == word
|
||||
and pos == "v"
|
||||
):
|
||||
continue
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
seg = new_seg
|
||||
new_seg = []
|
||||
# function 2
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if new_seg and new_seg[-1][0] == "一":
|
||||
new_seg[-1][0] = new_seg[-1][0] + word
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
return new_seg
|
||||
|
||||
# the first and the second words are all_tone_three
|
||||
def _merge_continuous_three_tones(
|
||||
self, seg: List[Tuple[str, str]]
|
||||
) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
sub_finals_list = [
|
||||
lazy_pinyin(word, neutral_tone_with_five=True, style=Style.FINALS_TONE3)
|
||||
for (word, pos) in seg
|
||||
]
|
||||
assert len(sub_finals_list) == len(seg)
|
||||
merge_last = [False] * len(seg)
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if (
|
||||
i - 1 >= 0
|
||||
and self._all_tone_three(sub_finals_list[i - 1])
|
||||
and self._all_tone_three(sub_finals_list[i])
|
||||
and not merge_last[i - 1]
|
||||
):
|
||||
# if the last word is reduplication, not merge, because reduplication need to be _neural_sandhi
|
||||
if (
|
||||
not self._is_reduplication(seg[i - 1][0])
|
||||
and len(seg[i - 1][0]) + len(seg[i][0]) <= 3
|
||||
):
|
||||
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
|
||||
merge_last[i] = True
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
|
||||
return new_seg
|
||||
|
||||
def _is_reduplication(self, word: str) -> bool:
|
||||
return len(word) == 2 and word[0] == word[1]
|
||||
|
||||
# the last char of first word and the first char of second word is tone_three
|
||||
def _merge_continuous_three_tones_2(
|
||||
self, seg: List[Tuple[str, str]]
|
||||
) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
sub_finals_list = [
|
||||
lazy_pinyin(word, neutral_tone_with_five=True, style=Style.FINALS_TONE3)
|
||||
for (word, pos) in seg
|
||||
]
|
||||
assert len(sub_finals_list) == len(seg)
|
||||
merge_last = [False] * len(seg)
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if (
|
||||
i - 1 >= 0
|
||||
and sub_finals_list[i - 1][-1][-1] == "3"
|
||||
and sub_finals_list[i][0][-1] == "3"
|
||||
and not merge_last[i - 1]
|
||||
):
|
||||
# if the last word is reduplication, not merge, because reduplication need to be _neural_sandhi
|
||||
if (
|
||||
not self._is_reduplication(seg[i - 1][0])
|
||||
and len(seg[i - 1][0]) + len(seg[i][0]) <= 3
|
||||
):
|
||||
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
|
||||
merge_last[i] = True
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
return new_seg
|
||||
|
||||
def _merge_er(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if i - 1 >= 0 and word == "儿" and seg[i - 1][0] != "#":
|
||||
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
return new_seg
|
||||
|
||||
def _merge_reduplication(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if new_seg and word == new_seg[-1][0]:
|
||||
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
return new_seg
|
||||
|
||||
def pre_merge_for_modify(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
seg = self._merge_bu(seg)
|
||||
try:
|
||||
seg = self._merge_yi(seg)
|
||||
except:
|
||||
print("_merge_yi failed")
|
||||
seg = self._merge_reduplication(seg)
|
||||
seg = self._merge_continuous_three_tones(seg)
|
||||
seg = self._merge_continuous_three_tones_2(seg)
|
||||
seg = self._merge_er(seg)
|
||||
return seg
|
||||
|
||||
def modified_tone(self, word: str, pos: str, finals: List[str]) -> List[str]:
|
||||
finals = self._bu_sandhi(word, finals)
|
||||
finals = self._yi_sandhi(word, finals)
|
||||
finals = self._neural_sandhi(word, pos, finals)
|
||||
finals = self._three_sandhi(word, finals)
|
||||
return finals
|
||||
171
oldVersion/V111/__init__.py
Normal file
171
oldVersion/V111/__init__.py
Normal file
@@ -0,0 +1,171 @@
|
||||
"""
|
||||
1.1.1版本兼容
|
||||
https://github.com/fishaudio/Bert-VITS2/releases/tag/1.1.1
|
||||
"""
|
||||
import torch
|
||||
import commons
|
||||
from .text.cleaner import clean_text, clean_text_fix
|
||||
from .text import cleaned_text_to_sequence
|
||||
from .text import get_bert, get_bert_fix
|
||||
|
||||
|
||||
def get_text(text, language_str, hps, device):
|
||||
norm_text, phone, tone, word2ph = clean_text(text, language_str)
|
||||
phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
|
||||
|
||||
if hps.data.add_blank:
|
||||
phone = commons.intersperse(phone, 0)
|
||||
tone = commons.intersperse(tone, 0)
|
||||
language = commons.intersperse(language, 0)
|
||||
for i in range(len(word2ph)):
|
||||
word2ph[i] = word2ph[i] * 2
|
||||
word2ph[0] += 1
|
||||
bert = get_bert(norm_text, word2ph, language_str, device)
|
||||
del word2ph
|
||||
assert bert.shape[-1] == len(phone), phone
|
||||
|
||||
if language_str == "ZH":
|
||||
bert = bert
|
||||
ja_bert = torch.zeros(768, len(phone))
|
||||
elif language_str == "JP":
|
||||
ja_bert = bert
|
||||
bert = torch.zeros(1024, len(phone))
|
||||
else:
|
||||
bert = torch.zeros(1024, len(phone))
|
||||
ja_bert = torch.zeros(768, len(phone))
|
||||
|
||||
assert bert.shape[-1] == len(
|
||||
phone
|
||||
), f"Bert seq len {bert.shape[-1]} != {len(phone)}"
|
||||
|
||||
phone = torch.LongTensor(phone)
|
||||
tone = torch.LongTensor(tone)
|
||||
language = torch.LongTensor(language)
|
||||
return bert, ja_bert, phone, tone, language
|
||||
|
||||
|
||||
def get_text_fix(text, language_str, hps, device):
|
||||
norm_text, phone, tone, word2ph = clean_text_fix(text, language_str)
|
||||
phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
|
||||
|
||||
if hps.data.add_blank:
|
||||
phone = commons.intersperse(phone, 0)
|
||||
tone = commons.intersperse(tone, 0)
|
||||
language = commons.intersperse(language, 0)
|
||||
for i in range(len(word2ph)):
|
||||
word2ph[i] = word2ph[i] * 2
|
||||
word2ph[0] += 1
|
||||
bert = get_bert_fix(norm_text, word2ph, language_str, device)
|
||||
del word2ph
|
||||
assert bert.shape[-1] == len(phone), phone
|
||||
|
||||
if language_str == "ZH":
|
||||
bert = bert
|
||||
ja_bert = torch.zeros(768, len(phone))
|
||||
elif language_str == "JP":
|
||||
ja_bert = bert
|
||||
bert = torch.zeros(1024, len(phone))
|
||||
else:
|
||||
bert = torch.zeros(1024, len(phone))
|
||||
ja_bert = torch.zeros(768, len(phone))
|
||||
|
||||
assert bert.shape[-1] == len(
|
||||
phone
|
||||
), f"Bert seq len {bert.shape[-1]} != {len(phone)}"
|
||||
|
||||
phone = torch.LongTensor(phone)
|
||||
tone = torch.LongTensor(tone)
|
||||
language = torch.LongTensor(language)
|
||||
return bert, ja_bert, phone, tone, language
|
||||
|
||||
|
||||
def infer(
|
||||
text,
|
||||
sdp_ratio,
|
||||
noise_scale,
|
||||
noise_scale_w,
|
||||
length_scale,
|
||||
sid,
|
||||
language,
|
||||
hps,
|
||||
net_g,
|
||||
device,
|
||||
):
|
||||
bert, ja_bert, phones, tones, lang_ids = get_text(text, language, hps, device)
|
||||
with torch.no_grad():
|
||||
x_tst = phones.to(device).unsqueeze(0)
|
||||
tones = tones.to(device).unsqueeze(0)
|
||||
lang_ids = lang_ids.to(device).unsqueeze(0)
|
||||
bert = bert.to(device).unsqueeze(0)
|
||||
ja_bert = ja_bert.to(device).unsqueeze(0)
|
||||
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
|
||||
del phones
|
||||
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
|
||||
audio = (
|
||||
net_g.infer(
|
||||
x_tst,
|
||||
x_tst_lengths,
|
||||
speakers,
|
||||
tones,
|
||||
lang_ids,
|
||||
bert,
|
||||
ja_bert,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise_scale,
|
||||
noise_scale_w=noise_scale_w,
|
||||
length_scale=length_scale,
|
||||
)[0][0, 0]
|
||||
.data.cpu()
|
||||
.float()
|
||||
.numpy()
|
||||
)
|
||||
del x_tst, x_tst_lengths, speakers, tones, lang_ids, bert, ja_bert
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
return audio
|
||||
|
||||
|
||||
def infer_fix(
|
||||
text,
|
||||
sdp_ratio,
|
||||
noise_scale,
|
||||
noise_scale_w,
|
||||
length_scale,
|
||||
sid,
|
||||
language,
|
||||
hps,
|
||||
net_g,
|
||||
device,
|
||||
):
|
||||
bert, ja_bert, phones, tones, lang_ids = get_text_fix(text, language, hps, device)
|
||||
with torch.no_grad():
|
||||
x_tst = phones.to(device).unsqueeze(0)
|
||||
tones = tones.to(device).unsqueeze(0)
|
||||
lang_ids = lang_ids.to(device).unsqueeze(0)
|
||||
bert = bert.to(device).unsqueeze(0)
|
||||
ja_bert = ja_bert.to(device).unsqueeze(0)
|
||||
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
|
||||
del phones
|
||||
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
|
||||
audio = (
|
||||
net_g.infer(
|
||||
x_tst,
|
||||
x_tst_lengths,
|
||||
speakers,
|
||||
tones,
|
||||
lang_ids,
|
||||
bert,
|
||||
ja_bert,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise_scale,
|
||||
noise_scale_w=noise_scale_w,
|
||||
length_scale=length_scale,
|
||||
)[0][0, 0]
|
||||
.data.cpu()
|
||||
.float()
|
||||
.numpy()
|
||||
)
|
||||
del x_tst, x_tst_lengths, speakers, tones, lang_ids, bert, ja_bert
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
return audio
|
||||
986
oldVersion/V111/models.py
Normal file
986
oldVersion/V111/models.py
Normal file
@@ -0,0 +1,986 @@
|
||||
import math
|
||||
import torch
|
||||
from torch import nn
|
||||
from torch.nn import functional as F
|
||||
|
||||
import commons
|
||||
import modules
|
||||
import attentions
|
||||
import monotonic_align
|
||||
|
||||
from torch.nn import Conv1d, ConvTranspose1d, Conv2d
|
||||
from torch.nn.utils import weight_norm, remove_weight_norm, spectral_norm
|
||||
|
||||
from commons import init_weights, get_padding
|
||||
from .text import symbols, num_tones, num_languages
|
||||
|
||||
|
||||
class DurationDiscriminator(nn.Module): # vits2
|
||||
def __init__(
|
||||
self, in_channels, filter_channels, kernel_size, p_dropout, gin_channels=0
|
||||
):
|
||||
super().__init__()
|
||||
|
||||
self.in_channels = in_channels
|
||||
self.filter_channels = filter_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.p_dropout = p_dropout
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.drop = nn.Dropout(p_dropout)
|
||||
self.conv_1 = nn.Conv1d(
|
||||
in_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.norm_1 = modules.LayerNorm(filter_channels)
|
||||
self.conv_2 = nn.Conv1d(
|
||||
filter_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.norm_2 = modules.LayerNorm(filter_channels)
|
||||
self.dur_proj = nn.Conv1d(1, filter_channels, 1)
|
||||
|
||||
self.pre_out_conv_1 = nn.Conv1d(
|
||||
2 * filter_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.pre_out_norm_1 = modules.LayerNorm(filter_channels)
|
||||
self.pre_out_conv_2 = nn.Conv1d(
|
||||
filter_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.pre_out_norm_2 = modules.LayerNorm(filter_channels)
|
||||
|
||||
if gin_channels != 0:
|
||||
self.cond = nn.Conv1d(gin_channels, in_channels, 1)
|
||||
|
||||
self.output_layer = nn.Sequential(nn.Linear(filter_channels, 1), nn.Sigmoid())
|
||||
|
||||
def forward_probability(self, x, x_mask, dur, g=None):
|
||||
dur = self.dur_proj(dur)
|
||||
x = torch.cat([x, dur], dim=1)
|
||||
x = self.pre_out_conv_1(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.pre_out_norm_1(x)
|
||||
x = self.drop(x)
|
||||
x = self.pre_out_conv_2(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.pre_out_norm_2(x)
|
||||
x = self.drop(x)
|
||||
x = x * x_mask
|
||||
x = x.transpose(1, 2)
|
||||
output_prob = self.output_layer(x)
|
||||
return output_prob
|
||||
|
||||
def forward(self, x, x_mask, dur_r, dur_hat, g=None):
|
||||
x = torch.detach(x)
|
||||
if g is not None:
|
||||
g = torch.detach(g)
|
||||
x = x + self.cond(g)
|
||||
x = self.conv_1(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.norm_1(x)
|
||||
x = self.drop(x)
|
||||
x = self.conv_2(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.norm_2(x)
|
||||
x = self.drop(x)
|
||||
|
||||
output_probs = []
|
||||
for dur in [dur_r, dur_hat]:
|
||||
output_prob = self.forward_probability(x, x_mask, dur, g)
|
||||
output_probs.append(output_prob)
|
||||
|
||||
return output_probs
|
||||
|
||||
|
||||
class TransformerCouplingBlock(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
channels,
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
n_flows=4,
|
||||
gin_channels=0,
|
||||
share_parameter=False,
|
||||
):
|
||||
super().__init__()
|
||||
self.channels = channels
|
||||
self.hidden_channels = hidden_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.n_layers = n_layers
|
||||
self.n_flows = n_flows
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.flows = nn.ModuleList()
|
||||
|
||||
self.wn = (
|
||||
attentions.FFT(
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
isflow=True,
|
||||
gin_channels=self.gin_channels,
|
||||
)
|
||||
if share_parameter
|
||||
else None
|
||||
)
|
||||
|
||||
for i in range(n_flows):
|
||||
self.flows.append(
|
||||
modules.TransformerCouplingLayer(
|
||||
channels,
|
||||
hidden_channels,
|
||||
kernel_size,
|
||||
n_layers,
|
||||
n_heads,
|
||||
p_dropout,
|
||||
filter_channels,
|
||||
mean_only=True,
|
||||
wn_sharing_parameter=self.wn,
|
||||
gin_channels=self.gin_channels,
|
||||
)
|
||||
)
|
||||
self.flows.append(modules.Flip())
|
||||
|
||||
def forward(self, x, x_mask, g=None, reverse=False):
|
||||
if not reverse:
|
||||
for flow in self.flows:
|
||||
x, _ = flow(x, x_mask, g=g, reverse=reverse)
|
||||
else:
|
||||
for flow in reversed(self.flows):
|
||||
x = flow(x, x_mask, g=g, reverse=reverse)
|
||||
return x
|
||||
|
||||
|
||||
class StochasticDurationPredictor(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
in_channels,
|
||||
filter_channels,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
n_flows=4,
|
||||
gin_channels=0,
|
||||
):
|
||||
super().__init__()
|
||||
filter_channels = in_channels # it needs to be removed from future version.
|
||||
self.in_channels = in_channels
|
||||
self.filter_channels = filter_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.p_dropout = p_dropout
|
||||
self.n_flows = n_flows
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.log_flow = modules.Log()
|
||||
self.flows = nn.ModuleList()
|
||||
self.flows.append(modules.ElementwiseAffine(2))
|
||||
for i in range(n_flows):
|
||||
self.flows.append(
|
||||
modules.ConvFlow(2, filter_channels, kernel_size, n_layers=3)
|
||||
)
|
||||
self.flows.append(modules.Flip())
|
||||
|
||||
self.post_pre = nn.Conv1d(1, filter_channels, 1)
|
||||
self.post_proj = nn.Conv1d(filter_channels, filter_channels, 1)
|
||||
self.post_convs = modules.DDSConv(
|
||||
filter_channels, kernel_size, n_layers=3, p_dropout=p_dropout
|
||||
)
|
||||
self.post_flows = nn.ModuleList()
|
||||
self.post_flows.append(modules.ElementwiseAffine(2))
|
||||
for i in range(4):
|
||||
self.post_flows.append(
|
||||
modules.ConvFlow(2, filter_channels, kernel_size, n_layers=3)
|
||||
)
|
||||
self.post_flows.append(modules.Flip())
|
||||
|
||||
self.pre = nn.Conv1d(in_channels, filter_channels, 1)
|
||||
self.proj = nn.Conv1d(filter_channels, filter_channels, 1)
|
||||
self.convs = modules.DDSConv(
|
||||
filter_channels, kernel_size, n_layers=3, p_dropout=p_dropout
|
||||
)
|
||||
if gin_channels != 0:
|
||||
self.cond = nn.Conv1d(gin_channels, filter_channels, 1)
|
||||
|
||||
def forward(self, x, x_mask, w=None, g=None, reverse=False, noise_scale=1.0):
|
||||
x = torch.detach(x)
|
||||
x = self.pre(x)
|
||||
if g is not None:
|
||||
g = torch.detach(g)
|
||||
x = x + self.cond(g)
|
||||
x = self.convs(x, x_mask)
|
||||
x = self.proj(x) * x_mask
|
||||
|
||||
if not reverse:
|
||||
flows = self.flows
|
||||
assert w is not None
|
||||
|
||||
logdet_tot_q = 0
|
||||
h_w = self.post_pre(w)
|
||||
h_w = self.post_convs(h_w, x_mask)
|
||||
h_w = self.post_proj(h_w) * x_mask
|
||||
e_q = (
|
||||
torch.randn(w.size(0), 2, w.size(2)).to(device=x.device, dtype=x.dtype)
|
||||
* x_mask
|
||||
)
|
||||
z_q = e_q
|
||||
for flow in self.post_flows:
|
||||
z_q, logdet_q = flow(z_q, x_mask, g=(x + h_w))
|
||||
logdet_tot_q += logdet_q
|
||||
z_u, z1 = torch.split(z_q, [1, 1], 1)
|
||||
u = torch.sigmoid(z_u) * x_mask
|
||||
z0 = (w - u) * x_mask
|
||||
logdet_tot_q += torch.sum(
|
||||
(F.logsigmoid(z_u) + F.logsigmoid(-z_u)) * x_mask, [1, 2]
|
||||
)
|
||||
logq = (
|
||||
torch.sum(-0.5 * (math.log(2 * math.pi) + (e_q**2)) * x_mask, [1, 2])
|
||||
- logdet_tot_q
|
||||
)
|
||||
|
||||
logdet_tot = 0
|
||||
z0, logdet = self.log_flow(z0, x_mask)
|
||||
logdet_tot += logdet
|
||||
z = torch.cat([z0, z1], 1)
|
||||
for flow in flows:
|
||||
z, logdet = flow(z, x_mask, g=x, reverse=reverse)
|
||||
logdet_tot = logdet_tot + logdet
|
||||
nll = (
|
||||
torch.sum(0.5 * (math.log(2 * math.pi) + (z**2)) * x_mask, [1, 2])
|
||||
- logdet_tot
|
||||
)
|
||||
return nll + logq # [b]
|
||||
else:
|
||||
flows = list(reversed(self.flows))
|
||||
flows = flows[:-2] + [flows[-1]] # remove a useless vflow
|
||||
z = (
|
||||
torch.randn(x.size(0), 2, x.size(2)).to(device=x.device, dtype=x.dtype)
|
||||
* noise_scale
|
||||
)
|
||||
for flow in flows:
|
||||
z = flow(z, x_mask, g=x, reverse=reverse)
|
||||
z0, z1 = torch.split(z, [1, 1], 1)
|
||||
logw = z0
|
||||
return logw
|
||||
|
||||
|
||||
class DurationPredictor(nn.Module):
|
||||
def __init__(
|
||||
self, in_channels, filter_channels, kernel_size, p_dropout, gin_channels=0
|
||||
):
|
||||
super().__init__()
|
||||
|
||||
self.in_channels = in_channels
|
||||
self.filter_channels = filter_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.p_dropout = p_dropout
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.drop = nn.Dropout(p_dropout)
|
||||
self.conv_1 = nn.Conv1d(
|
||||
in_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.norm_1 = modules.LayerNorm(filter_channels)
|
||||
self.conv_2 = nn.Conv1d(
|
||||
filter_channels, filter_channels, kernel_size, padding=kernel_size // 2
|
||||
)
|
||||
self.norm_2 = modules.LayerNorm(filter_channels)
|
||||
self.proj = nn.Conv1d(filter_channels, 1, 1)
|
||||
|
||||
if gin_channels != 0:
|
||||
self.cond = nn.Conv1d(gin_channels, in_channels, 1)
|
||||
|
||||
def forward(self, x, x_mask, g=None):
|
||||
x = torch.detach(x)
|
||||
if g is not None:
|
||||
g = torch.detach(g)
|
||||
x = x + self.cond(g)
|
||||
x = self.conv_1(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.norm_1(x)
|
||||
x = self.drop(x)
|
||||
x = self.conv_2(x * x_mask)
|
||||
x = torch.relu(x)
|
||||
x = self.norm_2(x)
|
||||
x = self.drop(x)
|
||||
x = self.proj(x * x_mask)
|
||||
return x * x_mask
|
||||
|
||||
|
||||
class TextEncoder(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
n_vocab,
|
||||
out_channels,
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
gin_channels=0,
|
||||
):
|
||||
super().__init__()
|
||||
self.n_vocab = n_vocab
|
||||
self.out_channels = out_channels
|
||||
self.hidden_channels = hidden_channels
|
||||
self.filter_channels = filter_channels
|
||||
self.n_heads = n_heads
|
||||
self.n_layers = n_layers
|
||||
self.kernel_size = kernel_size
|
||||
self.p_dropout = p_dropout
|
||||
self.gin_channels = gin_channels
|
||||
self.emb = nn.Embedding(len(symbols), hidden_channels)
|
||||
nn.init.normal_(self.emb.weight, 0.0, hidden_channels**-0.5)
|
||||
self.tone_emb = nn.Embedding(num_tones, hidden_channels)
|
||||
nn.init.normal_(self.tone_emb.weight, 0.0, hidden_channels**-0.5)
|
||||
self.language_emb = nn.Embedding(num_languages, hidden_channels)
|
||||
nn.init.normal_(self.language_emb.weight, 0.0, hidden_channels**-0.5)
|
||||
self.bert_proj = nn.Conv1d(1024, hidden_channels, 1)
|
||||
self.ja_bert_proj = nn.Conv1d(768, hidden_channels, 1)
|
||||
|
||||
self.encoder = attentions.Encoder(
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
gin_channels=self.gin_channels,
|
||||
)
|
||||
self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
|
||||
|
||||
def forward(self, x, x_lengths, tone, language, bert, ja_bert, g=None):
|
||||
bert_emb = self.bert_proj(bert).transpose(1, 2)
|
||||
ja_bert_emb = self.ja_bert_proj(ja_bert).transpose(1, 2)
|
||||
x = (
|
||||
self.emb(x)
|
||||
+ self.tone_emb(tone)
|
||||
+ self.language_emb(language)
|
||||
+ bert_emb
|
||||
+ ja_bert_emb
|
||||
) * math.sqrt(
|
||||
self.hidden_channels
|
||||
) # [b, t, h]
|
||||
x = torch.transpose(x, 1, -1) # [b, h, t]
|
||||
x_mask = torch.unsqueeze(commons.sequence_mask(x_lengths, x.size(2)), 1).to(
|
||||
x.dtype
|
||||
)
|
||||
|
||||
x = self.encoder(x * x_mask, x_mask, g=g)
|
||||
stats = self.proj(x) * x_mask
|
||||
|
||||
m, logs = torch.split(stats, self.out_channels, dim=1)
|
||||
return x, m, logs, x_mask
|
||||
|
||||
|
||||
class ResidualCouplingBlock(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
channels,
|
||||
hidden_channels,
|
||||
kernel_size,
|
||||
dilation_rate,
|
||||
n_layers,
|
||||
n_flows=4,
|
||||
gin_channels=0,
|
||||
):
|
||||
super().__init__()
|
||||
self.channels = channels
|
||||
self.hidden_channels = hidden_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.dilation_rate = dilation_rate
|
||||
self.n_layers = n_layers
|
||||
self.n_flows = n_flows
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.flows = nn.ModuleList()
|
||||
for i in range(n_flows):
|
||||
self.flows.append(
|
||||
modules.ResidualCouplingLayer(
|
||||
channels,
|
||||
hidden_channels,
|
||||
kernel_size,
|
||||
dilation_rate,
|
||||
n_layers,
|
||||
gin_channels=gin_channels,
|
||||
mean_only=True,
|
||||
)
|
||||
)
|
||||
self.flows.append(modules.Flip())
|
||||
|
||||
def forward(self, x, x_mask, g=None, reverse=False):
|
||||
if not reverse:
|
||||
for flow in self.flows:
|
||||
x, _ = flow(x, x_mask, g=g, reverse=reverse)
|
||||
else:
|
||||
for flow in reversed(self.flows):
|
||||
x = flow(x, x_mask, g=g, reverse=reverse)
|
||||
return x
|
||||
|
||||
|
||||
class PosteriorEncoder(nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
in_channels,
|
||||
out_channels,
|
||||
hidden_channels,
|
||||
kernel_size,
|
||||
dilation_rate,
|
||||
n_layers,
|
||||
gin_channels=0,
|
||||
):
|
||||
super().__init__()
|
||||
self.in_channels = in_channels
|
||||
self.out_channels = out_channels
|
||||
self.hidden_channels = hidden_channels
|
||||
self.kernel_size = kernel_size
|
||||
self.dilation_rate = dilation_rate
|
||||
self.n_layers = n_layers
|
||||
self.gin_channels = gin_channels
|
||||
|
||||
self.pre = nn.Conv1d(in_channels, hidden_channels, 1)
|
||||
self.enc = modules.WN(
|
||||
hidden_channels,
|
||||
kernel_size,
|
||||
dilation_rate,
|
||||
n_layers,
|
||||
gin_channels=gin_channels,
|
||||
)
|
||||
self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
|
||||
|
||||
def forward(self, x, x_lengths, g=None):
|
||||
x_mask = torch.unsqueeze(commons.sequence_mask(x_lengths, x.size(2)), 1).to(
|
||||
x.dtype
|
||||
)
|
||||
x = self.pre(x) * x_mask
|
||||
x = self.enc(x, x_mask, g=g)
|
||||
stats = self.proj(x) * x_mask
|
||||
m, logs = torch.split(stats, self.out_channels, dim=1)
|
||||
z = (m + torch.randn_like(m) * torch.exp(logs)) * x_mask
|
||||
return z, m, logs, x_mask
|
||||
|
||||
|
||||
class Generator(torch.nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
initial_channel,
|
||||
resblock,
|
||||
resblock_kernel_sizes,
|
||||
resblock_dilation_sizes,
|
||||
upsample_rates,
|
||||
upsample_initial_channel,
|
||||
upsample_kernel_sizes,
|
||||
gin_channels=0,
|
||||
):
|
||||
super(Generator, self).__init__()
|
||||
self.num_kernels = len(resblock_kernel_sizes)
|
||||
self.num_upsamples = len(upsample_rates)
|
||||
self.conv_pre = Conv1d(
|
||||
initial_channel, upsample_initial_channel, 7, 1, padding=3
|
||||
)
|
||||
resblock = modules.ResBlock1 if resblock == "1" else modules.ResBlock2
|
||||
|
||||
self.ups = nn.ModuleList()
|
||||
for i, (u, k) in enumerate(zip(upsample_rates, upsample_kernel_sizes)):
|
||||
self.ups.append(
|
||||
weight_norm(
|
||||
ConvTranspose1d(
|
||||
upsample_initial_channel // (2**i),
|
||||
upsample_initial_channel // (2 ** (i + 1)),
|
||||
k,
|
||||
u,
|
||||
padding=(k - u) // 2,
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
self.resblocks = nn.ModuleList()
|
||||
for i in range(len(self.ups)):
|
||||
ch = upsample_initial_channel // (2 ** (i + 1))
|
||||
for j, (k, d) in enumerate(
|
||||
zip(resblock_kernel_sizes, resblock_dilation_sizes)
|
||||
):
|
||||
self.resblocks.append(resblock(ch, k, d))
|
||||
|
||||
self.conv_post = Conv1d(ch, 1, 7, 1, padding=3, bias=False)
|
||||
self.ups.apply(init_weights)
|
||||
|
||||
if gin_channels != 0:
|
||||
self.cond = nn.Conv1d(gin_channels, upsample_initial_channel, 1)
|
||||
|
||||
def forward(self, x, g=None):
|
||||
x = self.conv_pre(x)
|
||||
if g is not None:
|
||||
x = x + self.cond(g)
|
||||
|
||||
for i in range(self.num_upsamples):
|
||||
x = F.leaky_relu(x, modules.LRELU_SLOPE)
|
||||
x = self.ups[i](x)
|
||||
xs = None
|
||||
for j in range(self.num_kernels):
|
||||
if xs is None:
|
||||
xs = self.resblocks[i * self.num_kernels + j](x)
|
||||
else:
|
||||
xs += self.resblocks[i * self.num_kernels + j](x)
|
||||
x = xs / self.num_kernels
|
||||
x = F.leaky_relu(x)
|
||||
x = self.conv_post(x)
|
||||
x = torch.tanh(x)
|
||||
|
||||
return x
|
||||
|
||||
def remove_weight_norm(self):
|
||||
print("Removing weight norm...")
|
||||
for layer in self.ups:
|
||||
remove_weight_norm(layer)
|
||||
for layer in self.resblocks:
|
||||
layer.remove_weight_norm()
|
||||
|
||||
|
||||
class DiscriminatorP(torch.nn.Module):
|
||||
def __init__(self, period, kernel_size=5, stride=3, use_spectral_norm=False):
|
||||
super(DiscriminatorP, self).__init__()
|
||||
self.period = period
|
||||
self.use_spectral_norm = use_spectral_norm
|
||||
norm_f = weight_norm if use_spectral_norm is False else spectral_norm
|
||||
self.convs = nn.ModuleList(
|
||||
[
|
||||
norm_f(
|
||||
Conv2d(
|
||||
1,
|
||||
32,
|
||||
(kernel_size, 1),
|
||||
(stride, 1),
|
||||
padding=(get_padding(kernel_size, 1), 0),
|
||||
)
|
||||
),
|
||||
norm_f(
|
||||
Conv2d(
|
||||
32,
|
||||
128,
|
||||
(kernel_size, 1),
|
||||
(stride, 1),
|
||||
padding=(get_padding(kernel_size, 1), 0),
|
||||
)
|
||||
),
|
||||
norm_f(
|
||||
Conv2d(
|
||||
128,
|
||||
512,
|
||||
(kernel_size, 1),
|
||||
(stride, 1),
|
||||
padding=(get_padding(kernel_size, 1), 0),
|
||||
)
|
||||
),
|
||||
norm_f(
|
||||
Conv2d(
|
||||
512,
|
||||
1024,
|
||||
(kernel_size, 1),
|
||||
(stride, 1),
|
||||
padding=(get_padding(kernel_size, 1), 0),
|
||||
)
|
||||
),
|
||||
norm_f(
|
||||
Conv2d(
|
||||
1024,
|
||||
1024,
|
||||
(kernel_size, 1),
|
||||
1,
|
||||
padding=(get_padding(kernel_size, 1), 0),
|
||||
)
|
||||
),
|
||||
]
|
||||
)
|
||||
self.conv_post = norm_f(Conv2d(1024, 1, (3, 1), 1, padding=(1, 0)))
|
||||
|
||||
def forward(self, x):
|
||||
fmap = []
|
||||
|
||||
# 1d to 2d
|
||||
b, c, t = x.shape
|
||||
if t % self.period != 0: # pad first
|
||||
n_pad = self.period - (t % self.period)
|
||||
x = F.pad(x, (0, n_pad), "reflect")
|
||||
t = t + n_pad
|
||||
x = x.view(b, c, t // self.period, self.period)
|
||||
|
||||
for layer in self.convs:
|
||||
x = layer(x)
|
||||
x = F.leaky_relu(x, modules.LRELU_SLOPE)
|
||||
fmap.append(x)
|
||||
x = self.conv_post(x)
|
||||
fmap.append(x)
|
||||
x = torch.flatten(x, 1, -1)
|
||||
|
||||
return x, fmap
|
||||
|
||||
|
||||
class DiscriminatorS(torch.nn.Module):
|
||||
def __init__(self, use_spectral_norm=False):
|
||||
super(DiscriminatorS, self).__init__()
|
||||
norm_f = weight_norm if use_spectral_norm is False else spectral_norm
|
||||
self.convs = nn.ModuleList(
|
||||
[
|
||||
norm_f(Conv1d(1, 16, 15, 1, padding=7)),
|
||||
norm_f(Conv1d(16, 64, 41, 4, groups=4, padding=20)),
|
||||
norm_f(Conv1d(64, 256, 41, 4, groups=16, padding=20)),
|
||||
norm_f(Conv1d(256, 1024, 41, 4, groups=64, padding=20)),
|
||||
norm_f(Conv1d(1024, 1024, 41, 4, groups=256, padding=20)),
|
||||
norm_f(Conv1d(1024, 1024, 5, 1, padding=2)),
|
||||
]
|
||||
)
|
||||
self.conv_post = norm_f(Conv1d(1024, 1, 3, 1, padding=1))
|
||||
|
||||
def forward(self, x):
|
||||
fmap = []
|
||||
|
||||
for layer in self.convs:
|
||||
x = layer(x)
|
||||
x = F.leaky_relu(x, modules.LRELU_SLOPE)
|
||||
fmap.append(x)
|
||||
x = self.conv_post(x)
|
||||
fmap.append(x)
|
||||
x = torch.flatten(x, 1, -1)
|
||||
|
||||
return x, fmap
|
||||
|
||||
|
||||
class MultiPeriodDiscriminator(torch.nn.Module):
|
||||
def __init__(self, use_spectral_norm=False):
|
||||
super(MultiPeriodDiscriminator, self).__init__()
|
||||
periods = [2, 3, 5, 7, 11]
|
||||
|
||||
discs = [DiscriminatorS(use_spectral_norm=use_spectral_norm)]
|
||||
discs = discs + [
|
||||
DiscriminatorP(i, use_spectral_norm=use_spectral_norm) for i in periods
|
||||
]
|
||||
self.discriminators = nn.ModuleList(discs)
|
||||
|
||||
def forward(self, y, y_hat):
|
||||
y_d_rs = []
|
||||
y_d_gs = []
|
||||
fmap_rs = []
|
||||
fmap_gs = []
|
||||
for i, d in enumerate(self.discriminators):
|
||||
y_d_r, fmap_r = d(y)
|
||||
y_d_g, fmap_g = d(y_hat)
|
||||
y_d_rs.append(y_d_r)
|
||||
y_d_gs.append(y_d_g)
|
||||
fmap_rs.append(fmap_r)
|
||||
fmap_gs.append(fmap_g)
|
||||
|
||||
return y_d_rs, y_d_gs, fmap_rs, fmap_gs
|
||||
|
||||
|
||||
class ReferenceEncoder(nn.Module):
|
||||
"""
|
||||
inputs --- [N, Ty/r, n_mels*r] mels
|
||||
outputs --- [N, ref_enc_gru_size]
|
||||
"""
|
||||
|
||||
def __init__(self, spec_channels, gin_channels=0):
|
||||
super().__init__()
|
||||
self.spec_channels = spec_channels
|
||||
ref_enc_filters = [32, 32, 64, 64, 128, 128]
|
||||
K = len(ref_enc_filters)
|
||||
filters = [1] + ref_enc_filters
|
||||
convs = [
|
||||
weight_norm(
|
||||
nn.Conv2d(
|
||||
in_channels=filters[i],
|
||||
out_channels=filters[i + 1],
|
||||
kernel_size=(3, 3),
|
||||
stride=(2, 2),
|
||||
padding=(1, 1),
|
||||
)
|
||||
)
|
||||
for i in range(K)
|
||||
]
|
||||
self.convs = nn.ModuleList(convs)
|
||||
# self.wns = nn.ModuleList([weight_norm(num_features=ref_enc_filters[i]) for i in range(K)]) # noqa: E501
|
||||
|
||||
out_channels = self.calculate_channels(spec_channels, 3, 2, 1, K)
|
||||
self.gru = nn.GRU(
|
||||
input_size=ref_enc_filters[-1] * out_channels,
|
||||
hidden_size=256 // 2,
|
||||
batch_first=True,
|
||||
)
|
||||
self.proj = nn.Linear(128, gin_channels)
|
||||
|
||||
def forward(self, inputs, mask=None):
|
||||
N = inputs.size(0)
|
||||
out = inputs.view(N, 1, -1, self.spec_channels) # [N, 1, Ty, n_freqs]
|
||||
for conv in self.convs:
|
||||
out = conv(out)
|
||||
# out = wn(out)
|
||||
out = F.relu(out) # [N, 128, Ty//2^K, n_mels//2^K]
|
||||
|
||||
out = out.transpose(1, 2) # [N, Ty//2^K, 128, n_mels//2^K]
|
||||
T = out.size(1)
|
||||
N = out.size(0)
|
||||
out = out.contiguous().view(N, T, -1) # [N, Ty//2^K, 128*n_mels//2^K]
|
||||
|
||||
self.gru.flatten_parameters()
|
||||
memory, out = self.gru(out) # out --- [1, N, 128]
|
||||
|
||||
return self.proj(out.squeeze(0))
|
||||
|
||||
def calculate_channels(self, L, kernel_size, stride, pad, n_convs):
|
||||
for i in range(n_convs):
|
||||
L = (L - kernel_size + 2 * pad) // stride + 1
|
||||
return L
|
||||
|
||||
|
||||
class SynthesizerTrn(nn.Module):
|
||||
"""
|
||||
Synthesizer for Training
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
n_vocab,
|
||||
spec_channels,
|
||||
segment_size,
|
||||
inter_channels,
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
resblock,
|
||||
resblock_kernel_sizes,
|
||||
resblock_dilation_sizes,
|
||||
upsample_rates,
|
||||
upsample_initial_channel,
|
||||
upsample_kernel_sizes,
|
||||
n_speakers=256,
|
||||
gin_channels=256,
|
||||
use_sdp=True,
|
||||
n_flow_layer=4,
|
||||
n_layers_trans_flow=6,
|
||||
flow_share_parameter=False,
|
||||
use_transformer_flow=True,
|
||||
**kwargs
|
||||
):
|
||||
super().__init__()
|
||||
self.n_vocab = n_vocab
|
||||
self.spec_channels = spec_channels
|
||||
self.inter_channels = inter_channels
|
||||
self.hidden_channels = hidden_channels
|
||||
self.filter_channels = filter_channels
|
||||
self.n_heads = n_heads
|
||||
self.n_layers = n_layers
|
||||
self.kernel_size = kernel_size
|
||||
self.p_dropout = p_dropout
|
||||
self.resblock = resblock
|
||||
self.resblock_kernel_sizes = resblock_kernel_sizes
|
||||
self.resblock_dilation_sizes = resblock_dilation_sizes
|
||||
self.upsample_rates = upsample_rates
|
||||
self.upsample_initial_channel = upsample_initial_channel
|
||||
self.upsample_kernel_sizes = upsample_kernel_sizes
|
||||
self.segment_size = segment_size
|
||||
self.n_speakers = n_speakers
|
||||
self.gin_channels = gin_channels
|
||||
self.n_layers_trans_flow = n_layers_trans_flow
|
||||
self.use_spk_conditioned_encoder = kwargs.get(
|
||||
"use_spk_conditioned_encoder", True
|
||||
)
|
||||
self.use_sdp = use_sdp
|
||||
self.use_noise_scaled_mas = kwargs.get("use_noise_scaled_mas", False)
|
||||
self.mas_noise_scale_initial = kwargs.get("mas_noise_scale_initial", 0.01)
|
||||
self.noise_scale_delta = kwargs.get("noise_scale_delta", 2e-6)
|
||||
self.current_mas_noise_scale = self.mas_noise_scale_initial
|
||||
if self.use_spk_conditioned_encoder and gin_channels > 0:
|
||||
self.enc_gin_channels = gin_channels
|
||||
self.enc_p = TextEncoder(
|
||||
n_vocab,
|
||||
inter_channels,
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers,
|
||||
kernel_size,
|
||||
p_dropout,
|
||||
gin_channels=self.enc_gin_channels,
|
||||
)
|
||||
self.dec = Generator(
|
||||
inter_channels,
|
||||
resblock,
|
||||
resblock_kernel_sizes,
|
||||
resblock_dilation_sizes,
|
||||
upsample_rates,
|
||||
upsample_initial_channel,
|
||||
upsample_kernel_sizes,
|
||||
gin_channels=gin_channels,
|
||||
)
|
||||
self.enc_q = PosteriorEncoder(
|
||||
spec_channels,
|
||||
inter_channels,
|
||||
hidden_channels,
|
||||
5,
|
||||
1,
|
||||
16,
|
||||
gin_channels=gin_channels,
|
||||
)
|
||||
if use_transformer_flow:
|
||||
self.flow = TransformerCouplingBlock(
|
||||
inter_channels,
|
||||
hidden_channels,
|
||||
filter_channels,
|
||||
n_heads,
|
||||
n_layers_trans_flow,
|
||||
5,
|
||||
p_dropout,
|
||||
n_flow_layer,
|
||||
gin_channels=gin_channels,
|
||||
share_parameter=flow_share_parameter,
|
||||
)
|
||||
else:
|
||||
self.flow = ResidualCouplingBlock(
|
||||
inter_channels,
|
||||
hidden_channels,
|
||||
5,
|
||||
1,
|
||||
n_flow_layer,
|
||||
gin_channels=gin_channels,
|
||||
)
|
||||
self.sdp = StochasticDurationPredictor(
|
||||
hidden_channels, 192, 3, 0.5, 4, gin_channels=gin_channels
|
||||
)
|
||||
self.dp = DurationPredictor(
|
||||
hidden_channels, 256, 3, 0.5, gin_channels=gin_channels
|
||||
)
|
||||
|
||||
if n_speakers > 0:
|
||||
self.emb_g = nn.Embedding(n_speakers, gin_channels)
|
||||
else:
|
||||
self.ref_enc = ReferenceEncoder(spec_channels, gin_channels)
|
||||
|
||||
def forward(self, x, x_lengths, y, y_lengths, sid, tone, language, bert, ja_bert):
|
||||
if self.n_speakers > 0:
|
||||
g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
|
||||
else:
|
||||
g = self.ref_enc(y.transpose(1, 2)).unsqueeze(-1)
|
||||
x, m_p, logs_p, x_mask = self.enc_p(
|
||||
x, x_lengths, tone, language, bert, ja_bert, g=g
|
||||
)
|
||||
z, m_q, logs_q, y_mask = self.enc_q(y, y_lengths, g=g)
|
||||
z_p = self.flow(z, y_mask, g=g)
|
||||
|
||||
with torch.no_grad():
|
||||
# negative cross-entropy
|
||||
s_p_sq_r = torch.exp(-2 * logs_p) # [b, d, t]
|
||||
neg_cent1 = torch.sum(
|
||||
-0.5 * math.log(2 * math.pi) - logs_p, [1], keepdim=True
|
||||
) # [b, 1, t_s]
|
||||
neg_cent2 = torch.matmul(
|
||||
-0.5 * (z_p**2).transpose(1, 2), s_p_sq_r
|
||||
) # [b, t_t, d] x [b, d, t_s] = [b, t_t, t_s]
|
||||
neg_cent3 = torch.matmul(
|
||||
z_p.transpose(1, 2), (m_p * s_p_sq_r)
|
||||
) # [b, t_t, d] x [b, d, t_s] = [b, t_t, t_s]
|
||||
neg_cent4 = torch.sum(
|
||||
-0.5 * (m_p**2) * s_p_sq_r, [1], keepdim=True
|
||||
) # [b, 1, t_s]
|
||||
neg_cent = neg_cent1 + neg_cent2 + neg_cent3 + neg_cent4
|
||||
if self.use_noise_scaled_mas:
|
||||
epsilon = (
|
||||
torch.std(neg_cent)
|
||||
* torch.randn_like(neg_cent)
|
||||
* self.current_mas_noise_scale
|
||||
)
|
||||
neg_cent = neg_cent + epsilon
|
||||
|
||||
attn_mask = torch.unsqueeze(x_mask, 2) * torch.unsqueeze(y_mask, -1)
|
||||
attn = (
|
||||
monotonic_align.maximum_path(neg_cent, attn_mask.squeeze(1))
|
||||
.unsqueeze(1)
|
||||
.detach()
|
||||
)
|
||||
|
||||
w = attn.sum(2)
|
||||
|
||||
l_length_sdp = self.sdp(x, x_mask, w, g=g)
|
||||
l_length_sdp = l_length_sdp / torch.sum(x_mask)
|
||||
|
||||
logw_ = torch.log(w + 1e-6) * x_mask
|
||||
logw = self.dp(x, x_mask, g=g)
|
||||
l_length_dp = torch.sum((logw - logw_) ** 2, [1, 2]) / torch.sum(
|
||||
x_mask
|
||||
) # for averaging
|
||||
|
||||
l_length = l_length_dp + l_length_sdp
|
||||
|
||||
# expand prior
|
||||
m_p = torch.matmul(attn.squeeze(1), m_p.transpose(1, 2)).transpose(1, 2)
|
||||
logs_p = torch.matmul(attn.squeeze(1), logs_p.transpose(1, 2)).transpose(1, 2)
|
||||
|
||||
z_slice, ids_slice = commons.rand_slice_segments(
|
||||
z, y_lengths, self.segment_size
|
||||
)
|
||||
o = self.dec(z_slice, g=g)
|
||||
return (
|
||||
o,
|
||||
l_length,
|
||||
attn,
|
||||
ids_slice,
|
||||
x_mask,
|
||||
y_mask,
|
||||
(z, z_p, m_p, logs_p, m_q, logs_q),
|
||||
(x, logw, logw_),
|
||||
)
|
||||
|
||||
def infer(
|
||||
self,
|
||||
x,
|
||||
x_lengths,
|
||||
sid,
|
||||
tone,
|
||||
language,
|
||||
bert,
|
||||
ja_bert,
|
||||
noise_scale=0.667,
|
||||
length_scale=1,
|
||||
noise_scale_w=0.8,
|
||||
max_len=None,
|
||||
sdp_ratio=0,
|
||||
y=None,
|
||||
):
|
||||
# x, m_p, logs_p, x_mask = self.enc_p(x, x_lengths, tone, language, bert)
|
||||
# g = self.gst(y)
|
||||
if self.n_speakers > 0:
|
||||
g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
|
||||
else:
|
||||
g = self.ref_enc(y.transpose(1, 2)).unsqueeze(-1)
|
||||
x, m_p, logs_p, x_mask = self.enc_p(
|
||||
x, x_lengths, tone, language, bert, ja_bert, g=g
|
||||
)
|
||||
logw = self.sdp(x, x_mask, g=g, reverse=True, noise_scale=noise_scale_w) * (
|
||||
sdp_ratio
|
||||
) + self.dp(x, x_mask, g=g) * (1 - sdp_ratio)
|
||||
w = torch.exp(logw) * x_mask * length_scale
|
||||
w_ceil = torch.ceil(w)
|
||||
y_lengths = torch.clamp_min(torch.sum(w_ceil, [1, 2]), 1).long()
|
||||
y_mask = torch.unsqueeze(commons.sequence_mask(y_lengths, None), 1).to(
|
||||
x_mask.dtype
|
||||
)
|
||||
attn_mask = torch.unsqueeze(x_mask, 2) * torch.unsqueeze(y_mask, -1)
|
||||
attn = commons.generate_path(w_ceil, attn_mask)
|
||||
|
||||
m_p = torch.matmul(attn.squeeze(1), m_p.transpose(1, 2)).transpose(
|
||||
1, 2
|
||||
) # [b, t', t], [b, t, d] -> [b, d, t']
|
||||
logs_p = torch.matmul(attn.squeeze(1), logs_p.transpose(1, 2)).transpose(
|
||||
1, 2
|
||||
) # [b, t', t], [b, t, d] -> [b, d, t']
|
||||
|
||||
z_p = m_p + torch.randn_like(m_p) * torch.exp(logs_p) * noise_scale
|
||||
z = self.flow(z_p, y_mask, g=g, reverse=True)
|
||||
o = self.dec((z * y_mask)[:, :, :max_len], g=g)
|
||||
return o, attn, y_mask, (z, z_p, m_p, logs_p)
|
||||
38
oldVersion/V111/text/__init__.py
Normal file
38
oldVersion/V111/text/__init__.py
Normal file
@@ -0,0 +1,38 @@
|
||||
from .symbols import *
|
||||
|
||||
_symbol_to_id = {s: i for i, s in enumerate(symbols)}
|
||||
|
||||
|
||||
def cleaned_text_to_sequence(cleaned_text, tones, language):
|
||||
"""Converts a string of text to a sequence of IDs corresponding to the symbols in the text.
|
||||
Args:
|
||||
text: string to convert to a sequence
|
||||
Returns:
|
||||
List of integers corresponding to the symbols in the text
|
||||
"""
|
||||
phones = [_symbol_to_id[symbol] for symbol in cleaned_text]
|
||||
tone_start = language_tone_start_map[language]
|
||||
tones = [i + tone_start for i in tones]
|
||||
lang_id = language_id_map[language]
|
||||
lang_ids = [lang_id for i in phones]
|
||||
return phones, tones, lang_ids
|
||||
|
||||
|
||||
def get_bert(norm_text, word2ph, language, device):
|
||||
from .chinese_bert import get_bert_feature as zh_bert
|
||||
from .english_bert_mock import get_bert_feature as en_bert
|
||||
from .japanese_bert import get_bert_feature as jp_bert
|
||||
|
||||
lang_bert_func_map = {"ZH": zh_bert, "EN": en_bert, "JP": jp_bert}
|
||||
bert = lang_bert_func_map[language](norm_text, word2ph, device)
|
||||
return bert
|
||||
|
||||
|
||||
def get_bert_fix(norm_text, word2ph, language, device):
|
||||
from .chinese_bert import get_bert_feature as zh_bert
|
||||
from .english_bert_mock import get_bert_feature as en_bert
|
||||
from .fix.japanese_bert import get_bert_feature as jp_bert
|
||||
|
||||
lang_bert_func_map = {"ZH": zh_bert, "EN": en_bert, "JP": jp_bert}
|
||||
bert = lang_bert_func_map[language](norm_text, word2ph, device)
|
||||
return bert
|
||||
198
oldVersion/V111/text/chinese.py
Normal file
198
oldVersion/V111/text/chinese.py
Normal file
@@ -0,0 +1,198 @@
|
||||
import os
|
||||
import re
|
||||
|
||||
import cn2an
|
||||
from pypinyin import lazy_pinyin, Style
|
||||
|
||||
from .symbols import punctuation
|
||||
from .tone_sandhi import ToneSandhi
|
||||
|
||||
current_file_path = os.path.dirname(__file__)
|
||||
pinyin_to_symbol_map = {
|
||||
line.split("\t")[0]: line.strip().split("\t")[1]
|
||||
for line in open(os.path.join(current_file_path, "opencpop-strict.txt")).readlines()
|
||||
}
|
||||
|
||||
import jieba.posseg as psg
|
||||
|
||||
|
||||
rep_map = {
|
||||
"：": ",",
|
||||
"；": ",",
|
||||
"，": ",",
|
||||
"。": ".",
|
||||
"！": "!",
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"·": ",",
|
||||
"、": ",",
|
||||
"...": "…",
|
||||
"$": ".",
|
||||
"“": "'",
|
||||
"”": "'",
|
||||
"‘": "'",
|
||||
"’": "'",
|
||||
"（": "'",
|
||||
"）": "'",
|
||||
"(": "'",
|
||||
")": "'",
|
||||
"《": "'",
|
||||
"》": "'",
|
||||
"【": "'",
|
||||
"】": "'",
|
||||
"[": "'",
|
||||
"]": "'",
|
||||
"—": "-",
|
||||
"～": "-",
|
||||
"~": "-",
|
||||
"「": "'",
|
||||
"」": "'",
|
||||
}
|
||||
|
||||
tone_modifier = ToneSandhi()
|
||||
|
||||
|
||||
def replace_punctuation(text):
|
||||
text = text.replace("嗯", "恩").replace("呣", "母")
|
||||
pattern = re.compile("|".join(re.escape(p) for p in rep_map.keys()))
|
||||
|
||||
replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
|
||||
|
||||
replaced_text = re.sub(
|
||||
r"[^\u4e00-\u9fa5" + "".join(punctuation) + r"]+", "", replaced_text
|
||||
)
|
||||
|
||||
return replaced_text
|
||||
|
||||
|
||||
def g2p(text):
|
||||
pattern = r"(?<=[{0}])\s*".format("".join(punctuation))
|
||||
sentences = [i for i in re.split(pattern, text) if i.strip() != ""]
|
||||
phones, tones, word2ph = _g2p(sentences)
|
||||
assert sum(word2ph) == len(phones)
|
||||
assert len(word2ph) == len(text) # Sometimes it will crash,you can add a try-catch.
|
||||
phones = ["_"] + phones + ["_"]
|
||||
tones = [0] + tones + [0]
|
||||
word2ph = [1] + word2ph + [1]
|
||||
return phones, tones, word2ph
|
||||
|
||||
|
||||
def _get_initials_finals(word):
|
||||
initials = []
|
||||
finals = []
|
||||
orig_initials = lazy_pinyin(word, neutral_tone_with_five=True, style=Style.INITIALS)
|
||||
orig_finals = lazy_pinyin(
|
||||
word, neutral_tone_with_five=True, style=Style.FINALS_TONE3
|
||||
)
|
||||
for c, v in zip(orig_initials, orig_finals):
|
||||
initials.append(c)
|
||||
finals.append(v)
|
||||
return initials, finals
|
||||
|
||||
|
||||
def _g2p(segments):
|
||||
phones_list = []
|
||||
tones_list = []
|
||||
word2ph = []
|
||||
for seg in segments:
|
||||
# Replace all English words in the sentence
|
||||
seg = re.sub("[a-zA-Z]+", "", seg)
|
||||
seg_cut = psg.lcut(seg)
|
||||
initials = []
|
||||
finals = []
|
||||
seg_cut = tone_modifier.pre_merge_for_modify(seg_cut)
|
||||
for word, pos in seg_cut:
|
||||
if pos == "eng":
|
||||
continue
|
||||
sub_initials, sub_finals = _get_initials_finals(word)
|
||||
sub_finals = tone_modifier.modified_tone(word, pos, sub_finals)
|
||||
initials.append(sub_initials)
|
||||
finals.append(sub_finals)
|
||||
|
||||
# assert len(sub_initials) == len(sub_finals) == len(word)
|
||||
initials = sum(initials, [])
|
||||
finals = sum(finals, [])
|
||||
#
|
||||
for c, v in zip(initials, finals):
|
||||
raw_pinyin = c + v
|
||||
# NOTE: post process for pypinyin outputs
|
||||
# we discriminate i, ii and iii
|
||||
if c == v:
|
||||
assert c in punctuation
|
||||
phone = [c]
|
||||
tone = "0"
|
||||
word2ph.append(1)
|
||||
else:
|
||||
v_without_tone = v[:-1]
|
||||
tone = v[-1]
|
||||
|
||||
pinyin = c + v_without_tone
|
||||
assert tone in "12345"
|
||||
|
||||
if c:
|
||||
# 多音节
|
||||
v_rep_map = {
|
||||
"uei": "ui",
|
||||
"iou": "iu",
|
||||
"uen": "un",
|
||||
}
|
||||
if v_without_tone in v_rep_map.keys():
|
||||
pinyin = c + v_rep_map[v_without_tone]
|
||||
else:
|
||||
# 单音节
|
||||
pinyin_rep_map = {
|
||||
"ing": "ying",
|
||||
"i": "yi",
|
||||
"in": "yin",
|
||||
"u": "wu",
|
||||
}
|
||||
if pinyin in pinyin_rep_map.keys():
|
||||
pinyin = pinyin_rep_map[pinyin]
|
||||
else:
|
||||
single_rep_map = {
|
||||
"v": "yu",
|
||||
"e": "e",
|
||||
"i": "y",
|
||||
"u": "w",
|
||||
}
|
||||
if pinyin[0] in single_rep_map.keys():
|
||||
pinyin = single_rep_map[pinyin[0]] + pinyin[1:]
|
||||
|
||||
assert pinyin in pinyin_to_symbol_map.keys(), (pinyin, seg, raw_pinyin)
|
||||
phone = pinyin_to_symbol_map[pinyin].split(" ")
|
||||
word2ph.append(len(phone))
|
||||
|
||||
phones_list += phone
|
||||
tones_list += [int(tone)] * len(phone)
|
||||
return phones_list, tones_list, word2ph
|
||||
|
||||
|
||||
def text_normalize(text):
|
||||
numbers = re.findall(r"\d+(?:\.?\d+)?", text)
|
||||
for number in numbers:
|
||||
text = text.replace(number, cn2an.an2cn(number), 1)
|
||||
text = replace_punctuation(text)
|
||||
return text
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph):
|
||||
from text import chinese_bert
|
||||
|
||||
return chinese_bert.get_bert_feature(text, word2ph)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from text.chinese_bert import get_bert_feature
|
||||
|
||||
text = "啊！但是《原神》是由,米哈\游自主， [研发]的一款全.新开放世界.冒险游戏"
|
||||
text = text_normalize(text)
|
||||
print(text)
|
||||
phones, tones, word2ph = g2p(text)
|
||||
bert = get_bert_feature(text, word2ph)
|
||||
|
||||
print(phones, tones, word2ph, bert.shape)
|
||||
|
||||
|
||||
# # 示例用法
|
||||
# text = "这是一个示例文本：,你好！这是一个测试...."
|
||||
# print(g2p_paddle(text)) # 输出: 这是一个示例文本你好这是一个测试
|
||||
100
oldVersion/V111/text/chinese_bert.py
Normal file
100
oldVersion/V111/text/chinese_bert.py
Normal file
@@ -0,0 +1,100 @@
|
||||
import torch
|
||||
import sys
|
||||
from transformers import AutoTokenizer, AutoModelForMaskedLM
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/chinese-roberta-wwm-ext-large")
|
||||
|
||||
models = dict()
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph, device=None):
|
||||
if (
|
||||
sys.platform == "darwin"
|
||||
and torch.backends.mps.is_available()
|
||||
and device == "cpu"
|
||||
):
|
||||
device = "mps"
|
||||
if not device:
|
||||
device = "cuda"
|
||||
if device not in models.keys():
|
||||
models[device] = AutoModelForMaskedLM.from_pretrained(
|
||||
"./bert/chinese-roberta-wwm-ext-large"
|
||||
).to(device)
|
||||
with torch.no_grad():
|
||||
inputs = tokenizer(text, return_tensors="pt")
|
||||
for i in inputs:
|
||||
inputs[i] = inputs[i].to(device)
|
||||
res = models[device](**inputs, output_hidden_states=True)
|
||||
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
|
||||
|
||||
assert len(word2ph) == len(text) + 2
|
||||
word2phone = word2ph
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
repeat_feature = res[i].repeat(word2phone[i], 1)
|
||||
phone_level_feature.append(repeat_feature)
|
||||
|
||||
phone_level_feature = torch.cat(phone_level_feature, dim=0)
|
||||
|
||||
return phone_level_feature.T
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import torch
|
||||
|
||||
word_level_feature = torch.rand(38, 1024) # 12个词,每个词1024维特征
|
||||
word2phone = [
|
||||
1,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
]
|
||||
|
||||
# 计算总帧数
|
||||
total_frames = sum(word2phone)
|
||||
print(word_level_feature.shape)
|
||||
print(word2phone)
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
print(word_level_feature[i].shape)
|
||||
|
||||
# 对每个词重复word2phone[i]次
|
||||
repeat_feature = word_level_feature[i].repeat(word2phone[i], 1)
|
||||
phone_level_feature.append(repeat_feature)
|
||||
|
||||
phone_level_feature = torch.cat(phone_level_feature, dim=0)
|
||||
print(phone_level_feature.shape) # torch.Size([36, 1024])
|
||||
38
oldVersion/V111/text/cleaner.py
Normal file
38
oldVersion/V111/text/cleaner.py
Normal file
@@ -0,0 +1,38 @@
|
||||
from . import chinese, japanese, cleaned_text_to_sequence
|
||||
from .fix import japanese as japanese_fix
|
||||
|
||||
|
||||
language_module_map = {"ZH": chinese, "JP": japanese}
|
||||
language_module_map_fix = {"ZH": chinese, "JP": japanese_fix}
|
||||
|
||||
|
||||
def clean_text(text, language):
|
||||
language_module = language_module_map[language]
|
||||
norm_text = language_module.text_normalize(text)
|
||||
phones, tones, word2ph = language_module.g2p(norm_text)
|
||||
return norm_text, phones, tones, word2ph
|
||||
|
||||
|
||||
def clean_text_fix(text, language):
|
||||
"""使用dev分支修复"""
|
||||
language_module = language_module_map_fix[language]
|
||||
norm_text = language_module.text_normalize(text)
|
||||
phones, tones, word2ph = language_module.g2p(norm_text)
|
||||
return norm_text, phones, tones, word2ph
|
||||
|
||||
|
||||
def clean_text_bert(text, language):
|
||||
language_module = language_module_map[language]
|
||||
norm_text = language_module.text_normalize(text)
|
||||
phones, tones, word2ph = language_module.g2p(norm_text)
|
||||
bert = language_module.get_bert_feature(norm_text, word2ph)
|
||||
return phones, tones, bert
|
||||
|
||||
|
||||
def text_to_sequence(text, language):
|
||||
norm_text, phones, tones, word2ph = clean_text(text, language)
|
||||
return cleaned_text_to_sequence(phones, tones, language)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pass
|
||||
214
oldVersion/V111/text/english.py
Normal file
214
oldVersion/V111/text/english.py
Normal file
@@ -0,0 +1,214 @@
|
||||
import pickle
|
||||
import os
|
||||
import re
|
||||
from g2p_en import G2p
|
||||
|
||||
from . import symbols
|
||||
|
||||
current_file_path = os.path.dirname(__file__)
|
||||
CMU_DICT_PATH = os.path.join(current_file_path, "cmudict.rep")
|
||||
CACHE_PATH = os.path.join(current_file_path, "cmudict_cache.pickle")
|
||||
_g2p = G2p()
|
||||
|
||||
arpa = {
|
||||
"AH0",
|
||||
"S",
|
||||
"AH1",
|
||||
"EY2",
|
||||
"AE2",
|
||||
"EH0",
|
||||
"OW2",
|
||||
"UH0",
|
||||
"NG",
|
||||
"B",
|
||||
"G",
|
||||
"AY0",
|
||||
"M",
|
||||
"AA0",
|
||||
"F",
|
||||
"AO0",
|
||||
"ER2",
|
||||
"UH1",
|
||||
"IY1",
|
||||
"AH2",
|
||||
"DH",
|
||||
"IY0",
|
||||
"EY1",
|
||||
"IH0",
|
||||
"K",
|
||||
"N",
|
||||
"W",
|
||||
"IY2",
|
||||
"T",
|
||||
"AA1",
|
||||
"ER1",
|
||||
"EH2",
|
||||
"OY0",
|
||||
"UH2",
|
||||
"UW1",
|
||||
"Z",
|
||||
"AW2",
|
||||
"AW1",
|
||||
"V",
|
||||
"UW2",
|
||||
"AA2",
|
||||
"ER",
|
||||
"AW0",
|
||||
"UW0",
|
||||
"R",
|
||||
"OW1",
|
||||
"EH1",
|
||||
"ZH",
|
||||
"AE0",
|
||||
"IH2",
|
||||
"IH",
|
||||
"Y",
|
||||
"JH",
|
||||
"P",
|
||||
"AY1",
|
||||
"EY0",
|
||||
"OY2",
|
||||
"TH",
|
||||
"HH",
|
||||
"D",
|
||||
"ER0",
|
||||
"CH",
|
||||
"AO1",
|
||||
"AE1",
|
||||
"AO2",
|
||||
"OY1",
|
||||
"AY2",
|
||||
"IH1",
|
||||
"OW0",
|
||||
"L",
|
||||
"SH",
|
||||
}
|
||||
|
||||
|
||||
def post_replace_ph(ph):
|
||||
rep_map = {
|
||||
"：": ",",
|
||||
"；": ",",
|
||||
"，": ",",
|
||||
"。": ".",
|
||||
"！": "!",
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"·": ",",
|
||||
"、": ",",
|
||||
"...": "…",
|
||||
"v": "V",
|
||||
}
|
||||
if ph in rep_map.keys():
|
||||
ph = rep_map[ph]
|
||||
if ph in symbols:
|
||||
return ph
|
||||
if ph not in symbols:
|
||||
ph = "UNK"
|
||||
return ph
|
||||
|
||||
|
||||
def read_dict():
|
||||
g2p_dict = {}
|
||||
start_line = 49
|
||||
with open(CMU_DICT_PATH) as f:
|
||||
line = f.readline()
|
||||
line_index = 1
|
||||
while line:
|
||||
if line_index >= start_line:
|
||||
line = line.strip()
|
||||
word_split = line.split(" ")
|
||||
word = word_split[0]
|
||||
|
||||
syllable_split = word_split[1].split(" - ")
|
||||
g2p_dict[word] = []
|
||||
for syllable in syllable_split:
|
||||
phone_split = syllable.split(" ")
|
||||
g2p_dict[word].append(phone_split)
|
||||
|
||||
line_index = line_index + 1
|
||||
line = f.readline()
|
||||
|
||||
return g2p_dict
|
||||
|
||||
|
||||
def cache_dict(g2p_dict, file_path):
|
||||
with open(file_path, "wb") as pickle_file:
|
||||
pickle.dump(g2p_dict, pickle_file)
|
||||
|
||||
|
||||
def get_dict():
|
||||
if os.path.exists(CACHE_PATH):
|
||||
with open(CACHE_PATH, "rb") as pickle_file:
|
||||
g2p_dict = pickle.load(pickle_file)
|
||||
else:
|
||||
g2p_dict = read_dict()
|
||||
cache_dict(g2p_dict, CACHE_PATH)
|
||||
|
||||
return g2p_dict
|
||||
|
||||
|
||||
eng_dict = get_dict()
|
||||
|
||||
|
||||
def refine_ph(phn):
|
||||
tone = 0
|
||||
if re.search(r"\d$", phn):
|
||||
tone = int(phn[-1]) + 1
|
||||
phn = phn[:-1]
|
||||
return phn.lower(), tone
|
||||
|
||||
|
||||
def refine_syllables(syllables):
|
||||
tones = []
|
||||
phonemes = []
|
||||
for phn_list in syllables:
|
||||
for i in range(len(phn_list)):
|
||||
phn = phn_list[i]
|
||||
phn, tone = refine_ph(phn)
|
||||
phonemes.append(phn)
|
||||
tones.append(tone)
|
||||
return phonemes, tones
|
||||
|
||||
|
||||
def text_normalize(text):
|
||||
# todo: eng text normalize
|
||||
return text
|
||||
|
||||
|
||||
def g2p(text):
|
||||
phones = []
|
||||
tones = []
|
||||
words = re.split(r"([,;.\-\?\!\s+])", text)
|
||||
for w in words:
|
||||
if w.upper() in eng_dict:
|
||||
phns, tns = refine_syllables(eng_dict[w.upper()])
|
||||
phones += phns
|
||||
tones += tns
|
||||
else:
|
||||
phone_list = list(filter(lambda p: p != " ", _g2p(w)))
|
||||
for ph in phone_list:
|
||||
if ph in arpa:
|
||||
ph, tn = refine_ph(ph)
|
||||
phones.append(ph)
|
||||
tones.append(tn)
|
||||
else:
|
||||
phones.append(ph)
|
||||
tones.append(0)
|
||||
# todo: implement word2ph
|
||||
word2ph = [1 for i in phones]
|
||||
|
||||
phones = [post_replace_ph(i) for i in phones]
|
||||
return phones, tones, word2ph
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# print(get_dict())
|
||||
# print(eng_word_to_phoneme("hello"))
|
||||
print(g2p("In this paper, we propose 1 DSPGAN, a GAN-based universal vocoder."))
|
||||
# all_phones = set()
|
||||
# for k, syllables in eng_dict.items():
|
||||
# for group in syllables:
|
||||
# for ph in group:
|
||||
# all_phones.add(ph)
|
||||
# print(all_phones)
|
||||
5
oldVersion/V111/text/english_bert_mock.py
Normal file
5
oldVersion/V111/text/english_bert_mock.py
Normal file
@@ -0,0 +1,5 @@
|
||||
import torch
|
||||
|
||||
|
||||
def get_bert_feature(norm_text, word2ph):
|
||||
return torch.zeros(1024, sum(word2ph))
|
||||
0
oldVersion/V111/text/fix/__init__.py
Normal file
0
oldVersion/V111/text/fix/__init__.py
Normal file
347
oldVersion/V111/text/fix/japanese.py
Normal file
347
oldVersion/V111/text/fix/japanese.py
Normal file
@@ -0,0 +1,347 @@
|
||||
# Convert Japanese text to phonemes which is
|
||||
# compatible with Julius https://github.com/julius-speech/segmentation-kit
|
||||
import re
|
||||
import unicodedata
|
||||
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
from .. import punctuation, symbols
|
||||
|
||||
from num2words import num2words
|
||||
|
||||
import pyopenjtalk
|
||||
import jaconv
|
||||
|
||||
|
||||
def kata2phoneme(text: str) -> str:
|
||||
"""Convert katakana text to phonemes."""
|
||||
text = text.strip()
|
||||
if text == "ー":
|
||||
return ["ー"]
|
||||
elif text.startswith("ー"):
|
||||
return ["ー"] + kata2phoneme(text[1:])
|
||||
res = []
|
||||
prev = None
|
||||
while text:
|
||||
if re.match(_MARKS, text):
|
||||
res.append(text)
|
||||
text = text[1:]
|
||||
continue
|
||||
if text.startswith("ー"):
|
||||
if prev:
|
||||
res.append(prev[-1])
|
||||
text = text[1:]
|
||||
continue
|
||||
res += pyopenjtalk.g2p(text).lower().replace("cl", "q").split(" ")
|
||||
break
|
||||
# res = _COLON_RX.sub(":", res)
|
||||
return res
|
||||
|
||||
|
||||
def hira2kata(text: str) -> str:
|
||||
return jaconv.hira2kata(text)
|
||||
|
||||
|
||||
_SYMBOL_TOKENS = set(list("・、。？！"))
|
||||
_NO_YOMI_TOKENS = set(list("「」『』―（）［］[]"))
|
||||
_MARKS = re.compile(
|
||||
r"[^A-Za-z\d\u3005\u3040-\u30ff\u4e00-\u9fff\uff11-\uff19\uff21-\uff3a\uff41-\uff5a\uff66-\uff9d]"
|
||||
)
|
||||
|
||||
|
||||
def text2kata(text: str) -> str:
|
||||
parsed = pyopenjtalk.run_frontend(text)
|
||||
|
||||
res = []
|
||||
for parts in parsed:
|
||||
word, yomi = replace_punctuation(parts["orig"]), parts["pron"].replace("’", "")
|
||||
if yomi:
|
||||
if re.match(_MARKS, yomi):
|
||||
if len(word) > 1:
|
||||
word = [replace_punctuation(i) for i in list(word)]
|
||||
yomi = word
|
||||
res += yomi
|
||||
sep += word
|
||||
continue
|
||||
elif word not in rep_map.keys() and word not in rep_map.values():
|
||||
word = ","
|
||||
yomi = word
|
||||
res.append(yomi)
|
||||
else:
|
||||
if word in _SYMBOL_TOKENS:
|
||||
res.append(word)
|
||||
elif word in ("っ", "ッ"):
|
||||
res.append("ッ")
|
||||
elif word in _NO_YOMI_TOKENS:
|
||||
pass
|
||||
else:
|
||||
res.append(word)
|
||||
return hira2kata("".join(res))
|
||||
|
||||
|
||||
def text2sep_kata(text: str) -> (list, list):
|
||||
parsed = pyopenjtalk.run_frontend(text)
|
||||
|
||||
res = []
|
||||
sep = []
|
||||
for parts in parsed:
|
||||
word, yomi = replace_punctuation(parts["orig"]), parts["pron"].replace("’", "")
|
||||
if yomi:
|
||||
if re.match(_MARKS, yomi):
|
||||
if len(word) > 1:
|
||||
word = [replace_punctuation(i) for i in list(word)]
|
||||
yomi = word
|
||||
res += yomi
|
||||
sep += word
|
||||
continue
|
||||
elif word not in rep_map.keys() and word not in rep_map.values():
|
||||
word = ","
|
||||
yomi = word
|
||||
res.append(yomi)
|
||||
else:
|
||||
if word in _SYMBOL_TOKENS:
|
||||
res.append(word)
|
||||
elif word in ("っ", "ッ"):
|
||||
res.append("ッ")
|
||||
elif word in _NO_YOMI_TOKENS:
|
||||
pass
|
||||
else:
|
||||
res.append(word)
|
||||
sep.append(word)
|
||||
return sep, [hira2kata(i) for i in res]
|
||||
|
||||
|
||||
_ALPHASYMBOL_YOMI = {
|
||||
"#": "シャープ",
|
||||
"%": "パーセント",
|
||||
"&": "アンド",
|
||||
"+": "プラス",
|
||||
"-": "マイナス",
|
||||
":": "コロン",
|
||||
";": "セミコロン",
|
||||
"<": "小なり",
|
||||
"=": "イコール",
|
||||
">": "大なり",
|
||||
"@": "アット",
|
||||
"a": "エー",
|
||||
"b": "ビー",
|
||||
"c": "シー",
|
||||
"d": "ディー",
|
||||
"e": "イー",
|
||||
"f": "エフ",
|
||||
"g": "ジー",
|
||||
"h": "エイチ",
|
||||
"i": "アイ",
|
||||
"j": "ジェー",
|
||||
"k": "ケー",
|
||||
"l": "エル",
|
||||
"m": "エム",
|
||||
"n": "エヌ",
|
||||
"o": "オー",
|
||||
"p": "ピー",
|
||||
"q": "キュー",
|
||||
"r": "アール",
|
||||
"s": "エス",
|
||||
"t": "ティー",
|
||||
"u": "ユー",
|
||||
"v": "ブイ",
|
||||
"w": "ダブリュー",
|
||||
"x": "エックス",
|
||||
"y": "ワイ",
|
||||
"z": "ゼット",
|
||||
"α": "アルファ",
|
||||
"β": "ベータ",
|
||||
"γ": "ガンマ",
|
||||
"δ": "デルタ",
|
||||
"ε": "イプシロン",
|
||||
"ζ": "ゼータ",
|
||||
"η": "イータ",
|
||||
"θ": "シータ",
|
||||
"ι": "イオタ",
|
||||
"κ": "カッパ",
|
||||
"λ": "ラムダ",
|
||||
"μ": "ミュー",
|
||||
"ν": "ニュー",
|
||||
"ξ": "クサイ",
|
||||
"ο": "オミクロン",
|
||||
"π": "パイ",
|
||||
"ρ": "ロー",
|
||||
"σ": "シグマ",
|
||||
"τ": "タウ",
|
||||
"υ": "ウプシロン",
|
||||
"φ": "ファイ",
|
||||
"χ": "カイ",
|
||||
"ψ": "プサイ",
|
||||
"ω": "オメガ",
|
||||
}
|
||||
|
||||
|
||||
_NUMBER_WITH_SEPARATOR_RX = re.compile("[0-9]{1,3}(,[0-9]{3})+")
|
||||
_CURRENCY_MAP = {"$": "ドル", "¥": "円", "£": "ポンド", "€": "ユーロ"}
|
||||
_CURRENCY_RX = re.compile(r"([$¥£€])([0-9.]*[0-9])")
|
||||
_NUMBER_RX = re.compile(r"[0-9]+(\.[0-9]+)?")
|
||||
|
||||
|
||||
def japanese_convert_numbers_to_words(text: str) -> str:
|
||||
res = _NUMBER_WITH_SEPARATOR_RX.sub(lambda m: m[0].replace(",", ""), text)
|
||||
res = _CURRENCY_RX.sub(lambda m: m[2] + _CURRENCY_MAP.get(m[1], m[1]), res)
|
||||
res = _NUMBER_RX.sub(lambda m: num2words(m[0], lang="ja"), res)
|
||||
return res
|
||||
|
||||
|
||||
def japanese_convert_alpha_symbols_to_words(text: str) -> str:
|
||||
return "".join([_ALPHASYMBOL_YOMI.get(ch, ch) for ch in text.lower()])
|
||||
|
||||
|
||||
def japanese_text_to_phonemes(text: str) -> str:
|
||||
"""Convert Japanese text to phonemes."""
|
||||
res = unicodedata.normalize("NFKC", text)
|
||||
res = japanese_convert_numbers_to_words(res)
|
||||
# res = japanese_convert_alpha_symbols_to_words(res)
|
||||
res = text2kata(res)
|
||||
res = kata2phoneme(res)
|
||||
return res
|
||||
|
||||
|
||||
def is_japanese_character(char):
|
||||
# 定义日语文字系统的 Unicode 范围
|
||||
japanese_ranges = [
|
||||
(0x3040, 0x309F), # 平假名
|
||||
(0x30A0, 0x30FF), # 片假名
|
||||
(0x4E00, 0x9FFF), # 汉字 (CJK Unified Ideographs)
|
||||
(0x3400, 0x4DBF), # 汉字扩展 A
|
||||
(0x20000, 0x2A6DF), # 汉字扩展 B
|
||||
# 可以根据需要添加其他汉字扩展范围
|
||||
]
|
||||
|
||||
# 将字符的 Unicode 编码转换为整数
|
||||
char_code = ord(char)
|
||||
|
||||
# 检查字符是否在任何一个日语范围内
|
||||
for start, end in japanese_ranges:
|
||||
if start <= char_code <= end:
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
rep_map = {
|
||||
"：": ",",
|
||||
"；": ",",
|
||||
"，": ",",
|
||||
"。": ".",
|
||||
"！": "!",
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"．": ".",
|
||||
"...": "…",
|
||||
"···": "…",
|
||||
"・・・": "…",
|
||||
"·": ",",
|
||||
"・": ",",
|
||||
"、": ",",
|
||||
"$": ".",
|
||||
"“": "'",
|
||||
"”": "'",
|
||||
"‘": "'",
|
||||
"’": "'",
|
||||
"（": "'",
|
||||
"）": "'",
|
||||
"(": "'",
|
||||
")": "'",
|
||||
"《": "'",
|
||||
"》": "'",
|
||||
"【": "'",
|
||||
"】": "'",
|
||||
"[": "'",
|
||||
"]": "'",
|
||||
"—": "-",
|
||||
"−": "-",
|
||||
"～": "-",
|
||||
"~": "-",
|
||||
"「": "'",
|
||||
"」": "'",
|
||||
}
|
||||
|
||||
|
||||
def replace_punctuation(text):
|
||||
pattern = re.compile("|".join(re.escape(p) for p in rep_map.keys()))
|
||||
|
||||
replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
|
||||
|
||||
replaced_text = re.sub(
|
||||
r"[^\u3040-\u309F\u30A0-\u30FF\u4E00-\u9FFF\u3400-\u4DBF"
|
||||
+ "".join(punctuation)
|
||||
+ r"]+",
|
||||
"",
|
||||
replaced_text,
|
||||
)
|
||||
|
||||
return replaced_text
|
||||
|
||||
|
||||
def text_normalize(text):
|
||||
res = unicodedata.normalize("NFKC", text)
|
||||
res = japanese_convert_numbers_to_words(res)
|
||||
# res = "".join([i for i in res if is_japanese_character(i)])
|
||||
res = replace_punctuation(res)
|
||||
return res
|
||||
|
||||
|
||||
def distribute_phone(n_phone, n_word):
|
||||
phones_per_word = [0] * n_word
|
||||
for task in range(n_phone):
|
||||
min_tasks = min(phones_per_word)
|
||||
min_index = phones_per_word.index(min_tasks)
|
||||
phones_per_word[min_index] += 1
|
||||
return phones_per_word
|
||||
|
||||
|
||||
def handle_long(sep_phonemes):
|
||||
for i in range(len(sep_phonemes)):
|
||||
if sep_phonemes[i][0] == "ー":
|
||||
sep_phonemes[i][0] = sep_phonemes[i - 1][-1]
|
||||
if "ー" in sep_phonemes[i]:
|
||||
for j in range(len(sep_phonemes[i])):
|
||||
if sep_phonemes[i][j] == "ー":
|
||||
sep_phonemes[i][j] = sep_phonemes[i][j - 1][-1]
|
||||
return sep_phonemes
|
||||
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/bert-base-japanese-v3")
|
||||
|
||||
|
||||
def g2p(norm_text):
|
||||
sep_text, sep_kata = text2sep_kata(norm_text)
|
||||
sep_tokenized = [tokenizer.tokenize(i) for i in sep_text]
|
||||
sep_phonemes = handle_long([kata2phoneme(i) for i in sep_kata])
|
||||
# 异常处理，MeCab不认识的词的话会一路传到这里来，然后炸掉。目前来看只有那些超级稀有的生僻词会出现这种情况
|
||||
for i in sep_phonemes:
|
||||
for j in i:
|
||||
assert j in symbols, (sep_text, sep_kata, sep_phonemes)
|
||||
|
||||
word2ph = []
|
||||
for token, phoneme in zip(sep_tokenized, sep_phonemes):
|
||||
phone_len = len(phoneme)
|
||||
word_len = len(token)
|
||||
|
||||
aaa = distribute_phone(phone_len, word_len)
|
||||
word2ph += aaa
|
||||
phones = ["_"] + [j for i in sep_phonemes for j in i] + ["_"]
|
||||
tones = [0 for i in phones]
|
||||
word2ph = [1] + word2ph + [1]
|
||||
return phones, tones, word2ph
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/bert-base-japanese-v3")
|
||||
text = "hello,こんにちは、世界ー！……"
|
||||
from text.japanese_bert import get_bert_feature
|
||||
|
||||
text = text_normalize(text)
|
||||
print(text)
|
||||
|
||||
phones, tones, word2ph = g2p(text)
|
||||
bert = get_bert_feature(text, word2ph)
|
||||
|
||||
print(phones, tones, word2ph, bert.shape)
|
||||
56
oldVersion/V111/text/fix/japanese_bert.py
Normal file
56
oldVersion/V111/text/fix/japanese_bert.py
Normal file
@@ -0,0 +1,56 @@
|
||||
import torch
|
||||
from transformers import AutoTokenizer, AutoModelForMaskedLM
|
||||
import sys
|
||||
from .japanese import text2sep_kata
|
||||
from config import config
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/bert-base-japanese-v3")
|
||||
|
||||
models = dict()
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph, device=config.bert_gen_config.device):
|
||||
sep_text, _ = text2sep_kata(text)
|
||||
sep_tokens = [tokenizer.tokenize(t) for t in sep_text]
|
||||
sep_ids = [tokenizer.convert_tokens_to_ids(t) for t in sep_tokens]
|
||||
sep_ids = [2] + [item for sublist in sep_ids for item in sublist] + [3]
|
||||
return get_bert_feature_with_token(sep_ids, word2ph, device)
|
||||
|
||||
|
||||
def get_bert_feature_with_token(tokens, word2ph, device=config.bert_gen_config.device):
|
||||
if (
|
||||
sys.platform == "darwin"
|
||||
and torch.backends.mps.is_available()
|
||||
and device == "cpu"
|
||||
):
|
||||
device = "mps"
|
||||
if not device:
|
||||
device = "cuda"
|
||||
if device not in models.keys():
|
||||
models[device] = AutoModelForMaskedLM.from_pretrained(
|
||||
"./bert/bert-base-japanese-v3"
|
||||
).to(device)
|
||||
with torch.no_grad():
|
||||
inputs = torch.tensor(tokens).to(device).unsqueeze(0)
|
||||
token_type_ids = torch.zeros_like(inputs).to(device)
|
||||
attention_mask = torch.ones_like(inputs).to(device)
|
||||
inputs = {
|
||||
"input_ids": inputs,
|
||||
"token_type_ids": token_type_ids,
|
||||
"attention_mask": attention_mask,
|
||||
}
|
||||
|
||||
# for i in inputs:
|
||||
# inputs[i] = inputs[i].to(device)
|
||||
res = models[device](**inputs, output_hidden_states=True)
|
||||
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
|
||||
assert inputs["input_ids"].shape[-1] == len(word2ph)
|
||||
word2phone = word2ph
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
repeat_feature = res[i].repeat(word2phone[i], 1)
|
||||
phone_level_feature.append(repeat_feature)
|
||||
|
||||
phone_level_feature = torch.cat(phone_level_feature, dim=0)
|
||||
|
||||
return phone_level_feature.T
|
||||
586
oldVersion/V111/text/japanese.py
Normal file
586
oldVersion/V111/text/japanese.py
Normal file
@@ -0,0 +1,586 @@
|
||||
# Convert Japanese text to phonemes which is
|
||||
# compatible with Julius https://github.com/julius-speech/segmentation-kit
|
||||
import re
|
||||
import unicodedata
|
||||
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
from . import punctuation, symbols
|
||||
|
||||
try:
|
||||
import MeCab
|
||||
except ImportError as e:
|
||||
raise ImportError("Japanese requires mecab-python3 and unidic-lite.") from e
|
||||
from num2words import num2words
|
||||
|
||||
_CONVRULES = [
|
||||
# Conversion of 2 letters
|
||||
"アァ/ a a",
|
||||
"イィ/ i i",
|
||||
"イェ/ i e",
|
||||
"イャ/ y a",
|
||||
"ウゥ/ u:",
|
||||
"エェ/ e e",
|
||||
"オォ/ o:",
|
||||
"カァ/ k a:",
|
||||
"キィ/ k i:",
|
||||
"クゥ/ k u:",
|
||||
"クャ/ ky a",
|
||||
"クュ/ ky u",
|
||||
"クョ/ ky o",
|
||||
"ケェ/ k e:",
|
||||
"コォ/ k o:",
|
||||
"ガァ/ g a:",
|
||||
"ギィ/ g i:",
|
||||
"グゥ/ g u:",
|
||||
"グャ/ gy a",
|
||||
"グュ/ gy u",
|
||||
"グョ/ gy o",
|
||||
"ゲェ/ g e:",
|
||||
"ゴォ/ g o:",
|
||||
"サァ/ s a:",
|
||||
"シィ/ sh i:",
|
||||
"スゥ/ s u:",
|
||||
"スャ/ sh a",
|
||||
"スュ/ sh u",
|
||||
"スョ/ sh o",
|
||||
"セェ/ s e:",
|
||||
"ソォ/ s o:",
|
||||
"ザァ/ z a:",
|
||||
"ジィ/ j i:",
|
||||
"ズゥ/ z u:",
|
||||
"ズャ/ zy a",
|
||||
"ズュ/ zy u",
|
||||
"ズョ/ zy o",
|
||||
"ゼェ/ z e:",
|
||||
"ゾォ/ z o:",
|
||||
"タァ/ t a:",
|
||||
"チィ/ ch i:",
|
||||
"ツァ/ ts a",
|
||||
"ツィ/ ts i",
|
||||
"ツゥ/ ts u:",
|
||||
"ツャ/ ch a",
|
||||
"ツュ/ ch u",
|
||||
"ツョ/ ch o",
|
||||
"ツェ/ ts e",
|
||||
"ツォ/ ts o",
|
||||
"テェ/ t e:",
|
||||
"トォ/ t o:",
|
||||
"ダァ/ d a:",
|
||||
"ヂィ/ j i:",
|
||||
"ヅゥ/ d u:",
|
||||
"ヅャ/ zy a",
|
||||
"ヅュ/ zy u",
|
||||
"ヅョ/ zy o",
|
||||
"デェ/ d e:",
|
||||
"ドォ/ d o:",
|
||||
"ナァ/ n a:",
|
||||
"ニィ/ n i:",
|
||||
"ヌゥ/ n u:",
|
||||
"ヌャ/ ny a",
|
||||
"ヌュ/ ny u",
|
||||
"ヌョ/ ny o",
|
||||
"ネェ/ n e:",
|
||||
"ノォ/ n o:",
|
||||
"ハァ/ h a:",
|
||||
"ヒィ/ h i:",
|
||||
"フゥ/ f u:",
|
||||
"フャ/ hy a",
|
||||
"フュ/ hy u",
|
||||
"フョ/ hy o",
|
||||
"ヘェ/ h e:",
|
||||
"ホォ/ h o:",
|
||||
"バァ/ b a:",
|
||||
"ビィ/ b i:",
|
||||
"ブゥ/ b u:",
|
||||
"フャ/ hy a",
|
||||
"ブュ/ by u",
|
||||
"フョ/ hy o",
|
||||
"ベェ/ b e:",
|
||||
"ボォ/ b o:",
|
||||
"パァ/ p a:",
|
||||
"ピィ/ p i:",
|
||||
"プゥ/ p u:",
|
||||
"プャ/ py a",
|
||||
"プュ/ py u",
|
||||
"プョ/ py o",
|
||||
"ペェ/ p e:",
|
||||
"ポォ/ p o:",
|
||||
"マァ/ m a:",
|
||||
"ミィ/ m i:",
|
||||
"ムゥ/ m u:",
|
||||
"ムャ/ my a",
|
||||
"ムュ/ my u",
|
||||
"ムョ/ my o",
|
||||
"メェ/ m e:",
|
||||
"モォ/ m o:",
|
||||
"ヤァ/ y a:",
|
||||
"ユゥ/ y u:",
|
||||
"ユャ/ y a:",
|
||||
"ユュ/ y u:",
|
||||
"ユョ/ y o:",
|
||||
"ヨォ/ y o:",
|
||||
"ラァ/ r a:",
|
||||
"リィ/ r i:",
|
||||
"ルゥ/ r u:",
|
||||
"ルャ/ ry a",
|
||||
"ルュ/ ry u",
|
||||
"ルョ/ ry o",
|
||||
"レェ/ r e:",
|
||||
"ロォ/ r o:",
|
||||
"ワァ/ w a:",
|
||||
"ヲォ/ o:",
|
||||
"ディ/ d i",
|
||||
"デェ/ d e:",
|
||||
"デャ/ dy a",
|
||||
"デュ/ dy u",
|
||||
"デョ/ dy o",
|
||||
"ティ/ t i",
|
||||
"テェ/ t e:",
|
||||
"テャ/ ty a",
|
||||
"テュ/ ty u",
|
||||
"テョ/ ty o",
|
||||
"スィ/ s i",
|
||||
"ズァ/ z u a",
|
||||
"ズィ/ z i",
|
||||
"ズゥ/ z u",
|
||||
"ズャ/ zy a",
|
||||
"ズュ/ zy u",
|
||||
"ズョ/ zy o",
|
||||
"ズェ/ z e",
|
||||
"ズォ/ z o",
|
||||
"キャ/ ky a",
|
||||
"キュ/ ky u",
|
||||
"キョ/ ky o",
|
||||
"シャ/ sh a",
|
||||
"シュ/ sh u",
|
||||
"シェ/ sh e",
|
||||
"ショ/ sh o",
|
||||
"チャ/ ch a",
|
||||
"チュ/ ch u",
|
||||
"チェ/ ch e",
|
||||
"チョ/ ch o",
|
||||
"トゥ/ t u",
|
||||
"トャ/ ty a",
|
||||
"トュ/ ty u",
|
||||
"トョ/ ty o",
|
||||
"ドァ/ d o a",
|
||||
"ドゥ/ d u",
|
||||
"ドャ/ dy a",
|
||||
"ドュ/ dy u",
|
||||
"ドョ/ dy o",
|
||||
"ドォ/ d o:",
|
||||
"ニャ/ ny a",
|
||||
"ニュ/ ny u",
|
||||
"ニョ/ ny o",
|
||||
"ヒャ/ hy a",
|
||||
"ヒュ/ hy u",
|
||||
"ヒョ/ hy o",
|
||||
"ミャ/ my a",
|
||||
"ミュ/ my u",
|
||||
"ミョ/ my o",
|
||||
"リャ/ ry a",
|
||||
"リュ/ ry u",
|
||||
"リョ/ ry o",
|
||||
"ギャ/ gy a",
|
||||
"ギュ/ gy u",
|
||||
"ギョ/ gy o",
|
||||
"ヂェ/ j e",
|
||||
"ヂャ/ j a",
|
||||
"ヂュ/ j u",
|
||||
"ヂョ/ j o",
|
||||
"ジェ/ j e",
|
||||
"ジャ/ j a",
|
||||
"ジュ/ j u",
|
||||
"ジョ/ j o",
|
||||
"ビャ/ by a",
|
||||
"ビュ/ by u",
|
||||
"ビョ/ by o",
|
||||
"ピャ/ py a",
|
||||
"ピュ/ py u",
|
||||
"ピョ/ py o",
|
||||
"ウァ/ u a",
|
||||
"ウィ/ w i",
|
||||
"ウェ/ w e",
|
||||
"ウォ/ w o",
|
||||
"ファ/ f a",
|
||||
"フィ/ f i",
|
||||
"フゥ/ f u",
|
||||
"フャ/ hy a",
|
||||
"フュ/ hy u",
|
||||
"フョ/ hy o",
|
||||
"フェ/ f e",
|
||||
"フォ/ f o",
|
||||
"ヴァ/ b a",
|
||||
"ヴィ/ b i",
|
||||
"ヴェ/ b e",
|
||||
"ヴォ/ b o",
|
||||
"ヴュ/ by u",
|
||||
# Conversion of 1 letter
|
||||
"ア/ a",
|
||||
"イ/ i",
|
||||
"ウ/ u",
|
||||
"エ/ e",
|
||||
"オ/ o",
|
||||
"カ/ k a",
|
||||
"キ/ k i",
|
||||
"ク/ k u",
|
||||
"ケ/ k e",
|
||||
"コ/ k o",
|
||||
"サ/ s a",
|
||||
"シ/ sh i",
|
||||
"ス/ s u",
|
||||
"セ/ s e",
|
||||
"ソ/ s o",
|
||||
"タ/ t a",
|
||||
"チ/ ch i",
|
||||
"ツ/ ts u",
|
||||
"テ/ t e",
|
||||
"ト/ t o",
|
||||
"ナ/ n a",
|
||||
"ニ/ n i",
|
||||
"ヌ/ n u",
|
||||
"ネ/ n e",
|
||||
"ノ/ n o",
|
||||
"ハ/ h a",
|
||||
"ヒ/ h i",
|
||||
"フ/ f u",
|
||||
"ヘ/ h e",
|
||||
"ホ/ h o",
|
||||
"マ/ m a",
|
||||
"ミ/ m i",
|
||||
"ム/ m u",
|
||||
"メ/ m e",
|
||||
"モ/ m o",
|
||||
"ラ/ r a",
|
||||
"リ/ r i",
|
||||
"ル/ r u",
|
||||
"レ/ r e",
|
||||
"ロ/ r o",
|
||||
"ガ/ g a",
|
||||
"ギ/ g i",
|
||||
"グ/ g u",
|
||||
"ゲ/ g e",
|
||||
"ゴ/ g o",
|
||||
"ザ/ z a",
|
||||
"ジ/ j i",
|
||||
"ズ/ z u",
|
||||
"ゼ/ z e",
|
||||
"ゾ/ z o",
|
||||
"ダ/ d a",
|
||||
"ヂ/ j i",
|
||||
"ヅ/ z u",
|
||||
"デ/ d e",
|
||||
"ド/ d o",
|
||||
"バ/ b a",
|
||||
"ビ/ b i",
|
||||
"ブ/ b u",
|
||||
"ベ/ b e",
|
||||
"ボ/ b o",
|
||||
"パ/ p a",
|
||||
"ピ/ p i",
|
||||
"プ/ p u",
|
||||
"ペ/ p e",
|
||||
"ポ/ p o",
|
||||
"ヤ/ y a",
|
||||
"ユ/ y u",
|
||||
"ヨ/ y o",
|
||||
"ワ/ w a",
|
||||
"ヰ/ i",
|
||||
"ヱ/ e",
|
||||
"ヲ/ o",
|
||||
"ン/ N",
|
||||
"ッ/ q",
|
||||
"ヴ/ b u",
|
||||
"ー/:",
|
||||
# Try converting broken text
|
||||
"ァ/ a",
|
||||
"ィ/ i",
|
||||
"ゥ/ u",
|
||||
"ェ/ e",
|
||||
"ォ/ o",
|
||||
"ヮ/ w a",
|
||||
"ォ/ o",
|
||||
# Symbols
|
||||
"、/ ,",
|
||||
"。/ .",
|
||||
"！/ !",
|
||||
"？/ ?",
|
||||
"・/ ,",
|
||||
]
|
||||
|
||||
_COLON_RX = re.compile(":+")
|
||||
_REJECT_RX = re.compile("[^ a-zA-Z:,.?]")
|
||||
|
||||
|
||||
def _makerulemap():
|
||||
l = [tuple(x.split("/")) for x in _CONVRULES]
|
||||
return tuple({k: v for k, v in l if len(k) == i} for i in (1, 2))
|
||||
|
||||
|
||||
_RULEMAP1, _RULEMAP2 = _makerulemap()
|
||||
|
||||
|
||||
def kata2phoneme(text: str) -> str:
|
||||
"""Convert katakana text to phonemes."""
|
||||
text = text.strip()
|
||||
res = []
|
||||
while text:
|
||||
if len(text) >= 2:
|
||||
x = _RULEMAP2.get(text[:2])
|
||||
if x is not None:
|
||||
text = text[2:]
|
||||
res += x.split(" ")[1:]
|
||||
continue
|
||||
x = _RULEMAP1.get(text[0])
|
||||
if x is not None:
|
||||
text = text[1:]
|
||||
res += x.split(" ")[1:]
|
||||
continue
|
||||
res.append(text[0])
|
||||
text = text[1:]
|
||||
# res = _COLON_RX.sub(":", res)
|
||||
return res
|
||||
|
||||
|
||||
_KATAKANA = "".join(chr(ch) for ch in range(ord("ァ"), ord("ン") + 1))
|
||||
_HIRAGANA = "".join(chr(ch) for ch in range(ord("ぁ"), ord("ん") + 1))
|
||||
_HIRA2KATATRANS = str.maketrans(_HIRAGANA, _KATAKANA)
|
||||
|
||||
|
||||
def hira2kata(text: str) -> str:
|
||||
text = text.translate(_HIRA2KATATRANS)
|
||||
return text.replace("う゛", "ヴ")
|
||||
|
||||
|
||||
_SYMBOL_TOKENS = set(list("・、。？！"))
|
||||
_NO_YOMI_TOKENS = set(list("「」『』―（）［］[]"))
|
||||
_TAGGER = MeCab.Tagger()
|
||||
|
||||
|
||||
def text2kata(text: str) -> str:
|
||||
parsed = _TAGGER.parse(text)
|
||||
res = []
|
||||
for line in parsed.split("\n"):
|
||||
if line == "EOS":
|
||||
break
|
||||
parts = line.split("\t")
|
||||
|
||||
word, yomi = parts[0], parts[1]
|
||||
if yomi:
|
||||
res.append(yomi)
|
||||
else:
|
||||
if word in _SYMBOL_TOKENS:
|
||||
res.append(word)
|
||||
elif word in ("っ", "ッ"):
|
||||
res.append("ッ")
|
||||
elif word in _NO_YOMI_TOKENS:
|
||||
pass
|
||||
else:
|
||||
res.append(word)
|
||||
return hira2kata("".join(res))
|
||||
|
||||
|
||||
_ALPHASYMBOL_YOMI = {
|
||||
"#": "シャープ",
|
||||
"%": "パーセント",
|
||||
"&": "アンド",
|
||||
"+": "プラス",
|
||||
"-": "マイナス",
|
||||
":": "コロン",
|
||||
";": "セミコロン",
|
||||
"<": "小なり",
|
||||
"=": "イコール",
|
||||
">": "大なり",
|
||||
"@": "アット",
|
||||
"a": "エー",
|
||||
"b": "ビー",
|
||||
"c": "シー",
|
||||
"d": "ディー",
|
||||
"e": "イー",
|
||||
"f": "エフ",
|
||||
"g": "ジー",
|
||||
"h": "エイチ",
|
||||
"i": "アイ",
|
||||
"j": "ジェー",
|
||||
"k": "ケー",
|
||||
"l": "エル",
|
||||
"m": "エム",
|
||||
"n": "エヌ",
|
||||
"o": "オー",
|
||||
"p": "ピー",
|
||||
"q": "キュー",
|
||||
"r": "アール",
|
||||
"s": "エス",
|
||||
"t": "ティー",
|
||||
"u": "ユー",
|
||||
"v": "ブイ",
|
||||
"w": "ダブリュー",
|
||||
"x": "エックス",
|
||||
"y": "ワイ",
|
||||
"z": "ゼット",
|
||||
"α": "アルファ",
|
||||
"β": "ベータ",
|
||||
"γ": "ガンマ",
|
||||
"δ": "デルタ",
|
||||
"ε": "イプシロン",
|
||||
"ζ": "ゼータ",
|
||||
"η": "イータ",
|
||||
"θ": "シータ",
|
||||
"ι": "イオタ",
|
||||
"κ": "カッパ",
|
||||
"λ": "ラムダ",
|
||||
"μ": "ミュー",
|
||||
"ν": "ニュー",
|
||||
"ξ": "クサイ",
|
||||
"ο": "オミクロン",
|
||||
"π": "パイ",
|
||||
"ρ": "ロー",
|
||||
"σ": "シグマ",
|
||||
"τ": "タウ",
|
||||
"υ": "ウプシロン",
|
||||
"φ": "ファイ",
|
||||
"χ": "カイ",
|
||||
"ψ": "プサイ",
|
||||
"ω": "オメガ",
|
||||
}
|
||||
|
||||
|
||||
_NUMBER_WITH_SEPARATOR_RX = re.compile("[0-9]{1,3}(,[0-9]{3})+")
|
||||
_CURRENCY_MAP = {"$": "ドル", "¥": "円", "£": "ポンド", "€": "ユーロ"}
|
||||
_CURRENCY_RX = re.compile(r"([$¥£€])([0-9.]*[0-9])")
|
||||
_NUMBER_RX = re.compile(r"[0-9]+(\.[0-9]+)?")
|
||||
|
||||
|
||||
def japanese_convert_numbers_to_words(text: str) -> str:
|
||||
res = _NUMBER_WITH_SEPARATOR_RX.sub(lambda m: m[0].replace(",", ""), text)
|
||||
res = _CURRENCY_RX.sub(lambda m: m[2] + _CURRENCY_MAP.get(m[1], m[1]), res)
|
||||
res = _NUMBER_RX.sub(lambda m: num2words(m[0], lang="ja"), res)
|
||||
return res
|
||||
|
||||
|
||||
def japanese_convert_alpha_symbols_to_words(text: str) -> str:
|
||||
return "".join([_ALPHASYMBOL_YOMI.get(ch, ch) for ch in text.lower()])
|
||||
|
||||
|
||||
def japanese_text_to_phonemes(text: str) -> str:
|
||||
"""Convert Japanese text to phonemes."""
|
||||
res = unicodedata.normalize("NFKC", text)
|
||||
res = japanese_convert_numbers_to_words(res)
|
||||
# res = japanese_convert_alpha_symbols_to_words(res)
|
||||
res = text2kata(res)
|
||||
res = kata2phoneme(res)
|
||||
return res
|
||||
|
||||
|
||||
def is_japanese_character(char):
|
||||
# 定义日语文字系统的 Unicode 范围
|
||||
japanese_ranges = [
|
||||
(0x3040, 0x309F), # 平假名
|
||||
(0x30A0, 0x30FF), # 片假名
|
||||
(0x4E00, 0x9FFF), # 汉字 (CJK Unified Ideographs)
|
||||
(0x3400, 0x4DBF), # 汉字扩展 A
|
||||
(0x20000, 0x2A6DF), # 汉字扩展 B
|
||||
# 可以根据需要添加其他汉字扩展范围
|
||||
]
|
||||
|
||||
# 将字符的 Unicode 编码转换为整数
|
||||
char_code = ord(char)
|
||||
|
||||
# 检查字符是否在任何一个日语范围内
|
||||
for start, end in japanese_ranges:
|
||||
if start <= char_code <= end:
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
|
||||
rep_map = {
|
||||
"：": ",",
|
||||
"；": ",",
|
||||
"，": ",",
|
||||
"。": ".",
|
||||
"！": "!",
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"·": ",",
|
||||
"、": ",",
|
||||
"...": "…",
|
||||
}
|
||||
|
||||
|
||||
def replace_punctuation(text):
|
||||
pattern = re.compile("|".join(re.escape(p) for p in rep_map.keys()))
|
||||
|
||||
replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
|
||||
|
||||
replaced_text = re.sub(
|
||||
r"[^\u3040-\u309F\u30A0-\u30FF\u4E00-\u9FFF\u3400-\u4DBF"
|
||||
+ "".join(punctuation)
|
||||
+ r"]+",
|
||||
"",
|
||||
replaced_text,
|
||||
)
|
||||
|
||||
return replaced_text
|
||||
|
||||
|
||||
def text_normalize(text):
|
||||
res = unicodedata.normalize("NFKC", text)
|
||||
res = japanese_convert_numbers_to_words(res)
|
||||
# res = "".join([i for i in res if is_japanese_character(i)])
|
||||
res = replace_punctuation(res)
|
||||
return res
|
||||
|
||||
|
||||
def distribute_phone(n_phone, n_word):
|
||||
phones_per_word = [0] * n_word
|
||||
for task in range(n_phone):
|
||||
min_tasks = min(phones_per_word)
|
||||
min_index = phones_per_word.index(min_tasks)
|
||||
phones_per_word[min_index] += 1
|
||||
return phones_per_word
|
||||
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/bert-base-japanese-v3")
|
||||
|
||||
|
||||
def g2p(norm_text):
|
||||
tokenized = tokenizer.tokenize(norm_text)
|
||||
phs = []
|
||||
ph_groups = []
|
||||
for t in tokenized:
|
||||
if not t.startswith("#"):
|
||||
ph_groups.append([t])
|
||||
else:
|
||||
ph_groups[-1].append(t.replace("#", ""))
|
||||
word2ph = []
|
||||
for group in ph_groups:
|
||||
phonemes = kata2phoneme(text2kata("".join(group)))
|
||||
# phonemes = [i for i in phonemes if i in symbols]
|
||||
for i in phonemes:
|
||||
assert i in symbols, (group, norm_text, tokenized)
|
||||
phone_len = len(phonemes)
|
||||
word_len = len(group)
|
||||
|
||||
aaa = distribute_phone(phone_len, word_len)
|
||||
word2ph += aaa
|
||||
|
||||
phs += phonemes
|
||||
phones = ["_"] + phs + ["_"]
|
||||
tones = [0 for i in phones]
|
||||
word2ph = [1] + word2ph + [1]
|
||||
return phones, tones, word2ph
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/bert-base-japanese-v3")
|
||||
text = "hello,こんにちは、世界！……"
|
||||
from text.japanese_bert import get_bert_feature
|
||||
|
||||
text = text_normalize(text)
|
||||
print(text)
|
||||
phones, tones, word2ph = g2p(text)
|
||||
bert = get_bert_feature(text, word2ph)
|
||||
|
||||
print(phones, tones, word2ph, bert.shape)
|
||||
38
oldVersion/V111/text/japanese_bert.py
Normal file
38
oldVersion/V111/text/japanese_bert.py
Normal file
@@ -0,0 +1,38 @@
|
||||
import torch
|
||||
from transformers import AutoTokenizer, AutoModelForMaskedLM
|
||||
import sys
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/bert-base-japanese-v3")
|
||||
|
||||
models = dict()
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph, device=None):
|
||||
if (
|
||||
sys.platform == "darwin"
|
||||
and torch.backends.mps.is_available()
|
||||
and device == "cpu"
|
||||
):
|
||||
device = "mps"
|
||||
if not device:
|
||||
device = "cuda"
|
||||
if device not in models.keys():
|
||||
models[device] = AutoModelForMaskedLM.from_pretrained(
|
||||
"./bert/bert-base-japanese-v3"
|
||||
).to(device)
|
||||
with torch.no_grad():
|
||||
inputs = tokenizer(text, return_tensors="pt")
|
||||
for i in inputs:
|
||||
inputs[i] = inputs[i].to(device)
|
||||
res = models[device](**inputs, output_hidden_states=True)
|
||||
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
|
||||
assert inputs["input_ids"].shape[-1] == len(word2ph)
|
||||
word2phone = word2ph
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
repeat_feature = res[i].repeat(word2phone[i], 1)
|
||||
phone_level_feature.append(repeat_feature)
|
||||
|
||||
phone_level_feature = torch.cat(phone_level_feature, dim=0)
|
||||
|
||||
return phone_level_feature.T
|
||||
429
oldVersion/V111/text/opencpop-strict.txt
Normal file
429
oldVersion/V111/text/opencpop-strict.txt
Normal file
@@ -0,0 +1,429 @@
|
||||
a AA a
|
||||
ai AA ai
|
||||
an AA an
|
||||
ang AA ang
|
||||
ao AA ao
|
||||
ba b a
|
||||
bai b ai
|
||||
ban b an
|
||||
bang b ang
|
||||
bao b ao
|
||||
bei b ei
|
||||
ben b en
|
||||
beng b eng
|
||||
bi b i
|
||||
bian b ian
|
||||
biao b iao
|
||||
bie b ie
|
||||
bin b in
|
||||
bing b ing
|
||||
bo b o
|
||||
bu b u
|
||||
ca c a
|
||||
cai c ai
|
||||
can c an
|
||||
cang c ang
|
||||
cao c ao
|
||||
ce c e
|
||||
cei c ei
|
||||
cen c en
|
||||
ceng c eng
|
||||
cha ch a
|
||||
chai ch ai
|
||||
chan ch an
|
||||
chang ch ang
|
||||
chao ch ao
|
||||
che ch e
|
||||
chen ch en
|
||||
cheng ch eng
|
||||
chi ch ir
|
||||
chong ch ong
|
||||
chou ch ou
|
||||
chu ch u
|
||||
chua ch ua
|
||||
chuai ch uai
|
||||
chuan ch uan
|
||||
chuang ch uang
|
||||
chui ch ui
|
||||
chun ch un
|
||||
chuo ch uo
|
||||
ci c i0
|
||||
cong c ong
|
||||
cou c ou
|
||||
cu c u
|
||||
cuan c uan
|
||||
cui c ui
|
||||
cun c un
|
||||
cuo c uo
|
||||
da d a
|
||||
dai d ai
|
||||
dan d an
|
||||
dang d ang
|
||||
dao d ao
|
||||
de d e
|
||||
dei d ei
|
||||
den d en
|
||||
deng d eng
|
||||
di d i
|
||||
dia d ia
|
||||
dian d ian
|
||||
diao d iao
|
||||
die d ie
|
||||
ding d ing
|
||||
diu d iu
|
||||
dong d ong
|
||||
dou d ou
|
||||
du d u
|
||||
duan d uan
|
||||
dui d ui
|
||||
dun d un
|
||||
duo d uo
|
||||
e EE e
|
||||
ei EE ei
|
||||
en EE en
|
||||
eng EE eng
|
||||
er EE er
|
||||
fa f a
|
||||
fan f an
|
||||
fang f ang
|
||||
fei f ei
|
||||
fen f en
|
||||
feng f eng
|
||||
fo f o
|
||||
fou f ou
|
||||
fu f u
|
||||
ga g a
|
||||
gai g ai
|
||||
gan g an
|
||||
gang g ang
|
||||
gao g ao
|
||||
ge g e
|
||||
gei g ei
|
||||
gen g en
|
||||
geng g eng
|
||||
gong g ong
|
||||
gou g ou
|
||||
gu g u
|
||||
gua g ua
|
||||
guai g uai
|
||||
guan g uan
|
||||
guang g uang
|
||||
gui g ui
|
||||
gun g un
|
||||
guo g uo
|
||||
ha h a
|
||||
hai h ai
|
||||
han h an
|
||||
hang h ang
|
||||
hao h ao
|
||||
he h e
|
||||
hei h ei
|
||||
hen h en
|
||||
heng h eng
|
||||
hong h ong
|
||||
hou h ou
|
||||
hu h u
|
||||
hua h ua
|
||||
huai h uai
|
||||
huan h uan
|
||||
huang h uang
|
||||
hui h ui
|
||||
hun h un
|
||||
huo h uo
|
||||
ji j i
|
||||
jia j ia
|
||||
jian j ian
|
||||
jiang j iang
|
||||
jiao j iao
|
||||
jie j ie
|
||||
jin j in
|
||||
jing j ing
|
||||
jiong j iong
|
||||
jiu j iu
|
||||
ju j v
|
||||
jv j v
|
||||
juan j van
|
||||
jvan j van
|
||||
jue j ve
|
||||
jve j ve
|
||||
jun j vn
|
||||
jvn j vn
|
||||
ka k a
|
||||
kai k ai
|
||||
kan k an
|
||||
kang k ang
|
||||
kao k ao
|
||||
ke k e
|
||||
kei k ei
|
||||
ken k en
|
||||
keng k eng
|
||||
kong k ong
|
||||
kou k ou
|
||||
ku k u
|
||||
kua k ua
|
||||
kuai k uai
|
||||
kuan k uan
|
||||
kuang k uang
|
||||
kui k ui
|
||||
kun k un
|
||||
kuo k uo
|
||||
la l a
|
||||
lai l ai
|
||||
lan l an
|
||||
lang l ang
|
||||
lao l ao
|
||||
le l e
|
||||
lei l ei
|
||||
leng l eng
|
||||
li l i
|
||||
lia l ia
|
||||
lian l ian
|
||||
liang l iang
|
||||
liao l iao
|
||||
lie l ie
|
||||
lin l in
|
||||
ling l ing
|
||||
liu l iu
|
||||
lo l o
|
||||
long l ong
|
||||
lou l ou
|
||||
lu l u
|
||||
luan l uan
|
||||
lun l un
|
||||
luo l uo
|
||||
lv l v
|
||||
lve l ve
|
||||
ma m a
|
||||
mai m ai
|
||||
man m an
|
||||
mang m ang
|
||||
mao m ao
|
||||
me m e
|
||||
mei m ei
|
||||
men m en
|
||||
meng m eng
|
||||
mi m i
|
||||
mian m ian
|
||||
miao m iao
|
||||
mie m ie
|
||||
min m in
|
||||
ming m ing
|
||||
miu m iu
|
||||
mo m o
|
||||
mou m ou
|
||||
mu m u
|
||||
na n a
|
||||
nai n ai
|
||||
nan n an
|
||||
nang n ang
|
||||
nao n ao
|
||||
ne n e
|
||||
nei n ei
|
||||
nen n en
|
||||
neng n eng
|
||||
ni n i
|
||||
nian n ian
|
||||
niang n iang
|
||||
niao n iao
|
||||
nie n ie
|
||||
nin n in
|
||||
ning n ing
|
||||
niu n iu
|
||||
nong n ong
|
||||
nou n ou
|
||||
nu n u
|
||||
nuan n uan
|
||||
nun n un
|
||||
nuo n uo
|
||||
nv n v
|
||||
nve n ve
|
||||
o OO o
|
||||
ou OO ou
|
||||
pa p a
|
||||
pai p ai
|
||||
pan p an
|
||||
pang p ang
|
||||
pao p ao
|
||||
pei p ei
|
||||
pen p en
|
||||
peng p eng
|
||||
pi p i
|
||||
pian p ian
|
||||
piao p iao
|
||||
pie p ie
|
||||
pin p in
|
||||
ping p ing
|
||||
po p o
|
||||
pou p ou
|
||||
pu p u
|
||||
qi q i
|
||||
qia q ia
|
||||
qian q ian
|
||||
qiang q iang
|
||||
qiao q iao
|
||||
qie q ie
|
||||
qin q in
|
||||
qing q ing
|
||||
qiong q iong
|
||||
qiu q iu
|
||||
qu q v
|
||||
qv q v
|
||||
quan q van
|
||||
qvan q van
|
||||
que q ve
|
||||
qve q ve
|
||||
qun q vn
|
||||
qvn q vn
|
||||
ran r an
|
||||
rang r ang
|
||||
rao r ao
|
||||
re r e
|
||||
ren r en
|
||||
reng r eng
|
||||
ri r ir
|
||||
rong r ong
|
||||
rou r ou
|
||||
ru r u
|
||||
rua r ua
|
||||
ruan r uan
|
||||
rui r ui
|
||||
run r un
|
||||
ruo r uo
|
||||
sa s a
|
||||
sai s ai
|
||||
san s an
|
||||
sang s ang
|
||||
sao s ao
|
||||
se s e
|
||||
sen s en
|
||||
seng s eng
|
||||
sha sh a
|
||||
shai sh ai
|
||||
shan sh an
|
||||
shang sh ang
|
||||
shao sh ao
|
||||
she sh e
|
||||
shei sh ei
|
||||
shen sh en
|
||||
sheng sh eng
|
||||
shi sh ir
|
||||
shou sh ou
|
||||
shu sh u
|
||||
shua sh ua
|
||||
shuai sh uai
|
||||
shuan sh uan
|
||||
shuang sh uang
|
||||
shui sh ui
|
||||
shun sh un
|
||||
shuo sh uo
|
||||
si s i0
|
||||
song s ong
|
||||
sou s ou
|
||||
su s u
|
||||
suan s uan
|
||||
sui s ui
|
||||
sun s un
|
||||
suo s uo
|
||||
ta t a
|
||||
tai t ai
|
||||
tan t an
|
||||
tang t ang
|
||||
tao t ao
|
||||
te t e
|
||||
tei t ei
|
||||
teng t eng
|
||||
ti t i
|
||||
tian t ian
|
||||
tiao t iao
|
||||
tie t ie
|
||||
ting t ing
|
||||
tong t ong
|
||||
tou t ou
|
||||
tu t u
|
||||
tuan t uan
|
||||
tui t ui
|
||||
tun t un
|
||||
tuo t uo
|
||||
wa w a
|
||||
wai w ai
|
||||
wan w an
|
||||
wang w ang
|
||||
wei w ei
|
||||
wen w en
|
||||
weng w eng
|
||||
wo w o
|
||||
wu w u
|
||||
xi x i
|
||||
xia x ia
|
||||
xian x ian
|
||||
xiang x iang
|
||||
xiao x iao
|
||||
xie x ie
|
||||
xin x in
|
||||
xing x ing
|
||||
xiong x iong
|
||||
xiu x iu
|
||||
xu x v
|
||||
xv x v
|
||||
xuan x van
|
||||
xvan x van
|
||||
xue x ve
|
||||
xve x ve
|
||||
xun x vn
|
||||
xvn x vn
|
||||
ya y a
|
||||
yan y En
|
||||
yang y ang
|
||||
yao y ao
|
||||
ye y E
|
||||
yi y i
|
||||
yin y in
|
||||
ying y ing
|
||||
yo y o
|
||||
yong y ong
|
||||
you y ou
|
||||
yu y v
|
||||
yv y v
|
||||
yuan y van
|
||||
yvan y van
|
||||
yue y ve
|
||||
yve y ve
|
||||
yun y vn
|
||||
yvn y vn
|
||||
za z a
|
||||
zai z ai
|
||||
zan z an
|
||||
zang z ang
|
||||
zao z ao
|
||||
ze z e
|
||||
zei z ei
|
||||
zen z en
|
||||
zeng z eng
|
||||
zha zh a
|
||||
zhai zh ai
|
||||
zhan zh an
|
||||
zhang zh ang
|
||||
zhao zh ao
|
||||
zhe zh e
|
||||
zhei zh ei
|
||||
zhen zh en
|
||||
zheng zh eng
|
||||
zhi zh ir
|
||||
zhong zh ong
|
||||
zhou zh ou
|
||||
zhu zh u
|
||||
zhua zh ua
|
||||
zhuai zh uai
|
||||
zhuan zh uan
|
||||
zhuang zh uang
|
||||
zhui zh ui
|
||||
zhun zh un
|
||||
zhuo zh uo
|
||||
zi z i0
|
||||
zong z ong
|
||||
zou z ou
|
||||
zu z u
|
||||
zuan z uan
|
||||
zui z ui
|
||||
zun z un
|
||||
zuo z uo
|
||||
187
oldVersion/V111/text/symbols.py
Normal file
187
oldVersion/V111/text/symbols.py
Normal file
@@ -0,0 +1,187 @@
|
||||
punctuation = ["!", "?", "…", ",", ".", "'", "-"]
|
||||
pu_symbols = punctuation + ["SP", "UNK"]
|
||||
pad = "_"
|
||||
|
||||
# chinese
|
||||
zh_symbols = [
|
||||
"E",
|
||||
"En",
|
||||
"a",
|
||||
"ai",
|
||||
"an",
|
||||
"ang",
|
||||
"ao",
|
||||
"b",
|
||||
"c",
|
||||
"ch",
|
||||
"d",
|
||||
"e",
|
||||
"ei",
|
||||
"en",
|
||||
"eng",
|
||||
"er",
|
||||
"f",
|
||||
"g",
|
||||
"h",
|
||||
"i",
|
||||
"i0",
|
||||
"ia",
|
||||
"ian",
|
||||
"iang",
|
||||
"iao",
|
||||
"ie",
|
||||
"in",
|
||||
"ing",
|
||||
"iong",
|
||||
"ir",
|
||||
"iu",
|
||||
"j",
|
||||
"k",
|
||||
"l",
|
||||
"m",
|
||||
"n",
|
||||
"o",
|
||||
"ong",
|
||||
"ou",
|
||||
"p",
|
||||
"q",
|
||||
"r",
|
||||
"s",
|
||||
"sh",
|
||||
"t",
|
||||
"u",
|
||||
"ua",
|
||||
"uai",
|
||||
"uan",
|
||||
"uang",
|
||||
"ui",
|
||||
"un",
|
||||
"uo",
|
||||
"v",
|
||||
"van",
|
||||
"ve",
|
||||
"vn",
|
||||
"w",
|
||||
"x",
|
||||
"y",
|
||||
"z",
|
||||
"zh",
|
||||
"AA",
|
||||
"EE",
|
||||
"OO",
|
||||
]
|
||||
num_zh_tones = 6
|
||||
|
||||
# japanese
|
||||
ja_symbols = [
|
||||
"N",
|
||||
"a",
|
||||
"a:",
|
||||
"b",
|
||||
"by",
|
||||
"ch",
|
||||
"d",
|
||||
"dy",
|
||||
"e",
|
||||
"e:",
|
||||
"f",
|
||||
"g",
|
||||
"gy",
|
||||
"h",
|
||||
"hy",
|
||||
"i",
|
||||
"i:",
|
||||
"j",
|
||||
"k",
|
||||
"ky",
|
||||
"m",
|
||||
"my",
|
||||
"n",
|
||||
"ny",
|
||||
"o",
|
||||
"o:",
|
||||
"p",
|
||||
"py",
|
||||
"q",
|
||||
"r",
|
||||
"ry",
|
||||
"s",
|
||||
"sh",
|
||||
"t",
|
||||
"ts",
|
||||
"ty",
|
||||
"u",
|
||||
"u:",
|
||||
"w",
|
||||
"y",
|
||||
"z",
|
||||
"zy",
|
||||
]
|
||||
num_ja_tones = 1
|
||||
|
||||
# English
|
||||
en_symbols = [
|
||||
"aa",
|
||||
"ae",
|
||||
"ah",
|
||||
"ao",
|
||||
"aw",
|
||||
"ay",
|
||||
"b",
|
||||
"ch",
|
||||
"d",
|
||||
"dh",
|
||||
"eh",
|
||||
"er",
|
||||
"ey",
|
||||
"f",
|
||||
"g",
|
||||
"hh",
|
||||
"ih",
|
||||
"iy",
|
||||
"jh",
|
||||
"k",
|
||||
"l",
|
||||
"m",
|
||||
"n",
|
||||
"ng",
|
||||
"ow",
|
||||
"oy",
|
||||
"p",
|
||||
"r",
|
||||
"s",
|
||||
"sh",
|
||||
"t",
|
||||
"th",
|
||||
"uh",
|
||||
"uw",
|
||||
"V",
|
||||
"w",
|
||||
"y",
|
||||
"z",
|
||||
"zh",
|
||||
]
|
||||
num_en_tones = 4
|
||||
|
||||
# combine all symbols
|
||||
normal_symbols = sorted(set(zh_symbols + ja_symbols + en_symbols))
|
||||
symbols = [pad] + normal_symbols + pu_symbols
|
||||
sil_phonemes_ids = [symbols.index(i) for i in pu_symbols]
|
||||
|
||||
# combine all tones
|
||||
num_tones = num_zh_tones + num_ja_tones + num_en_tones
|
||||
|
||||
# language maps
|
||||
language_id_map = {"ZH": 0, "JP": 1, "EN": 2}
|
||||
num_languages = len(language_id_map.keys())
|
||||
|
||||
language_tone_start_map = {
|
||||
"ZH": 0,
|
||||
"JP": num_zh_tones,
|
||||
"EN": num_zh_tones + num_ja_tones,
|
||||
}
|
||||
|
||||
if __name__ == "__main__":
|
||||
a = set(zh_symbols)
|
||||
b = set(en_symbols)
|
||||
print(sorted(a & b))
|
||||
769
oldVersion/V111/text/tone_sandhi.py
Normal file
769
oldVersion/V111/text/tone_sandhi.py
Normal file
@@ -0,0 +1,769 @@
|
||||
# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
from typing import List
|
||||
from typing import Tuple
|
||||
|
||||
import jieba
|
||||
from pypinyin import lazy_pinyin
|
||||
from pypinyin import Style
|
||||
|
||||
|
||||
class ToneSandhi:
|
||||
def __init__(self):
|
||||
self.must_neural_tone_words = {
|
||||
"麻烦",
|
||||
"麻利",
|
||||
"鸳鸯",
|
||||
"高粱",
|
||||
"骨头",
|
||||
"骆驼",
|
||||
"马虎",
|
||||
"首饰",
|
||||
"馒头",
|
||||
"馄饨",
|
||||
"风筝",
|
||||
"难为",
|
||||
"队伍",
|
||||
"阔气",
|
||||
"闺女",
|
||||
"门道",
|
||||
"锄头",
|
||||
"铺盖",
|
||||
"铃铛",
|
||||
"铁匠",
|
||||
"钥匙",
|
||||
"里脊",
|
||||
"里头",
|
||||
"部分",
|
||||
"那么",
|
||||
"道士",
|
||||
"造化",
|
||||
"迷糊",
|
||||
"连累",
|
||||
"这么",
|
||||
"这个",
|
||||
"运气",
|
||||
"过去",
|
||||
"软和",
|
||||
"转悠",
|
||||
"踏实",
|
||||
"跳蚤",
|
||||
"跟头",
|
||||
"趔趄",
|
||||
"财主",
|
||||
"豆腐",
|
||||
"讲究",
|
||||
"记性",
|
||||
"记号",
|
||||
"认识",
|
||||
"规矩",
|
||||
"见识",
|
||||
"裁缝",
|
||||
"补丁",
|
||||
"衣裳",
|
||||
"衣服",
|
||||
"衙门",
|
||||
"街坊",
|
||||
"行李",
|
||||
"行当",
|
||||
"蛤蟆",
|
||||
"蘑菇",
|
||||
"薄荷",
|
||||
"葫芦",
|
||||
"葡萄",
|
||||
"萝卜",
|
||||
"荸荠",
|
||||
"苗条",
|
||||
"苗头",
|
||||
"苍蝇",
|
||||
"芝麻",
|
||||
"舒服",
|
||||
"舒坦",
|
||||
"舌头",
|
||||
"自在",
|
||||
"膏药",
|
||||
"脾气",
|
||||
"脑袋",
|
||||
"脊梁",
|
||||
"能耐",
|
||||
"胳膊",
|
||||
"胭脂",
|
||||
"胡萝",
|
||||
"胡琴",
|
||||
"胡同",
|
||||
"聪明",
|
||||
"耽误",
|
||||
"耽搁",
|
||||
"耷拉",
|
||||
"耳朵",
|
||||
"老爷",
|
||||
"老实",
|
||||
"老婆",
|
||||
"老头",
|
||||
"老太",
|
||||
"翻腾",
|
||||
"罗嗦",
|
||||
"罐头",
|
||||
"编辑",
|
||||
"结实",
|
||||
"红火",
|
||||
"累赘",
|
||||
"糨糊",
|
||||
"糊涂",
|
||||
"精神",
|
||||
"粮食",
|
||||
"簸箕",
|
||||
"篱笆",
|
||||
"算计",
|
||||
"算盘",
|
||||
"答应",
|
||||
"笤帚",
|
||||
"笑语",
|
||||
"笑话",
|
||||
"窟窿",
|
||||
"窝囊",
|
||||
"窗户",
|
||||
"稳当",
|
||||
"稀罕",
|
||||
"称呼",
|
||||
"秧歌",
|
||||
"秀气",
|
||||
"秀才",
|
||||
"福气",
|
||||
"祖宗",
|
||||
"砚台",
|
||||
"码头",
|
||||
"石榴",
|
||||
"石头",
|
||||
"石匠",
|
||||
"知识",
|
||||
"眼睛",
|
||||
"眯缝",
|
||||
"眨巴",
|
||||
"眉毛",
|
||||
"相声",
|
||||
"盘算",
|
||||
"白净",
|
||||
"痢疾",
|
||||
"痛快",
|
||||
"疟疾",
|
||||
"疙瘩",
|
||||
"疏忽",
|
||||
"畜生",
|
||||
"生意",
|
||||
"甘蔗",
|
||||
"琵琶",
|
||||
"琢磨",
|
||||
"琉璃",
|
||||
"玻璃",
|
||||
"玫瑰",
|
||||
"玄乎",
|
||||
"狐狸",
|
||||
"状元",
|
||||
"特务",
|
||||
"牲口",
|
||||
"牙碜",
|
||||
"牌楼",
|
||||
"爽快",
|
||||
"爱人",
|
||||
"热闹",
|
||||
"烧饼",
|
||||
"烟筒",
|
||||
"烂糊",
|
||||
"点心",
|
||||
"炊帚",
|
||||
"灯笼",
|
||||
"火候",
|
||||
"漂亮",
|
||||
"滑溜",
|
||||
"溜达",
|
||||
"温和",
|
||||
"清楚",
|
||||
"消息",
|
||||
"浪头",
|
||||
"活泼",
|
||||
"比方",
|
||||
"正经",
|
||||
"欺负",
|
||||
"模糊",
|
||||
"槟榔",
|
||||
"棺材",
|
||||
"棒槌",
|
||||
"棉花",
|
||||
"核桃",
|
||||
"栅栏",
|
||||
"柴火",
|
||||
"架势",
|
||||
"枕头",
|
||||
"枇杷",
|
||||
"机灵",
|
||||
"本事",
|
||||
"木头",
|
||||
"木匠",
|
||||
"朋友",
|
||||
"月饼",
|
||||
"月亮",
|
||||
"暖和",
|
||||
"明白",
|
||||
"时候",
|
||||
"新鲜",
|
||||
"故事",
|
||||
"收拾",
|
||||
"收成",
|
||||
"提防",
|
||||
"挖苦",
|
||||
"挑剔",
|
||||
"指甲",
|
||||
"指头",
|
||||
"拾掇",
|
||||
"拳头",
|
||||
"拨弄",
|
||||
"招牌",
|
||||
"招呼",
|
||||
"抬举",
|
||||
"护士",
|
||||
"折腾",
|
||||
"扫帚",
|
||||
"打量",
|
||||
"打算",
|
||||
"打点",
|
||||
"打扮",
|
||||
"打听",
|
||||
"打发",
|
||||
"扎实",
|
||||
"扁担",
|
||||
"戒指",
|
||||
"懒得",
|
||||
"意识",
|
||||
"意思",
|
||||
"情形",
|
||||
"悟性",
|
||||
"怪物",
|
||||
"思量",
|
||||
"怎么",
|
||||
"念头",
|
||||
"念叨",
|
||||
"快活",
|
||||
"忙活",
|
||||
"志气",
|
||||
"心思",
|
||||
"得罪",
|
||||
"张罗",
|
||||
"弟兄",
|
||||
"开通",
|
||||
"应酬",
|
||||
"庄稼",
|
||||
"干事",
|
||||
"帮手",
|
||||
"帐篷",
|
||||
"希罕",
|
||||
"师父",
|
||||
"师傅",
|
||||
"巴结",
|
||||
"巴掌",
|
||||
"差事",
|
||||
"工夫",
|
||||
"岁数",
|
||||
"屁股",
|
||||
"尾巴",
|
||||
"少爷",
|
||||
"小气",
|
||||
"小伙",
|
||||
"将就",
|
||||
"对头",
|
||||
"对付",
|
||||
"寡妇",
|
||||
"家伙",
|
||||
"客气",
|
||||
"实在",
|
||||
"官司",
|
||||
"学问",
|
||||
"学生",
|
||||
"字号",
|
||||
"嫁妆",
|
||||
"媳妇",
|
||||
"媒人",
|
||||
"婆家",
|
||||
"娘家",
|
||||
"委屈",
|
||||
"姑娘",
|
||||
"姐夫",
|
||||
"妯娌",
|
||||
"妥当",
|
||||
"妖精",
|
||||
"奴才",
|
||||
"女婿",
|
||||
"头发",
|
||||
"太阳",
|
||||
"大爷",
|
||||
"大方",
|
||||
"大意",
|
||||
"大夫",
|
||||
"多少",
|
||||
"多么",
|
||||
"外甥",
|
||||
"壮实",
|
||||
"地道",
|
||||
"地方",
|
||||
"在乎",
|
||||
"困难",
|
||||
"嘴巴",
|
||||
"嘱咐",
|
||||
"嘟囔",
|
||||
"嘀咕",
|
||||
"喜欢",
|
||||
"喇嘛",
|
||||
"喇叭",
|
||||
"商量",
|
||||
"唾沫",
|
||||
"哑巴",
|
||||
"哈欠",
|
||||
"哆嗦",
|
||||
"咳嗽",
|
||||
"和尚",
|
||||
"告诉",
|
||||
"告示",
|
||||
"含糊",
|
||||
"吓唬",
|
||||
"后头",
|
||||
"名字",
|
||||
"名堂",
|
||||
"合同",
|
||||
"吆喝",
|
||||
"叫唤",
|
||||
"口袋",
|
||||
"厚道",
|
||||
"厉害",
|
||||
"千斤",
|
||||
"包袱",
|
||||
"包涵",
|
||||
"匀称",
|
||||
"勤快",
|
||||
"动静",
|
||||
"动弹",
|
||||
"功夫",
|
||||
"力气",
|
||||
"前头",
|
||||
"刺猬",
|
||||
"刺激",
|
||||
"别扭",
|
||||
"利落",
|
||||
"利索",
|
||||
"利害",
|
||||
"分析",
|
||||
"出息",
|
||||
"凑合",
|
||||
"凉快",
|
||||
"冷战",
|
||||
"冤枉",
|
||||
"冒失",
|
||||
"养活",
|
||||
"关系",
|
||||
"先生",
|
||||
"兄弟",
|
||||
"便宜",
|
||||
"使唤",
|
||||
"佩服",
|
||||
"作坊",
|
||||
"体面",
|
||||
"位置",
|
||||
"似的",
|
||||
"伙计",
|
||||
"休息",
|
||||
"什么",
|
||||
"人家",
|
||||
"亲戚",
|
||||
"亲家",
|
||||
"交情",
|
||||
"云彩",
|
||||
"事情",
|
||||
"买卖",
|
||||
"主意",
|
||||
"丫头",
|
||||
"丧气",
|
||||
"两口",
|
||||
"东西",
|
||||
"东家",
|
||||
"世故",
|
||||
"不由",
|
||||
"不在",
|
||||
"下水",
|
||||
"下巴",
|
||||
"上头",
|
||||
"上司",
|
||||
"丈夫",
|
||||
"丈人",
|
||||
"一辈",
|
||||
"那个",
|
||||
"菩萨",
|
||||
"父亲",
|
||||
"母亲",
|
||||
"咕噜",
|
||||
"邋遢",
|
||||
"费用",
|
||||
"冤家",
|
||||
"甜头",
|
||||
"介绍",
|
||||
"荒唐",
|
||||
"大人",
|
||||
"泥鳅",
|
||||
"幸福",
|
||||
"熟悉",
|
||||
"计划",
|
||||
"扑腾",
|
||||
"蜡烛",
|
||||
"姥爷",
|
||||
"照顾",
|
||||
"喉咙",
|
||||
"吉他",
|
||||
"弄堂",
|
||||
"蚂蚱",
|
||||
"凤凰",
|
||||
"拖沓",
|
||||
"寒碜",
|
||||
"糟蹋",
|
||||
"倒腾",
|
||||
"报复",
|
||||
"逻辑",
|
||||
"盘缠",
|
||||
"喽啰",
|
||||
"牢骚",
|
||||
"咖喱",
|
||||
"扫把",
|
||||
"惦记",
|
||||
}
|
||||
self.must_not_neural_tone_words = {
|
||||
"男子",
|
||||
"女子",
|
||||
"分子",
|
||||
"原子",
|
||||
"量子",
|
||||
"莲子",
|
||||
"石子",
|
||||
"瓜子",
|
||||
"电子",
|
||||
"人人",
|
||||
"虎虎",
|
||||
}
|
||||
self.punc = "：，；。？！“”‘’':,;.?!"
|
||||
|
||||
# the meaning of jieba pos tag: https://blog.csdn.net/weixin_44174352/article/details/113731041
|
||||
# e.g.
|
||||
# word: "家里"
|
||||
# pos: "s"
|
||||
# finals: ['ia1', 'i3']
|
||||
def _neural_sandhi(self, word: str, pos: str, finals: List[str]) -> List[str]:
|
||||
# reduplication words for n. and v. e.g. 奶奶, 试试, 旺旺
|
||||
for j, item in enumerate(word):
|
||||
if (
|
||||
j - 1 >= 0
|
||||
and item == word[j - 1]
|
||||
and pos[0] in {"n", "v", "a"}
|
||||
and word not in self.must_not_neural_tone_words
|
||||
):
|
||||
finals[j] = finals[j][:-1] + "5"
|
||||
ge_idx = word.find("个")
|
||||
if len(word) >= 1 and word[-1] in "吧呢啊呐噻嘛吖嗨呐哦哒额滴哩哟喽啰耶喔诶":
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
elif len(word) >= 1 and word[-1] in "的地得":
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
# e.g. 走了, 看着, 去过
|
||||
# elif len(word) == 1 and word in "了着过" and pos in {"ul", "uz", "ug"}:
|
||||
# finals[-1] = finals[-1][:-1] + "5"
|
||||
elif (
|
||||
len(word) > 1
|
||||
and word[-1] in "们子"
|
||||
and pos in {"r", "n"}
|
||||
and word not in self.must_not_neural_tone_words
|
||||
):
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
# e.g. 桌上, 地下, 家里
|
||||
elif len(word) > 1 and word[-1] in "上下里" and pos in {"s", "l", "f"}:
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
# e.g. 上来, 下去
|
||||
elif len(word) > 1 and word[-1] in "来去" and word[-2] in "上下进出回过起开":
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
# 个做量词
|
||||
elif (
|
||||
ge_idx >= 1
|
||||
and (word[ge_idx - 1].isnumeric() or word[ge_idx - 1] in "几有两半多各整每做是")
|
||||
) or word == "个":
|
||||
finals[ge_idx] = finals[ge_idx][:-1] + "5"
|
||||
else:
|
||||
if (
|
||||
word in self.must_neural_tone_words
|
||||
or word[-2:] in self.must_neural_tone_words
|
||||
):
|
||||
finals[-1] = finals[-1][:-1] + "5"
|
||||
|
||||
word_list = self._split_word(word)
|
||||
finals_list = [finals[: len(word_list[0])], finals[len(word_list[0]) :]]
|
||||
for i, word in enumerate(word_list):
|
||||
# conventional neural in Chinese
|
||||
if (
|
||||
word in self.must_neural_tone_words
|
||||
or word[-2:] in self.must_neural_tone_words
|
||||
):
|
||||
finals_list[i][-1] = finals_list[i][-1][:-1] + "5"
|
||||
finals = sum(finals_list, [])
|
||||
return finals
|
||||
|
||||
def _bu_sandhi(self, word: str, finals: List[str]) -> List[str]:
|
||||
# e.g. 看不懂
|
||||
if len(word) == 3 and word[1] == "不":
|
||||
finals[1] = finals[1][:-1] + "5"
|
||||
else:
|
||||
for i, char in enumerate(word):
|
||||
# "不" before tone4 should be bu2, e.g. 不怕
|
||||
if char == "不" and i + 1 < len(word) and finals[i + 1][-1] == "4":
|
||||
finals[i] = finals[i][:-1] + "2"
|
||||
return finals
|
||||
|
||||
def _yi_sandhi(self, word: str, finals: List[str]) -> List[str]:
|
||||
# "一" in number sequences, e.g. 一零零, 二一零
|
||||
if word.find("一") != -1 and all(
|
||||
[item.isnumeric() for item in word if item != "一"]
|
||||
):
|
||||
return finals
|
||||
# "一" between reduplication words should be yi5, e.g. 看一看
|
||||
elif len(word) == 3 and word[1] == "一" and word[0] == word[-1]:
|
||||
finals[1] = finals[1][:-1] + "5"
|
||||
# when "一" is ordinal word, it should be yi1
|
||||
elif word.startswith("第一"):
|
||||
finals[1] = finals[1][:-1] + "1"
|
||||
else:
|
||||
for i, char in enumerate(word):
|
||||
if char == "一" and i + 1 < len(word):
|
||||
# "一" before tone4 should be yi2, e.g. 一段
|
||||
if finals[i + 1][-1] == "4":
|
||||
finals[i] = finals[i][:-1] + "2"
|
||||
# "一" before non-tone4 should be yi4, e.g. 一天
|
||||
else:
|
||||
# "一" 后面如果是标点，还读一声
|
||||
if word[i + 1] not in self.punc:
|
||||
finals[i] = finals[i][:-1] + "4"
|
||||
return finals
|
||||
|
||||
def _split_word(self, word: str) -> List[str]:
|
||||
word_list = jieba.cut_for_search(word)
|
||||
word_list = sorted(word_list, key=lambda i: len(i), reverse=False)
|
||||
first_subword = word_list[0]
|
||||
first_begin_idx = word.find(first_subword)
|
||||
if first_begin_idx == 0:
|
||||
second_subword = word[len(first_subword) :]
|
||||
new_word_list = [first_subword, second_subword]
|
||||
else:
|
||||
second_subword = word[: -len(first_subword)]
|
||||
new_word_list = [second_subword, first_subword]
|
||||
return new_word_list
|
||||
|
||||
def _three_sandhi(self, word: str, finals: List[str]) -> List[str]:
|
||||
if len(word) == 2 and self._all_tone_three(finals):
|
||||
finals[0] = finals[0][:-1] + "2"
|
||||
elif len(word) == 3:
|
||||
word_list = self._split_word(word)
|
||||
if self._all_tone_three(finals):
|
||||
# disyllabic + monosyllabic, e.g. 蒙古/包
|
||||
if len(word_list[0]) == 2:
|
||||
finals[0] = finals[0][:-1] + "2"
|
||||
finals[1] = finals[1][:-1] + "2"
|
||||
# monosyllabic + disyllabic, e.g. 纸/老虎
|
||||
elif len(word_list[0]) == 1:
|
||||
finals[1] = finals[1][:-1] + "2"
|
||||
else:
|
||||
finals_list = [finals[: len(word_list[0])], finals[len(word_list[0]) :]]
|
||||
if len(finals_list) == 2:
|
||||
for i, sub in enumerate(finals_list):
|
||||
# e.g. 所有/人
|
||||
if self._all_tone_three(sub) and len(sub) == 2:
|
||||
finals_list[i][0] = finals_list[i][0][:-1] + "2"
|
||||
# e.g. 好/喜欢
|
||||
elif (
|
||||
i == 1
|
||||
and not self._all_tone_three(sub)
|
||||
and finals_list[i][0][-1] == "3"
|
||||
and finals_list[0][-1][-1] == "3"
|
||||
):
|
||||
finals_list[0][-1] = finals_list[0][-1][:-1] + "2"
|
||||
finals = sum(finals_list, [])
|
||||
# split idiom into two words who's length is 2
|
||||
elif len(word) == 4:
|
||||
finals_list = [finals[:2], finals[2:]]
|
||||
finals = []
|
||||
for sub in finals_list:
|
||||
if self._all_tone_three(sub):
|
||||
sub[0] = sub[0][:-1] + "2"
|
||||
finals += sub
|
||||
|
||||
return finals
|
||||
|
||||
def _all_tone_three(self, finals: List[str]) -> bool:
|
||||
return all(x[-1] == "3" for x in finals)
|
||||
|
||||
# merge "不" and the word behind it
|
||||
# if don't merge, "不" sometimes appears alone according to jieba, which may occur sandhi error
|
||||
def _merge_bu(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
last_word = ""
|
||||
for word, pos in seg:
|
||||
if last_word == "不":
|
||||
word = last_word + word
|
||||
if word != "不":
|
||||
new_seg.append((word, pos))
|
||||
last_word = word[:]
|
||||
if last_word == "不":
|
||||
new_seg.append((last_word, "d"))
|
||||
last_word = ""
|
||||
return new_seg
|
||||
|
||||
# function 1: merge "一" and reduplication words in it's left and right, e.g. "听","一","听" ->"听一听"
|
||||
# function 2: merge single "一" and the word behind it
|
||||
# if don't merge, "一" sometimes appears alone according to jieba, which may occur sandhi error
|
||||
# e.g.
|
||||
# input seg: [('听', 'v'), ('一', 'm'), ('听', 'v')]
|
||||
# output seg: [['听一听', 'v']]
|
||||
def _merge_yi(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
# function 1
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if (
|
||||
i - 1 >= 0
|
||||
and word == "一"
|
||||
and i + 1 < len(seg)
|
||||
and seg[i - 1][0] == seg[i + 1][0]
|
||||
and seg[i - 1][1] == "v"
|
||||
):
|
||||
new_seg[i - 1][0] = new_seg[i - 1][0] + "一" + new_seg[i - 1][0]
|
||||
else:
|
||||
if (
|
||||
i - 2 >= 0
|
||||
and seg[i - 1][0] == "一"
|
||||
and seg[i - 2][0] == word
|
||||
and pos == "v"
|
||||
):
|
||||
continue
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
seg = new_seg
|
||||
new_seg = []
|
||||
# function 2
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if new_seg and new_seg[-1][0] == "一":
|
||||
new_seg[-1][0] = new_seg[-1][0] + word
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
return new_seg
|
||||
|
||||
# the first and the second words are all_tone_three
|
||||
def _merge_continuous_three_tones(
|
||||
self, seg: List[Tuple[str, str]]
|
||||
) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
sub_finals_list = [
|
||||
lazy_pinyin(word, neutral_tone_with_five=True, style=Style.FINALS_TONE3)
|
||||
for (word, pos) in seg
|
||||
]
|
||||
assert len(sub_finals_list) == len(seg)
|
||||
merge_last = [False] * len(seg)
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if (
|
||||
i - 1 >= 0
|
||||
and self._all_tone_three(sub_finals_list[i - 1])
|
||||
and self._all_tone_three(sub_finals_list[i])
|
||||
and not merge_last[i - 1]
|
||||
):
|
||||
# if the last word is reduplication, not merge, because reduplication need to be _neural_sandhi
|
||||
if (
|
||||
not self._is_reduplication(seg[i - 1][0])
|
||||
and len(seg[i - 1][0]) + len(seg[i][0]) <= 3
|
||||
):
|
||||
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
|
||||
merge_last[i] = True
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
|
||||
return new_seg
|
||||
|
||||
def _is_reduplication(self, word: str) -> bool:
|
||||
return len(word) == 2 and word[0] == word[1]
|
||||
|
||||
# the last char of first word and the first char of second word is tone_three
|
||||
def _merge_continuous_three_tones_2(
|
||||
self, seg: List[Tuple[str, str]]
|
||||
) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
sub_finals_list = [
|
||||
lazy_pinyin(word, neutral_tone_with_five=True, style=Style.FINALS_TONE3)
|
||||
for (word, pos) in seg
|
||||
]
|
||||
assert len(sub_finals_list) == len(seg)
|
||||
merge_last = [False] * len(seg)
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if (
|
||||
i - 1 >= 0
|
||||
and sub_finals_list[i - 1][-1][-1] == "3"
|
||||
and sub_finals_list[i][0][-1] == "3"
|
||||
and not merge_last[i - 1]
|
||||
):
|
||||
# if the last word is reduplication, not merge, because reduplication need to be _neural_sandhi
|
||||
if (
|
||||
not self._is_reduplication(seg[i - 1][0])
|
||||
and len(seg[i - 1][0]) + len(seg[i][0]) <= 3
|
||||
):
|
||||
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
|
||||
merge_last[i] = True
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
return new_seg
|
||||
|
||||
def _merge_er(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if i - 1 >= 0 and word == "儿" and seg[i - 1][0] != "#":
|
||||
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
return new_seg
|
||||
|
||||
def _merge_reduplication(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
new_seg = []
|
||||
for i, (word, pos) in enumerate(seg):
|
||||
if new_seg and word == new_seg[-1][0]:
|
||||
new_seg[-1][0] = new_seg[-1][0] + seg[i][0]
|
||||
else:
|
||||
new_seg.append([word, pos])
|
||||
return new_seg
|
||||
|
||||
def pre_merge_for_modify(self, seg: List[Tuple[str, str]]) -> List[Tuple[str, str]]:
|
||||
seg = self._merge_bu(seg)
|
||||
try:
|
||||
seg = self._merge_yi(seg)
|
||||
except:
|
||||
print("_merge_yi failed")
|
||||
seg = self._merge_reduplication(seg)
|
||||
seg = self._merge_continuous_three_tones(seg)
|
||||
seg = self._merge_continuous_three_tones_2(seg)
|
||||
seg = self._merge_er(seg)
|
||||
return seg
|
||||
|
||||
def modified_tone(self, word: str, pos: str, finals: List[str]) -> List[str]:
|
||||
finals = self._bu_sandhi(word, finals)
|
||||
finals = self._yi_sandhi(word, finals)
|
||||
finals = self._neural_sandhi(word, pos, finals)
|
||||
finals = self._three_sandhi(word, finals)
|
||||
return finals
|
||||
3
oldVersion/__init__.py
Normal file
3
oldVersion/__init__.py
Normal file
@@ -0,0 +1,3 @@
|
||||
"""
|
||||
老版本模型推理兼容
|
||||
"""
|
||||
@@ -1,32 +1,37 @@
|
||||
import json
|
||||
import os.path
|
||||
from collections import defaultdict
|
||||
from random import shuffle
|
||||
from typing import Optional
|
||||
import os
|
||||
|
||||
from tqdm import tqdm
|
||||
import click
|
||||
from text.cleaner import clean_text
|
||||
from config import config
|
||||
from infer import latest_version
|
||||
|
||||
preprocess_text_config = config.preprocess_text_config
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--transcription-path",
|
||||
default="filelists/genshin.list",
|
||||
default=preprocess_text_config.transcription_path,
|
||||
type=click.Path(exists=True, file_okay=True, dir_okay=False),
|
||||
)
|
||||
@click.option("--cleaned-path", default=None)
|
||||
@click.option("--train-path", default="filelists/train.list")
|
||||
@click.option("--val-path", default="filelists/val.list")
|
||||
@click.option("--cleaned-path", default=preprocess_text_config.cleaned_path)
|
||||
@click.option("--train-path", default=preprocess_text_config.train_path)
|
||||
@click.option("--val-path", default=preprocess_text_config.val_path)
|
||||
@click.option(
|
||||
"--config-path",
|
||||
default="configs/config.json",
|
||||
default=preprocess_text_config.config_path,
|
||||
type=click.Path(exists=True, file_okay=True, dir_okay=False),
|
||||
)
|
||||
@click.option("--val-per-spk", default=4)
|
||||
@click.option("--max-val-total", default=8)
|
||||
@click.option("--clean/--no-clean", default=True)
|
||||
def main(
|
||||
@click.option("--val-per-spk", default=preprocess_text_config.val_per_spk)
|
||||
@click.option("--max-val-total", default=preprocess_text_config.max_val_total)
|
||||
@click.option("--clean/--no-clean", default=preprocess_text_config.clean)
|
||||
@click.option("-y", "--yml_config")
|
||||
def preprocess(
|
||||
transcription_path: str,
|
||||
cleaned_path: Optional[str],
|
||||
train_path: str,
|
||||
@@ -35,39 +40,43 @@ def main(
|
||||
val_per_spk: int,
|
||||
max_val_total: int,
|
||||
clean: bool,
|
||||
yml_config: str, # 这个不要删
|
||||
):
|
||||
if cleaned_path is None:
|
||||
if cleaned_path == "" or cleaned_path is None:
|
||||
cleaned_path = transcription_path + ".cleaned"
|
||||
|
||||
if clean:
|
||||
out_file = open(cleaned_path, "w", encoding="utf-8")
|
||||
for line in tqdm(open(transcription_path, encoding="utf-8").readlines()):
|
||||
try:
|
||||
utt, spk, language, text = line.strip().split("|")
|
||||
norm_text, phones, tones, word2ph = clean_text(text, language)
|
||||
out_file.write(
|
||||
"{}|{}|{}|{}|{}|{}|{}\n".format(
|
||||
utt,
|
||||
spk,
|
||||
language,
|
||||
norm_text,
|
||||
" ".join(phones),
|
||||
" ".join([str(i) for i in tones]),
|
||||
" ".join([str(i) for i in word2ph]),
|
||||
)
|
||||
)
|
||||
except Exception as error:
|
||||
print("err!", line, error)
|
||||
|
||||
out_file.close()
|
||||
|
||||
transcription_path = cleaned_path
|
||||
with open(cleaned_path, "w", encoding="utf-8") as out_file:
|
||||
with open(transcription_path, "r", encoding="utf-8") as trans_file:
|
||||
lines = trans_file.readlines()
|
||||
# print(lines, ' ', len(lines))
|
||||
if len(lines) != 0:
|
||||
for line in tqdm(lines):
|
||||
try:
|
||||
utt, spk, language, text = line.strip().split("|")
|
||||
norm_text, phones, tones, word2ph = clean_text(
|
||||
text, language
|
||||
)
|
||||
out_file.write(
|
||||
"{}|{}|{}|{}|{}|{}|{}\n".format(
|
||||
utt,
|
||||
spk,
|
||||
language,
|
||||
norm_text,
|
||||
" ".join(phones),
|
||||
" ".join([str(i) for i in tones]),
|
||||
" ".join([str(i) for i in word2ph]),
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
print(f"生成训练集和验证集时发生错误！, 详细信息:\n{e}")
|
||||
|
||||
transcription_path = cleaned_path
|
||||
spk_utt_map = defaultdict(list)
|
||||
spk_id_map = {}
|
||||
current_sid = 0
|
||||
|
||||
with open(transcription_path, encoding="utf-8") as f:
|
||||
with open(transcription_path, "r", encoding="utf-8") as f:
|
||||
audioPaths = set()
|
||||
countSame = 0
|
||||
countNotFound = 0
|
||||
@@ -79,6 +88,7 @@ def main(
|
||||
countSame += 1
|
||||
continue
|
||||
if not os.path.isfile(utt):
|
||||
# 过滤数据集错误：不存在对应音频
|
||||
print(f"没有找到对应的音频：{utt}")
|
||||
countNotFound += 1
|
||||
continue
|
||||
@@ -110,11 +120,20 @@ def main(
|
||||
for line in val_list:
|
||||
f.write(line)
|
||||
|
||||
config = json.load(open(config_path, encoding="utf-8"))
|
||||
config["data"]["spk2id"] = spk_id_map
|
||||
json_config = json.load(open(config_path, encoding="utf-8"))
|
||||
json_config["data"]["spk2id"] = spk_id_map
|
||||
# 新增写入：写入训练版本、数据集路径
|
||||
json_config["version"] = latest_version
|
||||
json_config["data"]["training_files"] = os.path.normpath(
|
||||
preprocess_text_config.train_path
|
||||
).replace("\\", "/")
|
||||
json_config["data"]["validation_files"] = os.path.normpath(
|
||||
preprocess_text_config.val_path
|
||||
).replace("\\", "/")
|
||||
with open(config_path, "w", encoding="utf-8") as f:
|
||||
json.dump(config, f, indent=2, ensure_ascii=False)
|
||||
json.dump(json_config, f, indent=2, ensure_ascii=False)
|
||||
print("训练集和验证集生成完成！")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
preprocess()
|
||||
|
||||
82
re_matching.py
Normal file
82
re_matching.py
Normal file
@@ -0,0 +1,82 @@
|
||||
import re
|
||||
|
||||
|
||||
def extract_language_and_text_updated(speaker, dialogue):
|
||||
# 使用正则表达式匹配<语言>标签和其后的文本
|
||||
pattern_language_text = r"<(\S+?)>([^<]+)"
|
||||
matches = re.findall(pattern_language_text, dialogue, re.DOTALL)
|
||||
speaker = speaker[1:-1]
|
||||
# 清理文本：去除两边的空白字符
|
||||
matches_cleaned = [(lang.upper(), text.strip()) for lang, text in matches]
|
||||
matches_cleaned.append(speaker)
|
||||
return matches_cleaned
|
||||
|
||||
|
||||
def validate_text(input_text):
|
||||
# 验证说话人的正则表达式
|
||||
pattern_speaker = r"(\[\S+?\])((?:\s*<\S+?>[^<\[\]]+?)+)"
|
||||
|
||||
# 使用re.DOTALL标志使.匹配包括换行符在内的所有字符
|
||||
matches = re.findall(pattern_speaker, input_text, re.DOTALL)
|
||||
|
||||
# 对每个匹配到的说话人内容进行进一步验证
|
||||
for _, dialogue in matches:
|
||||
language_text_matches = extract_language_and_text_updated(_, dialogue)
|
||||
if not language_text_matches:
|
||||
return (
|
||||
False,
|
||||
"Error: Invalid format detected in dialogue content. Please check your input.",
|
||||
)
|
||||
|
||||
# 如果输入的文本中没有找到任何匹配项
|
||||
if not matches:
|
||||
return (
|
||||
False,
|
||||
"Error: No valid speaker format detected. Please check your input.",
|
||||
)
|
||||
|
||||
return True, "Input is valid."
|
||||
|
||||
|
||||
def text_matching(text: str) -> list:
|
||||
speaker_pattern = r"(\[\S+?\])(.+?)(?=\[\S+?\]|$)"
|
||||
matches = re.findall(speaker_pattern, text, re.DOTALL)
|
||||
result = []
|
||||
for speaker, dialogue in matches:
|
||||
result.append(extract_language_and_text_updated(speaker, dialogue))
|
||||
print(result)
|
||||
return result
|
||||
|
||||
|
||||
def cut_para(text):
|
||||
splitted_para = re.split("[\n]", text) # 按段分
|
||||
splitted_para = [
|
||||
sentence.strip() for sentence in splitted_para if sentence.strip()
|
||||
] # 删除空字符串
|
||||
return splitted_para
|
||||
|
||||
|
||||
def cut_sent(para):
|
||||
para = re.sub("([。！;？\?])([^”’])", r"\1\n\2", para) # 单字符断句符
|
||||
para = re.sub("(\.{6})([^”’])", r"\1\n\2", para) # 英文省略号
|
||||
para = re.sub("(\…{2})([^”’])", r"\1\n\2", para) # 中文省略号
|
||||
para = re.sub("([。！？\?][”’])([^，。！？\?])", r"\1\n\2", para)
|
||||
para = para.rstrip() # 段尾如果有多余的\n就去掉它
|
||||
return para.split("\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
text = """
|
||||
[说话人1]
|
||||
[说话人2]<zh>你好吗？<jp>元気ですか？<jp>こんにちは，世界。<zh>你好吗？
|
||||
[说话人3]<zh>谢谢。<jp>どういたしまして。
|
||||
"""
|
||||
text_matching(text)
|
||||
# 测试函数
|
||||
test_text = """
|
||||
[说话人1]<zh>你好，こんにちは！<jp>こんにちは，世界。
|
||||
[说话人2]<zh>你好吗？
|
||||
"""
|
||||
text_matching(test_text)
|
||||
res = validate_text(test_text)
|
||||
print(res)
|
||||
@@ -5,8 +5,6 @@ numba
|
||||
phonemizer
|
||||
scipy
|
||||
tensorboard
|
||||
torch
|
||||
torchvision
|
||||
Unidecode
|
||||
amfm_decompy
|
||||
jieba
|
||||
@@ -21,3 +19,13 @@ unidic-lite
|
||||
cmudict
|
||||
fugashi
|
||||
num2words
|
||||
PyYAML
|
||||
requests
|
||||
pyopenjtalk; sys_platform == 'linux'
|
||||
openjtalk; sys_platform != 'linux'
|
||||
jaconv
|
||||
psutil
|
||||
GPUtil
|
||||
vector_quantize_pytorch
|
||||
g2p_en
|
||||
sentencepiece
|
||||
|
||||
77
resample.py
77
resample.py
@@ -6,43 +6,70 @@ from multiprocessing import Pool, cpu_count
|
||||
import soundfile
|
||||
from tqdm import tqdm
|
||||
|
||||
from config import config
|
||||
|
||||
|
||||
def process(item):
|
||||
spkdir, wav_name, args = item
|
||||
speaker = spkdir.replace("\\", "/").split("/")[-1]
|
||||
wav_path = os.path.join(args.in_dir, speaker, wav_name)
|
||||
wav_path = os.path.join(args.in_dir, spkdir, wav_name)
|
||||
if os.path.exists(wav_path) and ".wav" in wav_path:
|
||||
os.makedirs(os.path.join(args.out_dir, speaker), exist_ok=True)
|
||||
wav, sr = librosa.load(wav_path, sr=args.sr)
|
||||
soundfile.write(os.path.join(args.out_dir, speaker, wav_name), wav, sr)
|
||||
soundfile.write(os.path.join(args.out_dir, spkdir, wav_name), wav, sr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--sr", type=int, default=44100, help="sampling rate")
|
||||
parser.add_argument(
|
||||
"--in_dir", type=str, default="./raw", help="path to source dir"
|
||||
"--sr",
|
||||
type=int,
|
||||
default=config.resample_config.sampling_rate,
|
||||
help="sampling rate",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--out_dir", type=str, default="./dataset", help="path to target dir"
|
||||
"--in_dir",
|
||||
type=str,
|
||||
default=config.resample_config.in_dir,
|
||||
help="path to source dir",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
# processes = 8
|
||||
processes = cpu_count() - 2 if cpu_count() > 4 else 1
|
||||
parser.add_argument(
|
||||
"--out_dir",
|
||||
type=str,
|
||||
default=config.resample_config.out_dir,
|
||||
help="path to target dir",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--processes",
|
||||
type=int,
|
||||
default=0,
|
||||
help="cpu_processes",
|
||||
)
|
||||
args, _ = parser.parse_known_args()
|
||||
# autodl 无卡模式会识别出46个cpu
|
||||
if args.processes == 0:
|
||||
processes = cpu_count() - 2 if cpu_count() > 4 else 1
|
||||
else:
|
||||
processes = args.processes
|
||||
pool = Pool(processes=processes)
|
||||
|
||||
for speaker in os.listdir(args.in_dir):
|
||||
spk_dir = os.path.join(args.in_dir, speaker)
|
||||
if os.path.isdir(spk_dir):
|
||||
print(spk_dir)
|
||||
for _ in tqdm(
|
||||
pool.imap_unordered(
|
||||
process,
|
||||
[
|
||||
(spk_dir, i, args)
|
||||
for i in os.listdir(spk_dir)
|
||||
if i.endswith("wav")
|
||||
],
|
||||
)
|
||||
):
|
||||
pass
|
||||
tasks = []
|
||||
|
||||
for dirpath, _, filenames in os.walk(args.in_dir):
|
||||
# 子级目录
|
||||
spk_dir = os.path.relpath(dirpath, args.in_dir)
|
||||
spk_dir_out = os.path.join(args.out_dir, spk_dir)
|
||||
if not os.path.isdir(spk_dir_out):
|
||||
os.makedirs(spk_dir_out, exist_ok=True)
|
||||
for filename in filenames:
|
||||
if filename.endswith(".wav"):
|
||||
twople = (spk_dir, filename, args)
|
||||
tasks.append(twople)
|
||||
|
||||
for _ in tqdm(
|
||||
pool.imap_unordered(process, tasks),
|
||||
):
|
||||
pass
|
||||
|
||||
pool.close()
|
||||
pool.join()
|
||||
|
||||
print("音频重采样完毕!")
|
||||
|
||||
133
server.py
133
server.py
@@ -2,84 +2,19 @@ from flask import Flask, request, Response
|
||||
from io import BytesIO
|
||||
import torch
|
||||
from av import open as avopen
|
||||
from typing import Dict, List
|
||||
|
||||
import commons
|
||||
import utils
|
||||
from models import SynthesizerTrn
|
||||
from text.symbols import symbols
|
||||
from text import cleaned_text_to_sequence, get_bert
|
||||
from text.cleaner import clean_text
|
||||
from infer import infer, get_net_g, latest_version
|
||||
from scipy.io import wavfile
|
||||
|
||||
from config import config
|
||||
|
||||
# Flask Init
|
||||
app = Flask(__name__)
|
||||
app.config["JSON_AS_ASCII"] = False
|
||||
|
||||
|
||||
def get_text(text, language_str, hps):
|
||||
norm_text, phone, tone, word2ph = clean_text(text, language_str)
|
||||
phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
|
||||
|
||||
if hps.data.add_blank:
|
||||
phone = commons.intersperse(phone, 0)
|
||||
tone = commons.intersperse(tone, 0)
|
||||
language = commons.intersperse(language, 0)
|
||||
for i in range(len(word2ph)):
|
||||
word2ph[i] = word2ph[i] * 2
|
||||
word2ph[0] += 1
|
||||
bert = get_bert(norm_text, word2ph, language_str)
|
||||
del word2ph
|
||||
assert bert.shape[-1] == len(phone), phone
|
||||
|
||||
if language_str == "ZH":
|
||||
bert = bert
|
||||
ja_bert = torch.zeros(768, len(phone))
|
||||
elif language_str == "JA":
|
||||
ja_bert = bert
|
||||
bert = torch.zeros(1024, len(phone))
|
||||
else:
|
||||
bert = torch.zeros(1024, len(phone))
|
||||
ja_bert = torch.zeros(768, len(phone))
|
||||
assert bert.shape[-1] == len(
|
||||
phone
|
||||
), f"Bert seq len {bert.shape[-1]} != {len(phone)}"
|
||||
phone = torch.LongTensor(phone)
|
||||
tone = torch.LongTensor(tone)
|
||||
language = torch.LongTensor(language)
|
||||
return bert, ja_bert, phone, tone, language
|
||||
|
||||
|
||||
def infer(text, sdp_ratio, noise_scale, noise_scale_w, length_scale, sid, language):
|
||||
bert, ja_bert, phones, tones, lang_ids = get_text(text, language, hps)
|
||||
with torch.no_grad():
|
||||
x_tst = phones.to(dev).unsqueeze(0)
|
||||
tones = tones.to(dev).unsqueeze(0)
|
||||
lang_ids = lang_ids.to(dev).unsqueeze(0)
|
||||
bert = bert.to(dev).unsqueeze(0)
|
||||
ja_bert = ja_bert.to(device).unsqueeze(0)
|
||||
x_tst_lengths = torch.LongTensor([phones.size(0)]).to(dev)
|
||||
speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(dev)
|
||||
audio = (
|
||||
net_g.infer(
|
||||
x_tst,
|
||||
x_tst_lengths,
|
||||
speakers,
|
||||
tones,
|
||||
lang_ids,
|
||||
bert,
|
||||
ja_bert,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise_scale,
|
||||
noise_scale_w=noise_scale_w,
|
||||
length_scale=length_scale,
|
||||
)[0][0, 0]
|
||||
.data.cpu()
|
||||
.float()
|
||||
.numpy()
|
||||
)
|
||||
return audio
|
||||
|
||||
|
||||
def replace_punctuation(text, i=2):
|
||||
punctuation = "，。？！"
|
||||
for char in punctuation:
|
||||
@@ -106,26 +41,39 @@ def wav2(i, o, format):
|
||||
inp.close()
|
||||
|
||||
|
||||
# Load Generator
|
||||
hps = utils.get_hparams_from_file("./configs/config.json")
|
||||
net_g_List = []
|
||||
hps_List = []
|
||||
# 模型角色字典
|
||||
# 使用方法 chr_name = chrsMap[model_id][chr_id]
|
||||
chrsMap: List[Dict[int, str]] = list()
|
||||
|
||||
dev = "cuda"
|
||||
net_g = SynthesizerTrn(
|
||||
len(symbols),
|
||||
hps.data.filter_length // 2 + 1,
|
||||
hps.train.segment_size // hps.data.hop_length,
|
||||
n_speakers=hps.data.n_speakers,
|
||||
**hps.model,
|
||||
).to(dev)
|
||||
_ = net_g.eval()
|
||||
|
||||
_ = utils.load_checkpoint("logs/G_649000.pth", net_g, None, skip_optimizer=True)
|
||||
# 加载模型
|
||||
models = config.server_config.models
|
||||
for model in models:
|
||||
hps_List.append(utils.get_hparams_from_file(model["config"]))
|
||||
# 添加角色字典
|
||||
chrsMap.append(dict())
|
||||
for name, cid in hps_List[-1].data.spk2id.items():
|
||||
chrsMap[-1][cid] = name
|
||||
version = (
|
||||
hps_List[-1].version if hasattr(hps_List[-1], "version") else latest_version
|
||||
)
|
||||
net_g_List.append(
|
||||
get_net_g(
|
||||
model_path=model["model"],
|
||||
version=version,
|
||||
device=model["device"],
|
||||
hps=hps_List[-1],
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@app.route("/")
|
||||
def main():
|
||||
try:
|
||||
speaker = request.args.get("speaker")
|
||||
model = int(request.args.get("model"))
|
||||
speaker = request.args.get("speaker", "") # 指定人物名
|
||||
speaker_id = request.args.get("speaker_id", None) # 直接指定id
|
||||
text = request.args.get("text").replace("/n", "")
|
||||
sdp_ratio = float(request.args.get("sdp_ratio", 0.2))
|
||||
noise = float(request.args.get("noise", 0.5))
|
||||
@@ -141,24 +89,31 @@ def main():
|
||||
return "Missing Parameter"
|
||||
if fmt not in ("mp3", "wav", "ogg"):
|
||||
return "Invalid Format"
|
||||
if language not in ("JA", "ZH"):
|
||||
if language not in ("JP", "ZH"):
|
||||
return "Invalid language"
|
||||
except:
|
||||
return "Invalid Parameter"
|
||||
|
||||
if speaker_id is not None:
|
||||
if speaker_id.isdigit():
|
||||
speaker = chrsMap[model][int(speaker_id)]
|
||||
|
||||
with torch.no_grad():
|
||||
audio = infer(
|
||||
text,
|
||||
text=text,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise,
|
||||
noise_scale_w=noisew,
|
||||
length_scale=length,
|
||||
sid=speaker,
|
||||
language=language,
|
||||
language=models[model]["language"],
|
||||
hps=hps_List[model],
|
||||
net_g=net_g_List[model],
|
||||
device=models[model]["device"],
|
||||
)
|
||||
|
||||
with BytesIO() as wav:
|
||||
wavfile.write(wav, hps.data.sampling_rate, audio)
|
||||
wavfile.write(wav, hps_List[model].data.sampling_rate, audio)
|
||||
torch.cuda.empty_cache()
|
||||
if fmt == "wav":
|
||||
return Response(wav.getvalue(), mimetype="audio/wav")
|
||||
@@ -168,3 +123,7 @@ def main():
|
||||
return Response(
|
||||
ofp.getvalue(), mimetype="audio/mpeg" if fmt == "mp3" else "audio/ogg"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
app.run(port=config.server_config.port)
|
||||
|
||||
376
server_fastapi.py
Normal file
376
server_fastapi.py
Normal file
@@ -0,0 +1,376 @@
|
||||
"""
|
||||
api服务 多版本多模型 fastapi实现
|
||||
"""
|
||||
import logging
|
||||
|
||||
import utils
|
||||
from fastapi import FastAPI, Query
|
||||
from fastapi.responses import Response, FileResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from io import BytesIO
|
||||
from scipy.io import wavfile
|
||||
import uvicorn
|
||||
import torch
|
||||
import webbrowser
|
||||
import psutil
|
||||
import GPUtil
|
||||
from typing import Dict, Optional, List
|
||||
import os
|
||||
from loguru import logger
|
||||
|
||||
from infer import infer, get_net_g, latest_version
|
||||
import tools.translate as trans
|
||||
|
||||
from config import config
|
||||
|
||||
|
||||
class Model:
|
||||
"""模型封装类"""
|
||||
|
||||
def __init__(self, config_path: str, model_path: str, device: str, language: str):
|
||||
self.config_path: str = os.path.normpath(config_path)
|
||||
self.model_path: str = os.path.normpath(model_path)
|
||||
self.device: str = device
|
||||
self.language: str = language
|
||||
self.hps = utils.get_hparams_from_file(config_path)
|
||||
self.spk2id: Dict[str, int] = self.hps.data.spk2id # spk - id 映射字典
|
||||
self.id2spk: Dict[int, str] = dict() # id - spk 映射字典
|
||||
for speaker, speaker_id in self.hps.data.spk2id.items():
|
||||
self.id2spk[speaker_id] = speaker
|
||||
self.version: str = (
|
||||
self.hps.version if hasattr(self.hps, "version") else latest_version
|
||||
)
|
||||
self.net_g = get_net_g(
|
||||
model_path=model_path,
|
||||
version=self.version,
|
||||
device=device,
|
||||
hps=self.hps,
|
||||
)
|
||||
|
||||
def to_dict(self) -> Dict[str, any]:
|
||||
return {
|
||||
"config_path": self.config_path,
|
||||
"model_path": self.model_path,
|
||||
"device": self.device,
|
||||
"language": self.language,
|
||||
"spk2id": self.spk2id,
|
||||
"id2spk": self.id2spk,
|
||||
"version": self.version,
|
||||
}
|
||||
|
||||
|
||||
class Models:
|
||||
def __init__(self):
|
||||
self.models: Dict[int, Model] = dict()
|
||||
self.num = 0
|
||||
# spkInfo[角色名][模型id] = 角色id
|
||||
self.spk_info: Dict[str, Dict[int, int]] = dict()
|
||||
self.paths: Dict[str, int] = dict() # 路径, 引用数
|
||||
|
||||
def add_model(self, model: Model):
|
||||
"""添加一个模型"""
|
||||
self.models[self.num] = model
|
||||
# 添加角色信息
|
||||
for speaker, speaker_id in model.spk2id.items():
|
||||
if speaker not in self.spk_info.keys():
|
||||
self.spk_info[speaker] = {self.num: speaker_id}
|
||||
else:
|
||||
self.spk_info[speaker][self.num] = speaker_id
|
||||
# 添加路径信息
|
||||
model_path = os.path.realpath(model.model_path)
|
||||
if model_path not in self.paths.keys():
|
||||
self.paths[model_path] = 1
|
||||
else:
|
||||
self.paths[model_path] += 1
|
||||
# 修改计数
|
||||
self.num += 1
|
||||
|
||||
def init_model(
|
||||
self, config_path: str, model_path: str, device: str, language: str
|
||||
) -> int:
|
||||
"""
|
||||
初始化并添加一个模型
|
||||
|
||||
:param config_path: 模型config.json路径
|
||||
:param model_path: 模型路径
|
||||
:param device: 模型推理使用设备
|
||||
:param language: 模型推理默认语言
|
||||
"""
|
||||
self.models[self.num] = Model(
|
||||
config_path=config_path,
|
||||
model_path=model_path,
|
||||
device=device,
|
||||
language=language,
|
||||
)
|
||||
# 添加角色信息
|
||||
for speaker, speaker_id in self.models[self.num].spk2id.items():
|
||||
if speaker not in self.spk_info.keys():
|
||||
self.spk_info[speaker] = {self.num: speaker_id}
|
||||
else:
|
||||
self.spk_info[speaker][self.num] = speaker_id
|
||||
# 添加路径信息
|
||||
model_path = os.path.realpath(self.models[self.num].model_path)
|
||||
if model_path not in self.paths.keys():
|
||||
self.paths[model_path] = 1
|
||||
else:
|
||||
self.paths[model_path] += 1
|
||||
# 修改计数
|
||||
logger.success(f"添加模型{model_path}，使用配置文件{os.path.realpath(config_path)}")
|
||||
self.num += 1
|
||||
return self.num - 1
|
||||
|
||||
def del_model(self, index: int) -> Optional[int]:
|
||||
"""删除对应序号的模型，若不存在则返回None"""
|
||||
if index not in self.models.keys():
|
||||
return None
|
||||
# 删除角色信息
|
||||
for speaker, speaker_id in self.models[index].spk2id.items():
|
||||
self.spk_info[speaker].pop(index)
|
||||
if len(self.spk_info[speaker]) == 0:
|
||||
# 若对应角色的所有模型都被删除，则清除该角色信息
|
||||
self.spk_info.pop(speaker)
|
||||
# 删除路径信息
|
||||
model_path = os.path.realpath(self.models[index].model_path)
|
||||
self.paths[model_path] -= 1
|
||||
assert self.paths[model_path] >= 0
|
||||
if self.paths[model_path] == 0:
|
||||
# 引用数为零时予以清空
|
||||
self.paths.pop(model_path)
|
||||
# 删除模型
|
||||
logger.success(f"卸载模型{model_path}, id = {index}")
|
||||
self.models.pop(index)
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
return index
|
||||
|
||||
def get_models(self):
|
||||
"""获取所有模型"""
|
||||
return self.models
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
app = FastAPI()
|
||||
app.logger = logger
|
||||
# 挂载静态文件
|
||||
StaticDir: str = "./Web"
|
||||
dirs = [fir.name for fir in os.scandir(StaticDir) if fir.is_dir()]
|
||||
files = [fir.name for fir in os.scandir(StaticDir) if fir.is_dir()]
|
||||
for dirName in dirs:
|
||||
app.mount(
|
||||
f"/{dirName}",
|
||||
StaticFiles(directory=f"./{StaticDir}/{dirName}"),
|
||||
name=dirName,
|
||||
)
|
||||
loaded_models = Models()
|
||||
# 加载模型
|
||||
models_info = config.server_config.models
|
||||
for model_info in models_info:
|
||||
loaded_models.init_model(
|
||||
config_path=model_info["config"],
|
||||
model_path=model_info["model"],
|
||||
device=model_info["device"],
|
||||
language=model_info["language"],
|
||||
)
|
||||
|
||||
@app.get("/")
|
||||
async def index():
|
||||
return FileResponse("./Web/index.html")
|
||||
|
||||
@app.get("/voice")
|
||||
def voice(
|
||||
text: str = Query(..., description="输入文字"),
|
||||
model_id: int = Query(..., description="模型ID"), # 模型序号
|
||||
speaker_name: str = Query(
|
||||
None, description="说话人名"
|
||||
), # speaker_name与 speaker_id二者选其一
|
||||
speaker_id: int = Query(None, description="说话人id，与speaker_name二选一"),
|
||||
sdp_ratio: float = Query(0.2, description="SDP/DP混合比"),
|
||||
noise: float = Query(0.2, description="感情"),
|
||||
noisew: float = Query(0.9, description="音素长度"),
|
||||
length: float = Query(1, description="语速"),
|
||||
language: str = Query(None, description="语言"), # 若不指定使用语言则使用默认值
|
||||
):
|
||||
"""语音接口"""
|
||||
|
||||
# 检查模型是否存在
|
||||
if model_id not in loaded_models.models.keys():
|
||||
return {"status": 10, "detail": f"模型model_id={model_id}未加载"}
|
||||
# 检查是否提供speaker
|
||||
if speaker_name is None and speaker_id is None:
|
||||
return {"status": 11, "detail": "请提供speaker_name或speaker_id"}
|
||||
elif speaker_name is None:
|
||||
# 检查speaker_id是否存在
|
||||
if speaker_id not in loaded_models.models[model_id].id2spk.keys():
|
||||
return {"status": 12, "detail": f"角色speaker_id={speaker_id}不存在"}
|
||||
speaker_name = loaded_models.models[model_id].id2spk[speaker_id]
|
||||
# 检查speaker_name是否存在
|
||||
if speaker_name not in loaded_models.models[model_id].spk2id.keys():
|
||||
return {"status": 13, "detail": f"角色speaker_name={speaker_name}不存在"}
|
||||
if language is None:
|
||||
language = loaded_models.models[model_id].language
|
||||
with torch.no_grad():
|
||||
audio = infer(
|
||||
text=text,
|
||||
sdp_ratio=sdp_ratio,
|
||||
noise_scale=noise,
|
||||
noise_scale_w=noisew,
|
||||
length_scale=length,
|
||||
sid=speaker_name,
|
||||
language=language,
|
||||
hps=loaded_models.models[model_id].hps,
|
||||
net_g=loaded_models.models[model_id].net_g,
|
||||
device=loaded_models.models[model_id].device,
|
||||
)
|
||||
wavContent = BytesIO()
|
||||
wavfile.write(
|
||||
wavContent, loaded_models.models[model_id].hps.data.sampling_rate, audio
|
||||
)
|
||||
response = Response(content=wavContent.getvalue(), media_type="audio/wav")
|
||||
return response
|
||||
|
||||
@app.get("/models/info")
|
||||
def get_loaded_models_info():
|
||||
"""获取已加载模型信息"""
|
||||
|
||||
result: Dict[str, Dict] = dict()
|
||||
for key, model in loaded_models.models.items():
|
||||
result[str(key)] = model.to_dict()
|
||||
return result
|
||||
|
||||
@app.get("/models/delete")
|
||||
def delete_model(model_id: int = Query(..., description="删除模型id")):
|
||||
"""删除指定模型"""
|
||||
|
||||
result = loaded_models.del_model(model_id)
|
||||
if result is None:
|
||||
return {"status": 14, "detail": f"模型{model_id}不存在，删除失败"}
|
||||
return {"status": 0, "detail": "删除成功"}
|
||||
|
||||
@app.get("/models/add")
|
||||
def add_model(
|
||||
model_path: str = Query(..., description="添加模型路径"),
|
||||
config_path: str = Query(
|
||||
None, description="添加模型配置文件路径，不填则使用./config.json或../config.json"
|
||||
),
|
||||
device: str = Query("cuda", description="推理使用设备"),
|
||||
language: str = Query("ZH", description="模型默认语言"),
|
||||
):
|
||||
"""添加指定模型：允许重复添加相同路径模型，注意，当前实现中模型会重复加载，加载两次占用两份内存"""
|
||||
if config_path is None:
|
||||
model_dir = os.path.dirname(model_path)
|
||||
if os.path.isfile(os.path.join(model_dir, "config.json")):
|
||||
config_path = os.path.join(model_dir, "config.json")
|
||||
elif os.path.isfile(os.path.join(model_dir, "../config.json")):
|
||||
config_path = os.path.join(model_dir, "../config.json")
|
||||
else:
|
||||
return {
|
||||
"status": 15,
|
||||
"detail": "查询未传入配置文件路径，同时默认路径./与../中不存在配置文件config.json。",
|
||||
}
|
||||
try:
|
||||
model_id = loaded_models.init_model(
|
||||
config_path=config_path,
|
||||
model_path=model_path,
|
||||
device=device,
|
||||
language=language,
|
||||
)
|
||||
except Exception:
|
||||
logging.exception("模型加载出错")
|
||||
return {
|
||||
"status": 16,
|
||||
"detail": "模型加载出错，详细查看日志",
|
||||
}
|
||||
return {
|
||||
"status": 0,
|
||||
"detail": "模型添加成功",
|
||||
"Data": {
|
||||
"model_id": model_id,
|
||||
"model_info": loaded_models.models[model_id].to_dict(),
|
||||
},
|
||||
}
|
||||
|
||||
def _get_all_models(root_dir: str = "Data", only_unloaded: bool = False):
|
||||
result: Dict[str, List[str]] = dict()
|
||||
files = os.listdir(root_dir)
|
||||
for file in files:
|
||||
if os.path.isdir(os.path.join(root_dir, file)):
|
||||
sub_dir = os.path.join(root_dir, file)
|
||||
# 搜索 "sub_dir" 、 "sub_dir/models" 两个路径
|
||||
result[file] = list()
|
||||
sub_files = os.listdir(sub_dir)
|
||||
for sub_file in sub_files:
|
||||
relpath = os.path.realpath(os.path.join(sub_dir, sub_file))
|
||||
if only_unloaded and relpath in loaded_models.paths.keys():
|
||||
continue
|
||||
if sub_file.endswith(".pth") and sub_file.startswith("G_"):
|
||||
if os.path.isfile(relpath):
|
||||
result[file].append(sub_file)
|
||||
models_dir = os.path.join(sub_dir, "models")
|
||||
if os.path.isdir(models_dir):
|
||||
sub_files = os.listdir(models_dir)
|
||||
for sub_file in sub_files:
|
||||
relpath = os.path.realpath(os.path.join(models_dir, sub_file))
|
||||
if only_unloaded and relpath in loaded_models.paths.keys():
|
||||
continue
|
||||
if sub_file.endswith(".pth") and sub_file.startswith("G_"):
|
||||
if os.path.isfile(os.path.join(models_dir, sub_file)):
|
||||
result[file].append(f"models/{sub_file}")
|
||||
if len(result[file]) == 0:
|
||||
result.pop(file)
|
||||
return result
|
||||
|
||||
@app.get("/models/get_unloaded")
|
||||
def get_unloaded_models_info(root_dir: str = "Data"):
|
||||
"""获取未加载模型"""
|
||||
return _get_all_models(root_dir, only_unloaded=True)
|
||||
|
||||
@app.get("/models/get_local")
|
||||
def get_local_models_info(root_dir: str = "Data"):
|
||||
"""获取全部本地模型"""
|
||||
return _get_all_models(root_dir, only_unloaded=False)
|
||||
|
||||
@app.get("/status")
|
||||
def get_status():
|
||||
"""获取电脑运行状态"""
|
||||
cpu_percent = psutil.cpu_percent(interval=1)
|
||||
memory_info = psutil.virtual_memory()
|
||||
memory_total = memory_info.total
|
||||
memory_available = memory_info.available
|
||||
memory_used = memory_info.used
|
||||
memory_percent = memory_info.percent
|
||||
gpuInfo = []
|
||||
devices = ["cpu"]
|
||||
for i in range(torch.cuda.device_count()):
|
||||
devices.append(f"cuda:{i}")
|
||||
gpus = GPUtil.getGPUs()
|
||||
for gpu in gpus:
|
||||
gpuInfo.append(
|
||||
{
|
||||
"gpu_id": gpu.id,
|
||||
"gpu_load": gpu.load,
|
||||
"gpu_memory": {
|
||||
"total": gpu.memoryTotal,
|
||||
"used": gpu.memoryUsed,
|
||||
"free": gpu.memoryFree,
|
||||
},
|
||||
}
|
||||
)
|
||||
return {
|
||||
"devices": devices,
|
||||
"cpu_percent": cpu_percent,
|
||||
"memory_total": memory_total,
|
||||
"memory_available": memory_available,
|
||||
"memory_used": memory_used,
|
||||
"memory_percent": memory_percent,
|
||||
"gpu": gpuInfo,
|
||||
}
|
||||
|
||||
@app.get("/tools/translate")
|
||||
def translate(texts: str, to_language: str):
|
||||
"""翻译"""
|
||||
return {"texts": trans.translate(Sentence=texts, to_Language=to_language)}
|
||||
|
||||
logger.warning("本地服务，请勿将服务端口暴露于外网")
|
||||
print(f"api文档地址 http://127.0.0.1:{config.server_config.port}/docs")
|
||||
webbrowser.open(f"http://127.0.0.1:{config.server_config.port}")
|
||||
uvicorn.run(app, port=config.server_config.port)
|
||||
@@ -1,13 +1,14 @@
|
||||
import torch
|
||||
import sys
|
||||
from transformers import AutoTokenizer, AutoModelForMaskedLM
|
||||
from config import config
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/chinese-roberta-wwm-ext-large")
|
||||
|
||||
models = dict()
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph, device=None):
|
||||
def get_bert_feature(text, word2ph, device=config.bert_gen_config.device):
|
||||
if (
|
||||
sys.platform == "darwin"
|
||||
and torch.backends.mps.is_available()
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
from text import chinese, japanese, cleaned_text_to_sequence
|
||||
from text import chinese, japanese, english, cleaned_text_to_sequence
|
||||
|
||||
|
||||
language_module_map = {"ZH": chinese, "JP": japanese}
|
||||
language_module_map = {"ZH": chinese, "JP": japanese, "EN": english}
|
||||
|
||||
|
||||
def clean_text(text, language):
|
||||
|
||||
162
text/english.py
162
text/english.py
@@ -171,22 +171,160 @@ def refine_syllables(syllables):
|
||||
return phonemes, tones
|
||||
|
||||
|
||||
import re
|
||||
import inflect
|
||||
|
||||
_inflect = inflect.engine()
|
||||
_comma_number_re = re.compile(r"([0-9][0-9\,]+[0-9])")
|
||||
_decimal_number_re = re.compile(r"([0-9]+\.[0-9]+)")
|
||||
_pounds_re = re.compile(r"£([0-9\,]*[0-9]+)")
|
||||
_dollars_re = re.compile(r"\$([0-9\.\,]*[0-9]+)")
|
||||
_ordinal_re = re.compile(r"[0-9]+(st|nd|rd|th)")
|
||||
_number_re = re.compile(r"[0-9]+")
|
||||
|
||||
# List of (regular expression, replacement) pairs for abbreviations:
|
||||
_abbreviations = [
|
||||
(re.compile("\\b%s\\." % x[0], re.IGNORECASE), x[1])
|
||||
for x in [
|
||||
("mrs", "misess"),
|
||||
("mr", "mister"),
|
||||
("dr", "doctor"),
|
||||
("st", "saint"),
|
||||
("co", "company"),
|
||||
("jr", "junior"),
|
||||
("maj", "major"),
|
||||
("gen", "general"),
|
||||
("drs", "doctors"),
|
||||
("rev", "reverend"),
|
||||
("lt", "lieutenant"),
|
||||
("hon", "honorable"),
|
||||
("sgt", "sergeant"),
|
||||
("capt", "captain"),
|
||||
("esq", "esquire"),
|
||||
("ltd", "limited"),
|
||||
("col", "colonel"),
|
||||
("ft", "fort"),
|
||||
]
|
||||
]
|
||||
|
||||
|
||||
# List of (ipa, lazy ipa) pairs:
|
||||
_lazy_ipa = [
|
||||
(re.compile("%s" % x[0]), x[1])
|
||||
for x in [
|
||||
("r", "ɹ"),
|
||||
("æ", "e"),
|
||||
("ɑ", "a"),
|
||||
("ɔ", "o"),
|
||||
("ð", "z"),
|
||||
("θ", "s"),
|
||||
("ɛ", "e"),
|
||||
("ɪ", "i"),
|
||||
("ʊ", "u"),
|
||||
("ʒ", "ʥ"),
|
||||
("ʤ", "ʥ"),
|
||||
("ˈ", "↓"),
|
||||
]
|
||||
]
|
||||
|
||||
# List of (ipa, lazy ipa2) pairs:
|
||||
_lazy_ipa2 = [
|
||||
(re.compile("%s" % x[0]), x[1])
|
||||
for x in [
|
||||
("r", "ɹ"),
|
||||
("ð", "z"),
|
||||
("θ", "s"),
|
||||
("ʒ", "ʑ"),
|
||||
("ʤ", "dʑ"),
|
||||
("ˈ", "↓"),
|
||||
]
|
||||
]
|
||||
|
||||
# List of (ipa, ipa2) pairs
|
||||
_ipa_to_ipa2 = [
|
||||
(re.compile("%s" % x[0]), x[1]) for x in [("r", "ɹ"), ("ʤ", "dʒ"), ("ʧ", "tʃ")]
|
||||
]
|
||||
|
||||
|
||||
def _expand_dollars(m):
|
||||
match = m.group(1)
|
||||
parts = match.split(".")
|
||||
if len(parts) > 2:
|
||||
return match + " dollars" # Unexpected format
|
||||
dollars = int(parts[0]) if parts[0] else 0
|
||||
cents = int(parts[1]) if len(parts) > 1 and parts[1] else 0
|
||||
if dollars and cents:
|
||||
dollar_unit = "dollar" if dollars == 1 else "dollars"
|
||||
cent_unit = "cent" if cents == 1 else "cents"
|
||||
return "%s %s, %s %s" % (dollars, dollar_unit, cents, cent_unit)
|
||||
elif dollars:
|
||||
dollar_unit = "dollar" if dollars == 1 else "dollars"
|
||||
return "%s %s" % (dollars, dollar_unit)
|
||||
elif cents:
|
||||
cent_unit = "cent" if cents == 1 else "cents"
|
||||
return "%s %s" % (cents, cent_unit)
|
||||
else:
|
||||
return "zero dollars"
|
||||
|
||||
|
||||
def _remove_commas(m):
|
||||
return m.group(1).replace(",", "")
|
||||
|
||||
|
||||
def _expand_ordinal(m):
|
||||
return _inflect.number_to_words(m.group(0))
|
||||
|
||||
|
||||
def _expand_number(m):
|
||||
num = int(m.group(0))
|
||||
if num > 1000 and num < 3000:
|
||||
if num == 2000:
|
||||
return "two thousand"
|
||||
elif num > 2000 and num < 2010:
|
||||
return "two thousand " + _inflect.number_to_words(num % 100)
|
||||
elif num % 100 == 0:
|
||||
return _inflect.number_to_words(num // 100) + " hundred"
|
||||
else:
|
||||
return _inflect.number_to_words(
|
||||
num, andword="", zero="oh", group=2
|
||||
).replace(", ", " ")
|
||||
else:
|
||||
return _inflect.number_to_words(num, andword="")
|
||||
|
||||
|
||||
def _expand_decimal_point(m):
|
||||
return m.group(1).replace(".", " point ")
|
||||
|
||||
|
||||
def normalize_numbers(text):
|
||||
text = re.sub(_comma_number_re, _remove_commas, text)
|
||||
text = re.sub(_pounds_re, r"\1 pounds", text)
|
||||
text = re.sub(_dollars_re, _expand_dollars, text)
|
||||
text = re.sub(_decimal_number_re, _expand_decimal_point, text)
|
||||
text = re.sub(_ordinal_re, _expand_ordinal, text)
|
||||
text = re.sub(_number_re, _expand_number, text)
|
||||
return text
|
||||
|
||||
|
||||
def text_normalize(text):
|
||||
# todo: eng text normalize
|
||||
text = normalize_numbers(text)
|
||||
return text
|
||||
|
||||
|
||||
def g2p(text):
|
||||
phones = []
|
||||
tones = []
|
||||
word2ph = []
|
||||
words = re.split(r"([,;.\-\?\!\s+])", text)
|
||||
for w in words:
|
||||
if w.upper() in eng_dict:
|
||||
phns, tns = refine_syllables(eng_dict[w.upper()])
|
||||
words = [word for word in words if word.strip() != ""]
|
||||
for word in words:
|
||||
if word.upper() in eng_dict:
|
||||
phns, tns = refine_syllables(eng_dict[word.upper()])
|
||||
phones += phns
|
||||
tones += tns
|
||||
word2ph.append(len(phns))
|
||||
else:
|
||||
phone_list = list(filter(lambda p: p != " ", _g2p(w)))
|
||||
phone_list = list(filter(lambda p: p != " ", _g2p(word)))
|
||||
for ph in phone_list:
|
||||
if ph in arpa:
|
||||
ph, tn = refine_ph(ph)
|
||||
@@ -195,13 +333,23 @@ def g2p(text):
|
||||
else:
|
||||
phones.append(ph)
|
||||
tones.append(0)
|
||||
# todo: implement word2ph
|
||||
word2ph = [1 for i in phones]
|
||||
word2ph.append(len(phone_list))
|
||||
|
||||
phones = [post_replace_ph(i) for i in phones]
|
||||
|
||||
phones = ["_"] + phones + ["_"]
|
||||
tones = [0] + tones + [0]
|
||||
word2ph = [1] + word2ph + [1]
|
||||
|
||||
return phones, tones, word2ph
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph):
|
||||
from text import english_bert_mock
|
||||
|
||||
return english_bert_mock.get_bert_feature(text, word2ph)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# print(get_dict())
|
||||
# print(eng_word_to_phoneme("hello"))
|
||||
|
||||
@@ -1,5 +1,39 @@
|
||||
import torch
|
||||
from transformers import DebertaV2Model, DebertaV2Tokenizer
|
||||
from config import config
|
||||
import sys
|
||||
|
||||
tokenizer = DebertaV2Tokenizer.from_pretrained("./bert/deberta-v3-large")
|
||||
|
||||
models = dict()
|
||||
|
||||
|
||||
def get_bert_feature(norm_text, word2ph):
|
||||
return torch.zeros(1024, sum(word2ph))
|
||||
def get_bert_feature(text, word2ph, device=config.bert_gen_config.device):
|
||||
if (
|
||||
sys.platform == "darwin"
|
||||
and torch.backends.mps.is_available()
|
||||
and device == "cpu"
|
||||
):
|
||||
device = "mps"
|
||||
if not device:
|
||||
device = "cuda"
|
||||
if device not in models.keys():
|
||||
models[device] = DebertaV2Model.from_pretrained("./bert/deberta-v3-large").to(
|
||||
device
|
||||
)
|
||||
with torch.no_grad():
|
||||
inputs = tokenizer(text, return_tensors="pt")
|
||||
for i in inputs:
|
||||
inputs[i] = inputs[i].to(device)
|
||||
res = models[device](**inputs, output_hidden_states=True)
|
||||
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
|
||||
# assert len(word2ph) == len(text)+2
|
||||
word2phone = word2ph
|
||||
phone_level_feature = []
|
||||
for i in range(len(word2phone)):
|
||||
repeat_feature = res[i].repeat(word2phone[i], 1)
|
||||
phone_level_feature.append(repeat_feature)
|
||||
|
||||
phone_level_feature = torch.cat(phone_level_feature, dim=0)
|
||||
|
||||
return phone_level_feature.T
|
||||
|
||||
535
text/japanese.py
535
text/japanese.py
@@ -7,367 +7,65 @@ from transformers import AutoTokenizer
|
||||
|
||||
from text import punctuation, symbols
|
||||
|
||||
try:
|
||||
import MeCab
|
||||
except ImportError as e:
|
||||
raise ImportError("Japanese requires mecab-python3 and unidic-lite.") from e
|
||||
from num2words import num2words
|
||||
|
||||
_CONVRULES = [
|
||||
# Conversion of 2 letters
|
||||
"アァ/ a a",
|
||||
"イィ/ i i",
|
||||
"イェ/ i e",
|
||||
"イャ/ y a",
|
||||
"ウゥ/ u:",
|
||||
"エェ/ e e",
|
||||
"オォ/ o:",
|
||||
"カァ/ k a:",
|
||||
"キィ/ k i:",
|
||||
"クゥ/ k u:",
|
||||
"クャ/ ky a",
|
||||
"クュ/ ky u",
|
||||
"クョ/ ky o",
|
||||
"ケェ/ k e:",
|
||||
"コォ/ k o:",
|
||||
"ガァ/ g a:",
|
||||
"ギィ/ g i:",
|
||||
"グゥ/ g u:",
|
||||
"グャ/ gy a",
|
||||
"グュ/ gy u",
|
||||
"グョ/ gy o",
|
||||
"ゲェ/ g e:",
|
||||
"ゴォ/ g o:",
|
||||
"サァ/ s a:",
|
||||
"シィ/ sh i:",
|
||||
"スゥ/ s u:",
|
||||
"スャ/ sh a",
|
||||
"スュ/ sh u",
|
||||
"スョ/ sh o",
|
||||
"セェ/ s e:",
|
||||
"ソォ/ s o:",
|
||||
"ザァ/ z a:",
|
||||
"ジィ/ j i:",
|
||||
"ズゥ/ z u:",
|
||||
"ズャ/ zy a",
|
||||
"ズュ/ zy u",
|
||||
"ズョ/ zy o",
|
||||
"ゼェ/ z e:",
|
||||
"ゾォ/ z o:",
|
||||
"タァ/ t a:",
|
||||
"チィ/ ch i:",
|
||||
"ツァ/ ts a",
|
||||
"ツィ/ ts i",
|
||||
"ツゥ/ ts u:",
|
||||
"ツャ/ ch a",
|
||||
"ツュ/ ch u",
|
||||
"ツョ/ ch o",
|
||||
"ツェ/ ts e",
|
||||
"ツォ/ ts o",
|
||||
"テェ/ t e:",
|
||||
"トォ/ t o:",
|
||||
"ダァ/ d a:",
|
||||
"ヂィ/ j i:",
|
||||
"ヅゥ/ d u:",
|
||||
"ヅャ/ zy a",
|
||||
"ヅュ/ zy u",
|
||||
"ヅョ/ zy o",
|
||||
"デェ/ d e:",
|
||||
"ドォ/ d o:",
|
||||
"ナァ/ n a:",
|
||||
"ニィ/ n i:",
|
||||
"ヌゥ/ n u:",
|
||||
"ヌャ/ ny a",
|
||||
"ヌュ/ ny u",
|
||||
"ヌョ/ ny o",
|
||||
"ネェ/ n e:",
|
||||
"ノォ/ n o:",
|
||||
"ハァ/ h a:",
|
||||
"ヒィ/ h i:",
|
||||
"フゥ/ f u:",
|
||||
"フャ/ hy a",
|
||||
"フュ/ hy u",
|
||||
"フョ/ hy o",
|
||||
"ヘェ/ h e:",
|
||||
"ホォ/ h o:",
|
||||
"バァ/ b a:",
|
||||
"ビィ/ b i:",
|
||||
"ブゥ/ b u:",
|
||||
"フャ/ hy a",
|
||||
"ブュ/ by u",
|
||||
"フョ/ hy o",
|
||||
"ベェ/ b e:",
|
||||
"ボォ/ b o:",
|
||||
"パァ/ p a:",
|
||||
"ピィ/ p i:",
|
||||
"プゥ/ p u:",
|
||||
"プャ/ py a",
|
||||
"プュ/ py u",
|
||||
"プョ/ py o",
|
||||
"ペェ/ p e:",
|
||||
"ポォ/ p o:",
|
||||
"マァ/ m a:",
|
||||
"ミィ/ m i:",
|
||||
"ムゥ/ m u:",
|
||||
"ムャ/ my a",
|
||||
"ムュ/ my u",
|
||||
"ムョ/ my o",
|
||||
"メェ/ m e:",
|
||||
"モォ/ m o:",
|
||||
"ヤァ/ y a:",
|
||||
"ユゥ/ y u:",
|
||||
"ユャ/ y a:",
|
||||
"ユュ/ y u:",
|
||||
"ユョ/ y o:",
|
||||
"ヨォ/ y o:",
|
||||
"ラァ/ r a:",
|
||||
"リィ/ r i:",
|
||||
"ルゥ/ r u:",
|
||||
"ルャ/ ry a",
|
||||
"ルュ/ ry u",
|
||||
"ルョ/ ry o",
|
||||
"レェ/ r e:",
|
||||
"ロォ/ r o:",
|
||||
"ワァ/ w a:",
|
||||
"ヲォ/ o:",
|
||||
"ディ/ d i",
|
||||
"デェ/ d e:",
|
||||
"デャ/ dy a",
|
||||
"デュ/ dy u",
|
||||
"デョ/ dy o",
|
||||
"ティ/ t i",
|
||||
"テェ/ t e:",
|
||||
"テャ/ ty a",
|
||||
"テュ/ ty u",
|
||||
"テョ/ ty o",
|
||||
"スィ/ s i",
|
||||
"ズァ/ z u a",
|
||||
"ズィ/ z i",
|
||||
"ズゥ/ z u",
|
||||
"ズャ/ zy a",
|
||||
"ズュ/ zy u",
|
||||
"ズョ/ zy o",
|
||||
"ズェ/ z e",
|
||||
"ズォ/ z o",
|
||||
"キャ/ ky a",
|
||||
"キュ/ ky u",
|
||||
"キョ/ ky o",
|
||||
"シャ/ sh a",
|
||||
"シュ/ sh u",
|
||||
"シェ/ sh e",
|
||||
"ショ/ sh o",
|
||||
"チャ/ ch a",
|
||||
"チュ/ ch u",
|
||||
"チェ/ ch e",
|
||||
"チョ/ ch o",
|
||||
"トゥ/ t u",
|
||||
"トャ/ ty a",
|
||||
"トュ/ ty u",
|
||||
"トョ/ ty o",
|
||||
"ドァ/ d o a",
|
||||
"ドゥ/ d u",
|
||||
"ドャ/ dy a",
|
||||
"ドュ/ dy u",
|
||||
"ドョ/ dy o",
|
||||
"ドォ/ d o:",
|
||||
"ニャ/ ny a",
|
||||
"ニュ/ ny u",
|
||||
"ニョ/ ny o",
|
||||
"ヒャ/ hy a",
|
||||
"ヒュ/ hy u",
|
||||
"ヒョ/ hy o",
|
||||
"ミャ/ my a",
|
||||
"ミュ/ my u",
|
||||
"ミョ/ my o",
|
||||
"リャ/ ry a",
|
||||
"リュ/ ry u",
|
||||
"リョ/ ry o",
|
||||
"ギャ/ gy a",
|
||||
"ギュ/ gy u",
|
||||
"ギョ/ gy o",
|
||||
"ヂェ/ j e",
|
||||
"ヂャ/ j a",
|
||||
"ヂュ/ j u",
|
||||
"ヂョ/ j o",
|
||||
"ジェ/ j e",
|
||||
"ジャ/ j a",
|
||||
"ジュ/ j u",
|
||||
"ジョ/ j o",
|
||||
"ビャ/ by a",
|
||||
"ビュ/ by u",
|
||||
"ビョ/ by o",
|
||||
"ピャ/ py a",
|
||||
"ピュ/ py u",
|
||||
"ピョ/ py o",
|
||||
"ウァ/ u a",
|
||||
"ウィ/ w i",
|
||||
"ウェ/ w e",
|
||||
"ウォ/ w o",
|
||||
"ファ/ f a",
|
||||
"フィ/ f i",
|
||||
"フゥ/ f u",
|
||||
"フャ/ hy a",
|
||||
"フュ/ hy u",
|
||||
"フョ/ hy o",
|
||||
"フェ/ f e",
|
||||
"フォ/ f o",
|
||||
"ヴァ/ b a",
|
||||
"ヴィ/ b i",
|
||||
"ヴェ/ b e",
|
||||
"ヴォ/ b o",
|
||||
"ヴュ/ by u",
|
||||
# Conversion of 1 letter
|
||||
"ア/ a",
|
||||
"イ/ i",
|
||||
"ウ/ u",
|
||||
"エ/ e",
|
||||
"オ/ o",
|
||||
"カ/ k a",
|
||||
"キ/ k i",
|
||||
"ク/ k u",
|
||||
"ケ/ k e",
|
||||
"コ/ k o",
|
||||
"サ/ s a",
|
||||
"シ/ sh i",
|
||||
"ス/ s u",
|
||||
"セ/ s e",
|
||||
"ソ/ s o",
|
||||
"タ/ t a",
|
||||
"チ/ ch i",
|
||||
"ツ/ ts u",
|
||||
"テ/ t e",
|
||||
"ト/ t o",
|
||||
"ナ/ n a",
|
||||
"ニ/ n i",
|
||||
"ヌ/ n u",
|
||||
"ネ/ n e",
|
||||
"ノ/ n o",
|
||||
"ハ/ h a",
|
||||
"ヒ/ h i",
|
||||
"フ/ f u",
|
||||
"ヘ/ h e",
|
||||
"ホ/ h o",
|
||||
"マ/ m a",
|
||||
"ミ/ m i",
|
||||
"ム/ m u",
|
||||
"メ/ m e",
|
||||
"モ/ m o",
|
||||
"ラ/ r a",
|
||||
"リ/ r i",
|
||||
"ル/ r u",
|
||||
"レ/ r e",
|
||||
"ロ/ r o",
|
||||
"ガ/ g a",
|
||||
"ギ/ g i",
|
||||
"グ/ g u",
|
||||
"ゲ/ g e",
|
||||
"ゴ/ g o",
|
||||
"ザ/ z a",
|
||||
"ジ/ j i",
|
||||
"ズ/ z u",
|
||||
"ゼ/ z e",
|
||||
"ゾ/ z o",
|
||||
"ダ/ d a",
|
||||
"ヂ/ j i",
|
||||
"ヅ/ z u",
|
||||
"デ/ d e",
|
||||
"ド/ d o",
|
||||
"バ/ b a",
|
||||
"ビ/ b i",
|
||||
"ブ/ b u",
|
||||
"ベ/ b e",
|
||||
"ボ/ b o",
|
||||
"パ/ p a",
|
||||
"ピ/ p i",
|
||||
"プ/ p u",
|
||||
"ペ/ p e",
|
||||
"ポ/ p o",
|
||||
"ヤ/ y a",
|
||||
"ユ/ y u",
|
||||
"ヨ/ y o",
|
||||
"ワ/ w a",
|
||||
"ヰ/ i",
|
||||
"ヱ/ e",
|
||||
"ヲ/ o",
|
||||
"ン/ N",
|
||||
"ッ/ q",
|
||||
"ヴ/ b u",
|
||||
"ー/:",
|
||||
# Try converting broken text
|
||||
"ァ/ a",
|
||||
"ィ/ i",
|
||||
"ゥ/ u",
|
||||
"ェ/ e",
|
||||
"ォ/ o",
|
||||
"ヮ/ w a",
|
||||
"ォ/ o",
|
||||
# Symbols
|
||||
"、/ ,",
|
||||
"。/ .",
|
||||
"！/ !",
|
||||
"？/ ?",
|
||||
"・/ ,",
|
||||
]
|
||||
|
||||
_COLON_RX = re.compile(":+")
|
||||
_REJECT_RX = re.compile("[^ a-zA-Z:,.?]")
|
||||
|
||||
|
||||
def _makerulemap():
|
||||
l = [tuple(x.split("/")) for x in _CONVRULES]
|
||||
return tuple({k: v for k, v in l if len(k) == i} for i in (1, 2))
|
||||
|
||||
|
||||
_RULEMAP1, _RULEMAP2 = _makerulemap()
|
||||
import pyopenjtalk
|
||||
import jaconv
|
||||
|
||||
|
||||
def kata2phoneme(text: str) -> str:
|
||||
"""Convert katakana text to phonemes."""
|
||||
text = text.strip()
|
||||
if text == "ー":
|
||||
return ["ー"]
|
||||
elif text.startswith("ー"):
|
||||
return ["ー"] + kata2phoneme(text[1:])
|
||||
res = []
|
||||
prev = None
|
||||
while text:
|
||||
if len(text) >= 2:
|
||||
x = _RULEMAP2.get(text[:2])
|
||||
if x is not None:
|
||||
text = text[2:]
|
||||
res += x.split(" ")[1:]
|
||||
continue
|
||||
x = _RULEMAP1.get(text[0])
|
||||
if x is not None:
|
||||
if re.match(_MARKS, text):
|
||||
res.append(text)
|
||||
text = text[1:]
|
||||
res += x.split(" ")[1:]
|
||||
continue
|
||||
res.append(text[0])
|
||||
text = text[1:]
|
||||
if text.startswith("ー"):
|
||||
if prev:
|
||||
res.append(prev[-1])
|
||||
text = text[1:]
|
||||
continue
|
||||
res += pyopenjtalk.g2p(text).lower().replace("cl", "q").split(" ")
|
||||
break
|
||||
# res = _COLON_RX.sub(":", res)
|
||||
return res
|
||||
|
||||
|
||||
_KATAKANA = "".join(chr(ch) for ch in range(ord("ァ"), ord("ン") + 1))
|
||||
_HIRAGANA = "".join(chr(ch) for ch in range(ord("ぁ"), ord("ん") + 1))
|
||||
_HIRA2KATATRANS = str.maketrans(_HIRAGANA, _KATAKANA)
|
||||
|
||||
|
||||
def hira2kata(text: str) -> str:
|
||||
text = text.translate(_HIRA2KATATRANS)
|
||||
return text.replace("う゛", "ヴ")
|
||||
return jaconv.hira2kata(text)
|
||||
|
||||
|
||||
_SYMBOL_TOKENS = set(list("・、。？！"))
|
||||
_NO_YOMI_TOKENS = set(list("「」『』―（）［］[]"))
|
||||
_TAGGER = MeCab.Tagger()
|
||||
_MARKS = re.compile(
|
||||
r"[^A-Za-z\d\u3005\u3040-\u30ff\u4e00-\u9fff\uff11-\uff19\uff21-\uff3a\uff41-\uff5a\uff66-\uff9d]"
|
||||
)
|
||||
|
||||
|
||||
def text2kata(text: str) -> str:
|
||||
parsed = _TAGGER.parse(text)
|
||||
res = []
|
||||
for line in parsed.split("\n"):
|
||||
if line == "EOS":
|
||||
break
|
||||
parts = line.split("\t")
|
||||
parsed = pyopenjtalk.run_frontend(text)
|
||||
|
||||
word, yomi = parts[0], parts[1]
|
||||
res = []
|
||||
for parts in parsed:
|
||||
word, yomi = replace_punctuation(parts["orig"]), parts["pron"].replace("’", "")
|
||||
if yomi:
|
||||
if re.match(_MARKS, yomi):
|
||||
if len(word) > 1:
|
||||
word = [replace_punctuation(i) for i in list(word)]
|
||||
yomi = word
|
||||
res += yomi
|
||||
sep += word
|
||||
continue
|
||||
elif word not in rep_map.keys() and word not in rep_map.values():
|
||||
word = ","
|
||||
yomi = word
|
||||
res.append(yomi)
|
||||
else:
|
||||
if word in _SYMBOL_TOKENS:
|
||||
@@ -381,6 +79,66 @@ def text2kata(text: str) -> str:
|
||||
return hira2kata("".join(res))
|
||||
|
||||
|
||||
def text2sep_kata(text: str) -> (list, list):
|
||||
parsed = pyopenjtalk.run_frontend(text)
|
||||
|
||||
res = []
|
||||
sep = []
|
||||
for parts in parsed:
|
||||
word, yomi = replace_punctuation(parts["orig"]), parts["pron"].replace("’", "")
|
||||
if yomi:
|
||||
if re.match(_MARKS, yomi):
|
||||
if len(word) > 1:
|
||||
word = [replace_punctuation(i) for i in list(word)]
|
||||
yomi = word
|
||||
res += yomi
|
||||
sep += word
|
||||
continue
|
||||
elif word not in rep_map.keys() and word not in rep_map.values():
|
||||
word = ","
|
||||
yomi = word
|
||||
res.append(yomi)
|
||||
else:
|
||||
if word in _SYMBOL_TOKENS:
|
||||
res.append(word)
|
||||
elif word in ("っ", "ッ"):
|
||||
res.append("ッ")
|
||||
elif word in _NO_YOMI_TOKENS:
|
||||
pass
|
||||
else:
|
||||
res.append(word)
|
||||
sep.append(word)
|
||||
return sep, [hira2kata(i) for i in res], get_accent(parsed)
|
||||
|
||||
|
||||
def get_accent(parsed):
|
||||
labels = pyopenjtalk.make_label(parsed)
|
||||
|
||||
phonemes = []
|
||||
accents = []
|
||||
for n, label in enumerate(labels):
|
||||
phoneme = re.search(r"\-([^\+]*)\+", label).group(1)
|
||||
if phoneme not in ["sil", "pau"]:
|
||||
phonemes.append(phoneme.replace("cl", "q").lower())
|
||||
else:
|
||||
continue
|
||||
a1 = int(re.search(r"/A:(\-?[0-9]+)\+", label).group(1))
|
||||
a2 = int(re.search(r"\+(\d+)\+", label).group(1))
|
||||
if re.search(r"\-([^\+]*)\+", labels[n + 1]).group(1) in ["sil", "pau"]:
|
||||
a2_next = -1
|
||||
else:
|
||||
a2_next = int(re.search(r"\+(\d+)\+", labels[n + 1]).group(1))
|
||||
# Falling
|
||||
if a1 == 0 and a2_next == a2 + 1:
|
||||
accents.append(-1)
|
||||
# Rising
|
||||
elif a2 == 1 and a2_next == 2:
|
||||
accents.append(1)
|
||||
else:
|
||||
accents.append(0)
|
||||
return list(zip(phonemes, accents))
|
||||
|
||||
|
||||
_ALPHASYMBOL_YOMI = {
|
||||
"#": "シャープ",
|
||||
"%": "パーセント",
|
||||
@@ -503,9 +261,34 @@ rep_map = {
|
||||
"！": "!",
|
||||
"？": "?",
|
||||
"\n": ".",
|
||||
"·": ",",
|
||||
"、": ",",
|
||||
"．": ".",
|
||||
"...": "…",
|
||||
"···": "…",
|
||||
"・・・": "…",
|
||||
"·": ",",
|
||||
"・": ",",
|
||||
"、": ",",
|
||||
"$": ".",
|
||||
"“": "'",
|
||||
"”": "'",
|
||||
"‘": "'",
|
||||
"’": "'",
|
||||
"（": "'",
|
||||
"）": "'",
|
||||
"(": "'",
|
||||
")": "'",
|
||||
"《": "'",
|
||||
"》": "'",
|
||||
"【": "'",
|
||||
"】": "'",
|
||||
"[": "'",
|
||||
"]": "'",
|
||||
"—": "-",
|
||||
"−": "-",
|
||||
"～": "-",
|
||||
"~": "-",
|
||||
"「": "'",
|
||||
"」": "'",
|
||||
}
|
||||
|
||||
|
||||
@@ -515,7 +298,7 @@ def replace_punctuation(text):
|
||||
replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
|
||||
|
||||
replaced_text = re.sub(
|
||||
r"[^\u3040-\u309F\u30A0-\u30FF\u4E00-\u9FFF\u3400-\u4DBF"
|
||||
r"[^\u3040-\u309F\u30A0-\u30FF\u4E00-\u9FFF\u3400-\u4DBF\u3005"
|
||||
+ "".join(punctuation)
|
||||
+ r"]+",
|
||||
"",
|
||||
@@ -542,44 +325,74 @@ def distribute_phone(n_phone, n_word):
|
||||
return phones_per_word
|
||||
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/bert-base-japanese-v3")
|
||||
def handle_long(sep_phonemes):
|
||||
for i in range(len(sep_phonemes)):
|
||||
if sep_phonemes[i][0] == "ー":
|
||||
sep_phonemes[i][0] = sep_phonemes[i - 1][-1]
|
||||
if "ー" in sep_phonemes[i]:
|
||||
for j in range(len(sep_phonemes[i])):
|
||||
if sep_phonemes[i][j] == "ー":
|
||||
sep_phonemes[i][j] = sep_phonemes[i][j - 1][-1]
|
||||
return sep_phonemes
|
||||
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/deberta-v2-large-japanese")
|
||||
|
||||
|
||||
def align_tones(phones, tones):
|
||||
res = []
|
||||
for pho in phones:
|
||||
temp = [0] * len(pho)
|
||||
for idx, p in enumerate(pho):
|
||||
if len(tones) == 0:
|
||||
break
|
||||
if p == tones[0][0]:
|
||||
temp[idx] = tones[0][1]
|
||||
if idx > 0:
|
||||
temp[idx] += temp[idx - 1]
|
||||
tones.pop(0)
|
||||
temp = [0] + temp
|
||||
temp = temp[:-1]
|
||||
if -1 in temp:
|
||||
temp = [i + 1 for i in temp]
|
||||
res.append(temp)
|
||||
res = [i for j in res for i in j]
|
||||
assert not any([i < 0 for i in res]) and not any([i > 1 for i in res])
|
||||
return res
|
||||
|
||||
|
||||
def g2p(norm_text):
|
||||
tokenized = tokenizer.tokenize(norm_text)
|
||||
phs = []
|
||||
ph_groups = []
|
||||
for t in tokenized:
|
||||
if not t.startswith("#"):
|
||||
ph_groups.append([t])
|
||||
else:
|
||||
ph_groups[-1].append(t.replace("#", ""))
|
||||
sep_text, sep_kata, acc = text2sep_kata(norm_text)
|
||||
sep_tokenized = [tokenizer.tokenize(i) for i in sep_text]
|
||||
sep_phonemes = handle_long([kata2phoneme(i) for i in sep_kata])
|
||||
# 异常处理，MeCab不认识的词的话会一路传到这里来，然后炸掉。目前来看只有那些超级稀有的生僻词会出现这种情况
|
||||
for i in sep_phonemes:
|
||||
for j in i:
|
||||
assert j in symbols, (sep_text, sep_kata, sep_phonemes)
|
||||
tones = align_tones(sep_phonemes, acc)
|
||||
|
||||
word2ph = []
|
||||
for group in ph_groups:
|
||||
phonemes = kata2phoneme(text2kata("".join(group)))
|
||||
# phonemes = [i for i in phonemes if i in symbols]
|
||||
for i in phonemes:
|
||||
assert i in symbols, (group, norm_text, tokenized)
|
||||
phone_len = len(phonemes)
|
||||
word_len = len(group)
|
||||
for token, phoneme in zip(sep_tokenized, sep_phonemes):
|
||||
phone_len = len(phoneme)
|
||||
word_len = len(token)
|
||||
|
||||
aaa = distribute_phone(phone_len, word_len)
|
||||
word2ph += aaa
|
||||
|
||||
phs += phonemes
|
||||
phones = ["_"] + phs + ["_"]
|
||||
tones = [0 for i in phones]
|
||||
phones = ["_"] + [j for i in sep_phonemes for j in i] + ["_"]
|
||||
tones = [0] + tones + [0]
|
||||
word2ph = [1] + word2ph + [1]
|
||||
assert len(phones) == len(tones)
|
||||
return phones, tones, word2ph
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/bert-base-japanese-v3")
|
||||
text = "hello,こんにちは、世界！……"
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/deberta-v2-large-japanese")
|
||||
text = "hello,こんにちは、世界ー！……"
|
||||
from text.japanese_bert import get_bert_feature
|
||||
|
||||
text = text_normalize(text)
|
||||
print(text)
|
||||
|
||||
phones, tones, word2ph = g2p(text)
|
||||
bert = get_bert_feature(text, word2ph)
|
||||
|
||||
|
||||
@@ -1,13 +1,23 @@
|
||||
import torch
|
||||
from transformers import AutoTokenizer, AutoModelForMaskedLM
|
||||
import sys
|
||||
from text.japanese import text2sep_kata
|
||||
from config import config
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/bert-base-japanese-v3")
|
||||
tokenizer = AutoTokenizer.from_pretrained("./bert/deberta-v2-large-japanese")
|
||||
|
||||
models = dict()
|
||||
|
||||
|
||||
def get_bert_feature(text, word2ph, device=None):
|
||||
def get_bert_feature(text, word2ph, device=config.bert_gen_config.device):
|
||||
sep_text, _, _ = text2sep_kata(text)
|
||||
sep_tokens = [tokenizer.tokenize(t) for t in sep_text]
|
||||
sep_ids = [tokenizer.convert_tokens_to_ids(t) for t in sep_tokens]
|
||||
sep_ids = [2] + [item for sublist in sep_ids for item in sublist] + [3]
|
||||
return get_bert_feature_with_token(sep_ids, word2ph, device)
|
||||
|
||||
|
||||
def get_bert_feature_with_token(tokens, word2ph, device=config.bert_gen_config.device):
|
||||
if (
|
||||
sys.platform == "darwin"
|
||||
and torch.backends.mps.is_available()
|
||||
@@ -18,12 +28,20 @@ def get_bert_feature(text, word2ph, device=None):
|
||||
device = "cuda"
|
||||
if device not in models.keys():
|
||||
models[device] = AutoModelForMaskedLM.from_pretrained(
|
||||
"./bert/bert-base-japanese-v3"
|
||||
"./bert/deberta-v2-large-japanese"
|
||||
).to(device)
|
||||
with torch.no_grad():
|
||||
inputs = tokenizer(text, return_tensors="pt")
|
||||
for i in inputs:
|
||||
inputs[i] = inputs[i].to(device)
|
||||
inputs = torch.tensor(tokens).to(device).unsqueeze(0)
|
||||
token_type_ids = torch.zeros_like(inputs).to(device)
|
||||
attention_mask = torch.ones_like(inputs).to(device)
|
||||
inputs = {
|
||||
"input_ids": inputs,
|
||||
"token_type_ids": token_type_ids,
|
||||
"attention_mask": attention_mask,
|
||||
}
|
||||
|
||||
# for i in inputs:
|
||||
# inputs[i] = inputs[i].to(device)
|
||||
res = models[device](**inputs, output_hidden_states=True)
|
||||
res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
|
||||
assert inputs["input_ids"].shape[-1] == len(word2ph)
|
||||
|
||||
@@ -117,7 +117,7 @@ ja_symbols = [
|
||||
"z",
|
||||
"zy",
|
||||
]
|
||||
num_ja_tones = 1
|
||||
num_ja_tones = 2
|
||||
|
||||
# English
|
||||
en_symbols = [
|
||||
|
||||
3
tools/__init__.py
Normal file
3
tools/__init__.py
Normal file
@@ -0,0 +1,3 @@
|
||||
"""
|
||||
工具包
|
||||
"""
|
||||
61
tools/translate.py
Normal file
61
tools/translate.py
Normal file
@@ -0,0 +1,61 @@
|
||||
"""
|
||||
翻译api
|
||||
"""
|
||||
from config import config
|
||||
|
||||
import random
|
||||
import hashlib
|
||||
import requests
|
||||
|
||||
|
||||
def translate(Sentence: str, to_Language: str = "jp", from_Language: str = ""):
|
||||
"""
|
||||
:param Sentence: 待翻译语句
|
||||
:param from_Language: 待翻译语句语言
|
||||
:param to_Language: 目标语言
|
||||
:return: 翻译后语句 出错时返回None
|
||||
|
||||
常见语言代码：中文 zh 英语 en 日语 jp
|
||||
"""
|
||||
appid = config.translate_config.app_key
|
||||
key = config.translate_config.secret_key
|
||||
if appid == "" or key == "":
|
||||
return "请在开发者config.yml中配置app_key与secret_key"
|
||||
url = "https://fanyi-api.baidu.com/api/trans/vip/translate"
|
||||
texts = Sentence.split("\n")
|
||||
outTexts = []
|
||||
for t in texts:
|
||||
if t != "":
|
||||
# 签名计算 参考文档 https://api.fanyi.baidu.com/product/113
|
||||
salt = str(random.randint(1, 100000))
|
||||
signString = appid + Sentence + salt + key
|
||||
hs = hashlib.md5()
|
||||
hs.update(signString.encode("utf-8"))
|
||||
signString = hs.hexdigest()
|
||||
if from_Language == "":
|
||||
from_Language = "auto"
|
||||
headers = {"Content-Type": "application/x-www-form-urlencoded"}
|
||||
payload = {
|
||||
"q": Sentence,
|
||||
"from": from_Language,
|
||||
"to": to_Language,
|
||||
"appid": appid,
|
||||
"salt": salt,
|
||||
"sign": signString,
|
||||
}
|
||||
# 发送请求
|
||||
try:
|
||||
response = requests.post(
|
||||
url=url, data=payload, headers=headers, timeout=3
|
||||
)
|
||||
response = response.json()
|
||||
if "trans_result" in response.keys():
|
||||
result = response["trans_result"][0]
|
||||
if "dst" in result.keys():
|
||||
dst = result["dst"]
|
||||
outTexts.append(dst)
|
||||
except Exception:
|
||||
return Sentence
|
||||
else:
|
||||
outTexts.append(t)
|
||||
return "\n".join(outTexts)
|
||||
56
train_ms.py
56
train_ms.py
@@ -10,6 +10,8 @@ from torch.nn.parallel import DistributedDataParallel as DDP
|
||||
from torch.cuda.amp import autocast, GradScaler
|
||||
from tqdm import tqdm
|
||||
import logging
|
||||
from config import config
|
||||
import argparse
|
||||
|
||||
logging.getLogger("numba").setLevel(logging.WARNING)
|
||||
import commons
|
||||
@@ -44,13 +46,54 @@ global_step = 0
|
||||
|
||||
|
||||
def run():
|
||||
# 环境变量解析
|
||||
envs = config.train_ms_config.env
|
||||
for env_name, env_value in envs.items():
|
||||
if env_name not in os.environ.keys():
|
||||
os.environ[env_name] = str(env_value)
|
||||
|
||||
# 多卡训练设置
|
||||
dist.init_process_group(
|
||||
backend="gloo",
|
||||
init_method="env://", # Due to some training problem,we proposed to use gloo instead of nccl.
|
||||
backend="nccl",
|
||||
init_method="env://", # If Windows,switch to gloo backend.
|
||||
) # Use torchrun instead of mp.spawn
|
||||
rank = dist.get_rank()
|
||||
n_gpus = dist.get_world_size()
|
||||
hps = utils.get_hparams()
|
||||
|
||||
# 命令行/config.yml配置解析
|
||||
# hps = utils.get_hparams()
|
||||
parser = argparse.ArgumentParser()
|
||||
# 非必要不建议使用命令行配置，请使用config.yml文件
|
||||
parser.add_argument(
|
||||
"-c",
|
||||
"--config",
|
||||
type=str,
|
||||
default=config.train_ms_config.config_path,
|
||||
help="JSON file for configuration",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"-m",
|
||||
"--model",
|
||||
type=str,
|
||||
help="数据集文件夹路径，请注意，数据不再默认放在/logs文件夹下。如果需要用命令行配置，请声明相对于根目录的路径",
|
||||
default=config.dataset_path,
|
||||
)
|
||||
args = parser.parse_args()
|
||||
model_dir = os.path.join(args.model, config.train_ms_config.model)
|
||||
if not os.path.exists(model_dir):
|
||||
os.makedirs(model_dir)
|
||||
hps = utils.get_hparams_from_file(args.config)
|
||||
hps.model_dir = model_dir
|
||||
# 比较路径是否相同
|
||||
if os.path.realpath(args.config) != os.path.realpath(
|
||||
config.train_ms_config.config_path
|
||||
):
|
||||
with open(args.config, "r", encoding="utf-8") as f:
|
||||
data = f.read()
|
||||
with open(config.train_ms_config.config_path, "w", encoding="utf-8") as f:
|
||||
f.write(data)
|
||||
|
||||
torch.manual_seed(hps.train.seed)
|
||||
torch.cuda.set_device(rank)
|
||||
global global_step
|
||||
@@ -159,6 +202,7 @@ def run():
|
||||
optim_dur_disc = None
|
||||
net_g = DDP(net_g, device_ids=[rank], find_unused_parameters=True)
|
||||
net_d = DDP(net_d, device_ids=[rank], find_unused_parameters=True)
|
||||
dur_resume_lr = None
|
||||
if net_dur_disc is not None:
|
||||
net_dur_disc = DDP(net_dur_disc, device_ids=[rank], find_unused_parameters=True)
|
||||
try:
|
||||
@@ -279,6 +323,7 @@ def train_and_evaluate(
|
||||
language,
|
||||
bert,
|
||||
ja_bert,
|
||||
en_bert,
|
||||
) in tqdm(enumerate(train_loader)):
|
||||
if net_g.module.use_noise_scaled_mas:
|
||||
current_mas_noise_scale = (
|
||||
@@ -300,6 +345,7 @@ def train_and_evaluate(
|
||||
language = language.cuda(rank, non_blocking=True)
|
||||
bert = bert.cuda(rank, non_blocking=True)
|
||||
ja_bert = ja_bert.cuda(rank, non_blocking=True)
|
||||
en_bert = en_bert.cuda(rank, non_blocking=True)
|
||||
|
||||
with autocast(enabled=hps.train.fp16_run):
|
||||
(
|
||||
@@ -321,6 +367,7 @@ def train_and_evaluate(
|
||||
language,
|
||||
bert,
|
||||
ja_bert,
|
||||
en_bert,
|
||||
)
|
||||
mel = spec_to_mel_torch(
|
||||
spec,
|
||||
@@ -515,6 +562,7 @@ def evaluate(hps, generator, eval_loader, writer_eval):
|
||||
language,
|
||||
bert,
|
||||
ja_bert,
|
||||
en_bert,
|
||||
) in enumerate(eval_loader):
|
||||
x, x_lengths = x.cuda(), x_lengths.cuda()
|
||||
spec, spec_lengths = spec.cuda(), spec_lengths.cuda()
|
||||
@@ -522,6 +570,7 @@ def evaluate(hps, generator, eval_loader, writer_eval):
|
||||
speakers = speakers.cuda()
|
||||
bert = bert.cuda()
|
||||
ja_bert = ja_bert.cuda()
|
||||
en_bert = en_bert.cuda()
|
||||
tone = tone.cuda()
|
||||
language = language.cuda()
|
||||
for use_sdp in [True, False]:
|
||||
@@ -533,6 +582,7 @@ def evaluate(hps, generator, eval_loader, writer_eval):
|
||||
language,
|
||||
bert,
|
||||
ja_bert,
|
||||
en_bert,
|
||||
y=spec,
|
||||
max_len=1000,
|
||||
sdp_ratio=0.0 if not use_sdp else 1.0,
|
||||
|
||||
89
update_status.py
Normal file
89
update_status.py
Normal file
@@ -0,0 +1,89 @@
|
||||
import os
|
||||
import gradio as gr
|
||||
|
||||
lang_dict = {"EN(英文)": "_en", "ZH(中文)": "_zh", "JP(日语)": "_jp"}
|
||||
|
||||
|
||||
def raw_dir_convert_to_path(target_dir: str, lang):
|
||||
res = target_dir.rstrip("/").rstrip("\\")
|
||||
if (not target_dir.startswith("raw")) and (not target_dir.startswith("./raw")):
|
||||
res = os.path.join("./raw", res)
|
||||
if (
|
||||
(not res.endswith("_zh"))
|
||||
and (not res.endswith("_jp"))
|
||||
and (not res.endswith("_en"))
|
||||
):
|
||||
res += lang_dict[lang]
|
||||
return res
|
||||
|
||||
|
||||
def update_g_files():
|
||||
g_files = []
|
||||
cnt = 0
|
||||
for root, dirs, files in os.walk(os.path.abspath("./logs")):
|
||||
for file in files:
|
||||
if file.startswith("G_") and file.endswith(".pth"):
|
||||
g_files.append(os.path.join(root, file))
|
||||
cnt += 1
|
||||
print(g_files)
|
||||
return f"更新模型列表完成, 共找到{cnt}个模型", gr.Dropdown.update(choices=g_files)
|
||||
|
||||
|
||||
def update_c_files():
|
||||
c_files = []
|
||||
cnt = 0
|
||||
for root, dirs, files in os.walk(os.path.abspath("./logs")):
|
||||
for file in files:
|
||||
if file.startswith("config.json"):
|
||||
c_files.append(os.path.join(root, file))
|
||||
cnt += 1
|
||||
print(c_files)
|
||||
return f"更新模型列表完成, 共找到{cnt}个配置文件", gr.Dropdown.update(choices=c_files)
|
||||
|
||||
|
||||
def update_model_folders():
|
||||
subdirs = []
|
||||
cnt = 0
|
||||
for root, dirs, files in os.walk(os.path.abspath("./logs")):
|
||||
for dir_name in dirs:
|
||||
if os.path.basename(dir_name) != "eval":
|
||||
subdirs.append(os.path.join(root, dir_name))
|
||||
cnt += 1
|
||||
print(subdirs)
|
||||
return f"更新模型文件夹列表完成, 共找到{cnt}个文件夹", gr.Dropdown.update(choices=subdirs)
|
||||
|
||||
|
||||
def update_wav_lab_pairs():
|
||||
wav_count = tot_count = 0
|
||||
for root, _, files in os.walk("./raw"):
|
||||
for file in files:
|
||||
# print(file)
|
||||
file_path = os.path.join(root, file)
|
||||
if file.lower().endswith(".wav"):
|
||||
lab_file = os.path.splitext(file_path)[0] + ".lab"
|
||||
if os.path.exists(lab_file):
|
||||
wav_count += 1
|
||||
tot_count += 1
|
||||
return f"{wav_count} / {tot_count}"
|
||||
|
||||
|
||||
def update_raw_folders():
|
||||
subdirs = []
|
||||
cnt = 0
|
||||
script_path = os.path.dirname(os.path.abspath(__file__)) # 获取当前脚本的绝对路径
|
||||
raw_path = os.path.join(script_path, "raw")
|
||||
print(raw_path)
|
||||
os.makedirs(raw_path, exist_ok=True)
|
||||
for root, dirs, files in os.walk(raw_path):
|
||||
for dir_name in dirs:
|
||||
relative_path = os.path.relpath(
|
||||
os.path.join(root, dir_name), script_path
|
||||
) # 获取相对路径
|
||||
subdirs.append(relative_path)
|
||||
cnt += 1
|
||||
print(subdirs)
|
||||
return (
|
||||
f"更新raw音频文件夹列表完成, 共找到{cnt}个文件夹",
|
||||
gr.Dropdown.update(choices=subdirs),
|
||||
gr.Textbox.update(value=update_wav_lab_pairs()),
|
||||
)
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user