Dev emo (#171)

* SYNC CHANGE TO EMO BRANCH (#162) * Update README.md * 更新 bert_models.json * fix * Update data_utils.py * Update infer.py * performance improve * Feat: support auto split in webui (#158) * Feat: support auto split in webui * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix: change /voice api to post (#160) * Fix: change /voice api to post * Fix: support /voice api get * Fix: Add missing torch.cuda.empty_cache() (#161) --------- Co-authored-by: Sora <atri@suzakuintsubaki.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Artrajz <969242373@qq.com> * sync (#163) * Update README.md * 更新 bert_models.json * fix * Update data_utils.py * Update infer.py * performance improve * Feat: support auto split in webui (#158) * Feat: support auto split in webui * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix: change /voice api to post (#160) * Fix: change /voice api to post * Fix: support /voice api get * Fix: Add missing torch.cuda.empty_cache() (#161) * del emo * del emo * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Sora <atri@suzakuintsubaki.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Artrajz <969242373@qq.com> * Add files via upload * Update infer.py * add emo * add emo * Update default_config.yml * Fix slice segments GPU perf (#165) * Fix slice segments GPU perf * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update commons.py --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update infer.py * Update models.py * Update infer.py * remove spec cache * Update data_utils.py * Update data_utils.py * Update train_ms.py * Revert "Fix slice segments GPU perf (#165)" (#169) This reverts commit 28430fc76bc628297bb59d8f8d25100dbe46ab59. * Update train_ms.py * Update train_ms.py * Update data_utils.py * Update data_utils.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update default_config.yml * Switch to Japanese wwm DeBERTa (#172) * Switch to Japanese wwm DeBERTa * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix wrong ellipsis g2p (#173) * Switch to Japanese wwm DeBERTa * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix ellipsis g2p * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Add files via upload * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix English phones not aligned with BERT features (#174) * Fix English phones not aligned with BERT features * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix english bert gen (#175) * Update webui.py * Update webui.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add NCCL timeout * Update train_ms.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update train_ms.py * Update default_config.yml * Update infer.py * Update models.py * Update train_ms.py * Update infer.py * Update emo_gen.py * Feat: Support load and infer 2.0 models (#178) * Feat: Support load and infer 2.0 models * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * 复用相同逻辑，修正静音添加错误 (#181) * Refactor: reuse the same part of voice api. * Fix: server_fastapi.py * Update train_ms.py * Update data_utils.py * Update data_utils.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update data_utils.py * Update data_utils.py * Add files via upload * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update default_config.yml * Update utils.py * Update train_ms.py * Update utils.py * Update default_config.yml * Update data_utils.py * Update default_config.yml * Update train_ms.py * Update train_ms.py * Update config.py * Update utils.py * Update train_ms.py * Update train_ms.py * feat: add voice mix and tone mix (#187) * feat: add voice mix and tone mix * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Stardust·减 <star_dust_chen@foxmail.com> * Add files via upload * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Sora <atri@suzakuintsubaki.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Artrajz <969242373@qq.com> Co-authored-by: Leng Yue <lengyue@lengyue.me> Co-authored-by: OedoSoldier <31711261+OedoSoldier@users.noreply.github.com> Co-authored-by: 潮幻Mark <sunyunfei201@gmail.com>
2023-11-25 13:08:51 +08:00
parent ec9145cb59
commit b186499ae9
38 changed files with 156708 additions and 875 deletions
--- a/webui.py
+++ b/webui.py
@@ -1,7 +1,6 @@
 # flake8: noqa: E402
 import os
 import logging
-
 import re_matching
 from tools.sentence import split_by_language

@@ -24,6 +23,7 @@ import webbrowser
 import numpy as np
 from config import config
 from tools.translate import translate
+import librosa

 net_g = None

@@ -40,6 +40,8 @@ def generate_audio(
    length_scale,
    speaker,
    language,
+    reference_audio,
+    emotion,
    skip_start=False,
    skip_end=False,
 ):
@@ -51,6 +53,8 @@ def generate_audio(
            skip_end = (idx != len(slices) - 1) and skip_end
            audio = infer(
                piece,
+                reference_audio=reference_audio,
+                emotion=emotion,
                sdp_ratio=sdp_ratio,
                noise_scale=noise_scale,
                noise_scale_w=noise_scale_w,
@@ -117,6 +121,8 @@ def tts_split(
    cut_by_sent,
    interval_between_para,
    interval_between_sent,
+    reference_audio,
+    emotion,
 ):
    if language == "mix":
        return ("invalid", None)
@@ -130,6 +136,8 @@ def tts_split(
            skip_end = idx != len(para_list) - 1
            audio = infer(
                p,
+                reference_audio=reference_audio,
+                emotion=emotion,
                sdp_ratio=sdp_ratio,
                noise_scale=noise_scale,
                noise_scale_w=noise_scale_w,
@@ -157,6 +165,8 @@ def tts_split(
                skip_end = (idx != len(sent_list) - 1) and skip_end
                audio = infer(
                    s,
+                    reference_audio=reference_audio,
+                    emotion=emotion,
                    sdp_ratio=sdp_ratio,
                    noise_scale=noise_scale,
                    noise_scale_w=noise_scale_w,
@@ -193,6 +203,8 @@ def tts_fn(
    noise_scale_w,
    length_scale,
    language,
+    reference_audio,
+    emotion,
 ):
    audio_list = []
    if language == "mix":
@@ -261,6 +273,8 @@ def tts_fn(
                        length_scale,
                        _speaker,
                        lang_to_generate,
+                        reference_audio,
+                        emotion,
                        skip_start,
                        skip_end,
                    )
@@ -305,6 +319,8 @@ def tts_fn(
                        noise_scale,
                        noise_scale_w,
                        length_scale,
+                        reference_audio,
+                        emotion,
                        speaker,
                        lang_to_generate,
                        skip_start,
@@ -322,6 +338,8 @@ def tts_fn(
                length_scale,
                speaker,
                language,
+                reference_audio,
+                emotion,
            )
        )

@@ -360,22 +378,25 @@ if __name__ == "__main__":
                trans = gr.Button("中翻日", variant="primary")
                slicer = gr.Button("快速切分", variant="primary")
                speaker = gr.Dropdown(
-                    choices=speakers, value=speakers[0], label="选择说话人"
+                    choices=speakers, value=speakers[0], label="Speaker"
+                )
+                emotion = gr.Slider(
+                    minimum=0, maximum=9, value=0, step=1, label="Emotion"
                )
                sdp_ratio = gr.Slider(
-                    minimum=0, maximum=1, value=0.2, step=0.1, label="SDP/DP混合比"
+                    minimum=0, maximum=1, value=0.2, step=0.1, label="SDP Ratio"
                )
                noise_scale = gr.Slider(
-                    minimum=0.1, maximum=2, value=0.6, step=0.1, label="感情"
+                    minimum=0.1, maximum=2, value=0.6, step=0.1, label="Noise"
                )
                noise_scale_w = gr.Slider(
-                    minimum=0.1, maximum=2, value=0.8, step=0.1, label="音素长度"
+                    minimum=0.1, maximum=2, value=0.8, step=0.1, label="Noise_W"
                )
                length_scale = gr.Slider(
-                    minimum=0.1, maximum=2, value=1.0, step=0.1, label="语速"
+                    minimum=0.1, maximum=2, value=1.0, step=0.1, label="Length"
                )
                language = gr.Dropdown(
-                    choices=languages, value=languages[0], label="选择语言(新增mix混合选项)"
+                    choices=languages, value=languages[0], label="Language"
                )
                btn = gr.Button("生成音频！", variant="primary")
            with gr.Column():
@@ -408,6 +429,8 @@ if __name__ == "__main__":
                #     show_download_button=False,
                #     value=os.path.abspath("./img/参数说明.png"),
                # )
+                reference_text = gr.Markdown(value="## 情感参考音频（WAV 格式）：用于生成语音的情感参考。")
+                reference_audio = gr.Audio(label="情感参考音频（WAV 格式）", type="filepath")
        btn.click(
            tts_fn,
            inputs=[
@@ -418,6 +441,8 @@ if __name__ == "__main__":
                noise_scale_w,
                length_scale,
                language,
+                reference_audio,
+                emotion,
            ],
            outputs=[text_output, audio_output],
        )
@@ -440,10 +465,17 @@ if __name__ == "__main__":
                opt_cut_by_sent,
                interval_between_para,
                interval_between_sent,
+                reference_audio,
+                emotion,
            ],
            outputs=[text_output, audio_output],
        )

+        reference_audio.upload(
+            lambda x: librosa.load(x, 16000)[::-1],
+            inputs=[reference_audio],
+            outputs=[reference_audio],
+        )
    print("推理页面已开启!")
    webbrowser.open(f"http://127.0.0.1:{config.webui_config.port}")
    app.launch(share=config.webui_config.share, server_port=config.webui_config.port)