update

.gitignore

@@ -19,3 +19,4 @@ venv/
 *.zip
 *.csv
 *.bak
+mos_*.png

README.md

@@ -2,6 +2,8 @@
 
 Bert-VITS2 with more controllable voice styles.
 
+[English README](docs/README_en.md)
+
 https://github.com/litagin02/Style-Bert-VITS2/assets/139731664/b907c1b8-43aa-46e6-b03f-f6362f5a5a1e
 
 **注意**: 上記動画のライセンス表記は誤っていました、正しくはCC BY-SA 4.0で商用利用に制限はありません。近日訂正版動画に差し替えます。

docs/CHANGELOG.md

@@ -7,9 +7,9 @@
 - クラウド実行等の際にパスの指定をこちらでできるように、パスの設定を`configs/paths.yml`にまとめました（colabの[ノートブック](http://colab.research.google.com/github/litagin02/Style-Bert-VITS2/blob/master/colab.ipynb)もそれに伴って変えたので新しいものを使ってください）。デフォルトは`dataset_root: Data`と`assets_root: model_assets`です。クラウド等でやる方はここを変更してください。
 - どのステップ数の出力がよいかの「一つの」指標として [SpeechMOS](https://github.com/tarepan/SpeechMOS) を使うスクリプトを追加：
 ```bash
-python speech_mos.py -m <model_name> [-o <output_csv_file>]
+python speech_mos.py -m <model_name>
 ```
-ステップごとの自然性評価が表示されるはず。読み上げさせたい文章を変えたかったら中のファイルを弄って各自調整してください。またあくまで目安のひとつなので、実際に読み上げさせて選別するのが一番だと思います。
+ステップごとの自然性評価が表示され、`mos_{model_name}.csv`と`mos_{model_name}.png`に結果が保存される。読み上げさせたい文章を変えたかったら中のファイルを弄って各自調整してください。またあくまで目安のひとつなので、実際に読み上げさせて選別するのが一番だと思います。
 
 ## v1.2 (2023-12-31)
 

docs/CLI.md

@@ -1,6 +1,6 @@
 # CLI
 
-
+**WIP**
 
 ## Dataset
 

docs/README_en.md

@@ -0,0 +1,123 @@
+# Style-Bert-VITS2
+
+Bert-VITS2 with more controllable voice styles.
+
+https://github.com/litagin02/Style-Bert-VITS2/assets/139731664/b907c1b8-43aa-46e6-b03f-f6362f5a5a1e
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](http://colab.research.google.com/github/litagin02/Style-Bert-VITS2/blob/master/colab.ipynb)
+
+Online demo: https://huggingface.co/spaces/litagin/Style-Bert-VITS2-JVNV
+
+This repository is based on [Bert-VITS2](https://github.com/fishaudio/Bert-VITS2) v2.1, so many thanks to the original author!
+
+- [Update History](docs/CHANGELOG.md)
+
+**Overview**
+
+- Based on Bert-VITS2 v2.1, which generates emotionally rich voices from entered text, this version allows free control of emotions and speaking styles, including intensity.
+- Easy to install and train for people without Git or Python (for Windows users), much is borrowed from [EasyBertVits2](https://github.com/Zuntan03/EasyBertVits2/). Training on Google Colab is also supported: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](http://colab.research.google.com/github/litagin02/Style-Bert-VITS2/blob/master/colab.ipynb)
+- If used only for voice synthesis, it can operate on CPU without a graphics card.
+- Also includes an API server for integration with others (PR by [@darai0512](https://github.com/darai0512), thank you).
+- Originally, Bert-VITS2's strength was to read "happy text happily, sad text sadly", so even without using the added style specification in this fork, you can generate emotionally rich voices.
+
+
+## How to Use
+
+<!-- For more details, please refer to [here](docs/tutorial.md). -->
+
+### Operating Environment
+
+We have confirmed the operation in Windows Command Prompt, WSL2, and Linux (Ubuntu Desktop) for each UI and API Server (please be creative with path specifications in WSL).
+
+### Installation
+
+#### For Those Unfamiliar with Git or Python
+
+Assuming Windows:
+
+1. Download and unzip [this zip file](https://github.com/litagin02/Style-Bert-VITS2/releases/download/1.3/Style-Bert-VITS2.zip).
+   - If you have a graphics card, double-click `Install-Style-Bert-VITS2.bat`.
+   - If you don't have a graphics card, double-click `Install-Style-Bert-VITS2-CPU.bat`.
+2. Wait for the necessary environment to install automatically.
+3. After that, if the WebUI for voice synthesis launches automatically, the installation is successful. The default model will be downloaded, so you can play with it immediately.
+
+For updates, please double-click `Update-Style-Bert-VITS2.bat`.
+
+#### For Those Familiar with Git and Python
+
+```bash
+git clone https://github.com/litagin02/Style-Bert-VITS2.git
+cd Style-Bert-VITS2
+python -m venv venv
+venv\Scripts\activate
+pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
+pip install -r requirements.txt
+python initialize.py  # Download necessary models and default TTS model
+```
+Don't forget the last step.
+
+### Voice Synthesis
+Double-click `App.bat`` or run `python app.py`` to launch the WebUI. The default model is downloaded during installation, so you can use it even if you haven't trained it.
+
+The structure of the model files required for voice synthesis is as follows (you don't need to place them manually):
+
+```
+model_assets
+├── your_model
+│   ├── config.json
+│   ├── your_model_file1.safetensors
+│   ├── your_model_file2.safetensors
+│   ├── ...
+│   └── style_vectors.npy
+└── another_model
+    ├── ...
+```
+
+For inference, `config.json`, `*.safetensors`, and `style_vectors.npy` are necessary. If you want to share a model, please share these three files.
+
+Among them, `style_vectors.npy` is a file necessary to control the style. By default, the average style "Neutral" is generated during training.
+If you want to use multiple styles for more detailed control, please refer to "Generating Styles" below (even with only the average style, if the training data is emotionally rich, sufficiently emotionally rich voices can be generated).
+
+### Training
+
+Double-click Train.bat or run `python webui_train.py` to launch the WebUI.
+
+### Generating Styles
+For those who want to use styles other than the default "Neutral".
+
+- Double-click `Style.bat` or run `python webui_style_vectors.py` to launch the WebUI.
+- It is independent of training, so you can do it even during training, and you can redo it any number of times after training is complete (preprocessing must be finished).
+- For more details on the specifications of the style, please refer to [clustering.ipynb](../clustering.ipynb).
+
+### Dataset Creation
+
+- Double-click `Dataset.bat` or run `python webui_dataset.py` to launch the WebUI for creating datasets from audio files. You can use this tool to learn from audio files only.
+
+Note: If you want to manually correct the dataset, remove noise, etc., you may find [Aivis](https://github.com/tsukumijima/Aivis) or its Windows-compatible dataset part [Aivis Dataset](https://github.com/litagin02/Aivis-Dataset) useful. However, if there are many files, etc., it may be sufficient to simply cut out and create a dataset with this tool.
+
+Please experiment to see what kind of dataset is best.
+
+### API Server
+Run `python server_fastapi.py` in the constructed environment to launch the API server.
+Please check the API specification after launching at `/docs`.
+
+By default, CORS settings are allowed for all domains.
+As much as possible, change the value of server.origins in `config.yml` and limit it to trusted domains (if you delete the key, you can disable the CORS settings).
+
+### Merging
+You can create a new model by mixing two models in terms of "voice", "emotional expression", and "tempo".
+Double-click `Merge.bat` or run `python webui_merge.py` to launch the WebUI.
+
+## Relation to Bert-VITS2 v2.1
+Basically, it's just a slight modification of the Bert-VITS2 v2.1 model structure. The [pre-trained model](https://huggingface.co/litagin/Style-Bert-VITS2-1.0-base) is also essentially the same as Bert-VITS2 v2.1 (unnecessary weights have been removed and converted to safetensors).
+
+The differences are as follows:
+
+- Like [EasyBertVits2](https://github.com/Zuntan03/EasyBertVits2), it is easy to use even for people who do not know Python or Git.
+- Changed the model for emotional embedding (from 1024-dimensional [wav2vec2-large-robust-12-ft-emotion-msp-dim](https://huggingface.co/audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim) to 256-dimensional [wespeaker-voxceleb-resnet34-LM](https://huggingface.co/pyannote/wespeaker-voxceleb-resnet34-LM), which is more for speaker identification than emotional embedding)
+- Removed vector quantization from embeddings and replaced it with just a fully connected layer.
+- By creating a style vector file `style_vectors.npy`, you can generate voices using that style and continuously specify the strength of the effect.
+- Various WebUIs created
+- Support for bf16 training
+- Support for safetensors format, defaulting to using safetensors
+- Other minor bug fixes and refactoring

slice.py

@@ -14,6 +14,7 @@ vad_model, utils = torch.hub.load(
     repo_or_dir="snakers4/silero-vad",
     model="silero_vad",
     onnx=True,
+    trust_repo=True,
 )
 
 (get_speech_timestamps, _, read_audio, *_) = utils

speech_mos.py

@@ -36,7 +36,7 @@ predictor = torch.hub.load(
 
 parser = argparse.ArgumentParser()
 parser.add_argument("--model_name", "-m", type=str, required=True)
-parser.add_argument("--device", "-d", type=str, default="cuda"))
+parser.add_argument("--device", "-d", type=str, default="cuda")
 
 args = parser.parse_args()
 
@@ -121,22 +121,20 @@ plt.ylabel("MOS")
 
 # ステップ数の軸ラベルを1000単位で表示するように調整
 plt.xticks(
-    ticks=np.arange(0, max(steps) + 1000, 1000),
-    labels=[f"{int(x/1000)}k" for x in np.arange(0, max(steps) + 1000, 5000)],
+    ticks=np.arange(0, max(steps) + 1000, 2000),
+    labels=[f"{int(x/1000)}" for x in np.arange(0, max(steps) + 1000, 2000)],
 )
 
 # 縦の補助線を追加
 plt.grid(True, axis="x")
 
-# 凡例をグラフの外側に配置
-plt.legend(loc="center left", bbox_to_anchor=(1, 0.5))
+# 凡例をグラフの右外側に配置
+plt.legend(loc="upper left", bbox_to_anchor=(1, 1))
 
-# 凡例を表示
-plt.legend()
+plt.tight_layout()
+
+# グラフを画像として保存（plt.show() の前に実行する）
+plt.savefig(f"mos_{model_name}.png")
 
 # グラフを表示
 plt.show()
-
-plt.savefig(f"mos_{model_name}.png", bbox_inches="tight")
-
-logger.info(f"mos_{model_name}.png has been saved.")

transcribe.py

@@ -1,5 +1,6 @@
 import argparse
 import os
+import sys
 
 from faster_whisper import WhisperModel
 from tqdm import tqdm
@@ -79,3 +80,4 @@ if __name__ == "__main__":
             text = transcribe(wav_file, initial_prompt=initial_prompt)
             f.write(f"{file_name}|{speaker_name}|{language}|{text}\n")
             f.flush()
+    sys.exit(0)

webui_style_vectors.py

@@ -170,7 +170,7 @@ def do_clustering_gradio(n_clusters=4, method="KMeans"):
         )
     plt.legend()
 
-    return [plt.gcf(), gr.Slider(maximum=n_clusters, value=1)] + [
+    return [plt, gr.Slider(maximum=n_clusters, value=1)] + [
         gr.Audio(visible=False)
     ] * MAX_AUDIO_NUM
 

webui_train.py

@@ -329,11 +329,11 @@ if __name__ == "__main__":
                 )
                 normalize = gr.Checkbox(
                     label="音声の音量を正規化する",
-                    value=True,
+                    value=False,
                 )
                 trim = gr.Checkbox(
                     label="音声の最初と最後の無音を取り除く",
-                    value=True,
+                    value=False,
                 )
             with gr.Column():
                 preprocess_button = gr.Button(value="自動前処理を実行", variant="primary")
@@ -400,14 +400,6 @@ if __name__ == "__main__":
             with gr.Row(variant="panel"):
                 with gr.Column():
                     gr.Markdown(value="#### Step 4: BERT特徴ファイルの生成")
-                    # num_processes_bert = gr.Slider(
-                    #     label="プロセス数",
-                    #     value=cpu_count() // 2,
-                    #     minimum=1,
-                    #     maximum=cpu_count(),
-                    #     step=1,
-                    # )
-                    # bert_genは重いようで、初期の4がよいみたい
                 with gr.Column():
                     bert_gen_btn = gr.Button(value="実行", variant="primary")
                     info_bert = gr.Textbox(label="状況")