Dev no emo (#123)

* Create emo_gen.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update server.py, fix bugs in func get_text() and infer(). (#52)

* Extract get_text() and infer() from webui.py. (#53)

* Extract get_text() and infer() from webui.py.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* add emo emb

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* init emo gen

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* init emo

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* init emo

* Delete bert/bert-base-japanese-v3 directory

* Create .gitkeep

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Create add_punc.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix bug in bert_gen.py (#54)

* Update README.md

* fix bug in models.py (#56)

* 更新 models.py

* Fix japanese cleaner (#61)

* 初步，睡觉明天继续写（

* 好好好放错分支了，熬夜是大忌

* [pre-commit.ci] pre-commit autoupdate (#55)

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/pre-commit/pre-commit-hooks: v4.4.0 → v4.5.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.4.0...v4.5.0)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Create tokenizer_config.json

* update preprocess_text.py：过滤一个音频匹配多个文本的情况 (#57)

* update preprocess_text.py：过滤音频不存在的情况 (#58)

* 修复日语cleaner和bert

* better

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: Stardust·减 <star_dust_chen@foxmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sora <atri@suzakuintsubaki.com>

* Apply Code Formatter Change

* Add config.yml for global configuration. (#62)

* Add config.yml for global configuration.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix bug in webui.py.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Rename config.yml to default_config.yml. Add ./config.yml to gitignore.

* Add config.py to parse config.yml

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update webui.py (#65)

* Update webui.py:
1. Add auto translation from Chinese to Japanese.
2. Start to use config.py in webui.py to set config instead of using the command line.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix (#68)

* 加上ー

* fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update infer.py and webui.py.  Supports loading and inference models of 1.1.1 version. (#66)

* Update infer.py and webui.py. Supports loading and inference models of 1.1.1 version.

* Update config.json

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix bug in translate.py (#69)

* Supports loading and inference models of 1.1、1.0.1、1.0 version. (#70)

* Supports loading and inference models of 1.1、1.0.1、1.0 version.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Delete useless file in OldVersion

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update japanese.py (#71)

Handling JA long pronunciations

* 使用配置文件配置bert_gen.py, preprocess_text.py, resample.py (#72)

* Update bert_gen.py, preprocess_text.py, resample.py. Support using config.yml in these files.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update bert_gen.py

* Update bert_gen.py, fix bug.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Delete bert/bert-base-japanese-v3 directory

* Create config.json

* Create tokenizer_config.json

* Create vocab.txt

* Update server.py. 支持多版本多模型 (#76)

* Update server.py. 支持多版本多模型

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Dev webui (#77)

* 申请pr (#75)

* 2023/10/11 update

界面优化

* Update webui.py

翻译英文页面为中文

* Update train_ms.py

单卡训练

* 加入图片

* Update extern_subprocess.py

* Update asr_transcript.py

* Update asr_transcript.py

* Update asr_transcript.py

* Update extern_subprocess.py

* Update asr_transcript.py

* Update asr_transcript.py

* Update asr_transcript.py

* Update all_process.py

* Update extern_subprocess.py

* Update all_process.py

* Update all_process.py

* Update asr_transcript.py

* Update extern_subprocess.py

* Update webui.py

* Create re_matching.py

* Update webui.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update all_process.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update all_process.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update all_process.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update asr_transcript.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Pack 'update' functions into a module

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update all_process.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update asr_transcript.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update extern_subprocess.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update all_process.py

* Update asr_transcript.py

* Update webui.py

* Add files via upload

* Update extern_subprocess.py

* Update all_process.py

* Update asr_transcript.py

* Update bert_gen.py

* Update extern_subprocess.py

* Update preprocess_text.py

* Update re_matching.py

* Update resample.py

* Update update_status.py

* Update update_status.py

* Update webui.py

* Update all_process.py

* Update preprocess_text.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update train_ms.py

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Stardust·减 <star_dust_chen@foxmail.com>
Co-authored-by: innnky <67028263+innnky@users.noreply.github.com>

* Delete all_process.py

* Delete asr_transcript.py

* Delete extern_subprocess.py

---------

Co-authored-by: spicysama <122108331+AnyaCoder@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: innnky <67028263+innnky@users.noreply.github.com>

* Create config.json

* Create  preprocessor_config.json

* Create vocab.json

* Delete emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim/.gitkeep

* Update emo_gen.py

* Delete add_punc.py

* add emotion_clustering.i

* Apply Code Formatter Change

* Update models.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update preprocess_text.py (#78)

* Update preprocess_text.py. 检测重复以及不存在的音频 (#79)

* Handle Janpanese long pronunciations (#80)

* Handle Janpanese long pronunciations

* Update japanese.py

* Update japanese.py

* Use unified phonemes for Japanese long vowel (#82)

* Use an unified phoneme for Japanese long vowel

`symbol.py` has not been updated to ensure compatibility with older version models.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* 增加一个按钮，点击后可以按句子切分，添加“|” (#81)

* Update re_matching.py

* Update webui.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix phonemer bug (#83)

* Fix phonemer bug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix long vowel handler bug (#84)

* Fix long vowel handler bug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* 加入整合包管理器的特性：长文本合成可以自定义句间段间停顿 (#85)

* Update webui.py

* Update re_matching.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update train_ms.py

* fix'

* Update cleaner.py

* add en

* add en

* Update english.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add en

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add en

* add en

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add en

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 更新 README.md

* 更新 README.md

* 更新 README.md

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Change phonemer to pyopenjtalk (#86)

* Change phonemer to pyopenjtalk

* 修改为openjtalk便于安装

---------

Co-authored-by: Stardust·减 <star_dust_chen@foxmail.com>

* 更新 english.py

* Fix english_bert_mock.py. (#87)

* Add punctuation execptions (#88)

* Add punctuation execptions

* Ellipses exceptions

* remove get bert

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix bug in oldVersion. (#89)

* Update requirements.txt

* change to large

* rollback requirements.txt

* Feat: Enable 1.1.1 models using fix-ver infer. (#91)

* Feat: Enable 1.1.1 models using fix-ver infer.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add Japanese accent (high-low) (#90)

* Add punctuation execptions

* Ellipses exceptions

* Add Japanese accent

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Do not replace iteration mark (#92)

* Add punctuation execptions

* Ellipses exceptions

* Add Japanese accent

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Do not replace iteration mark

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix: fix import error in oldVersion (#93)

* Refactor: reusing model loading in webui.py and server.py. (#94)

* Feat: Enable using config.yml in train_ms.py (#96)

* 更新 emo_gen.py

* Change emo_gen.py (#97)

* Fix emo_gen bugs

* Add multiprocess

* Fix queue (#98)

* Fix emo_gen bugs

* Add multiprocess

* Del var

* Fix queue

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix training bugs (#99)

* Updatge cluster notebook

* Fix train

* Fix filename

* Update infer.py (#100)

* Update infer.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add reference audio (#101)

* Add reference audio

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update

* Update

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Stardust·减 <star_dust_chen@foxmail.com>

* Fix: fix 1.1.1-fix (#102)

* Fix infer bug (#103)

* Feat: Add server_fastapi.py. (#104)

* Feat: Add server_fastapi.py.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix: Update requirements.txt.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix: requirements.txt. (#105)

* Swith to deberta-v3-large (#106)

* Swith to deberta-v3-large

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Feat: Update config.py. (#107)

* Feat: Update config.py.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Dev fix (#108)

* fix bugs when deploying

* fix bugs when deploying

* fix bugs when deploying

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Revert "Dev fix (#108)" (#109)

This reverts commit 685e18a10498d602b1a9a26079340d11925646f0.

* Dev fix (#110)

* fix bugs when deploying

* fix bugs when deploying

* fix bugs when deploying

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix fixed bugs

* fix fixed bugs

* fix fixed bug 3

* fix fixed bug 4

* fix fixed bug 5

* fix

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add emo vec quantizer (#111)

Co-authored-by: Stardust·减 <star_dust_chen@foxmail.com>

* Clean req and gitignore (#112)

* Clean req and gitignore

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Switch to deberta-v2-large-japanese (#113)

* Switch to deberta-v2-large-japanese

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix emo bugs (#114)

* Fix english (#115)

* Remove emo (#117)

* Don't train codebook

* Remove emo

* Update

* Update

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Merge dev into no-emo (#122)

* [pre-commit.ci] pre-commit autoupdate (#95)

* [pre-commit.ci] pre-commit autoupdate

updates:
- [github.com/astral-sh/ruff-pre-commit: v0.0.292 → v0.1.1](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.292...v0.1.1)
- [github.com/psf/black: 23.9.1 → 23.10.0](https://github.com/psf/black/compare/23.9.1...23.10.0)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Don't train codebook (#116)

* Update requirements.txt

* Update english_bert_mock.py

* Fix: server_fastapi.py (#118)

* Fix: server_fastapi.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix: don't print debug logging. (#119)

* Fix: don't print debug logging.

* Feat: support emo_gen config

* Fix config

* Apply Code Formatter Change

* 更新，修正bug (#121)

* Feat: Update infer.py preprocess_text.py server_fastapi.py.

* Fix resample.py. Maintain same directory structure in out_dir as in_dir.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update resample.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update server_fastapi.py to no-emo ver

* Update config.py, no emo config

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: OedoSoldier <31711261+OedoSoldier@users.noreply.github.com>
Co-authored-by: Stardust·减 <star_dust_chen@foxmail.com>
Co-authored-by: Stardust-minus <Stardust-minus@users.noreply.github.com>

* Update train_ms.py

* Update latest version info (#124)

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: jiangyuxiaoxiao <atri@suzakuintsubaki.com>
Co-authored-by: AkitoLiu <39857739+Akito-UzukiP@users.noreply.github.com>
Co-authored-by: Stardust-minus <Stardust-minus@users.noreply.github.com>
Co-authored-by: OedoSoldier <31711261+OedoSoldier@users.noreply.github.com>
Co-authored-by: spicysama <122108331+AnyaCoder@users.noreply.github.com>
Co-authored-by: innnky <67028263+innnky@users.noreply.github.com>
Co-authored-by: YYuX-1145 <138500330+YYuX-1145@users.noreply.github.com>

oldVersion/V110/text/chinese.py

@@ -0,0 +1,198 @@
+import os
+import re
+
+import cn2an
+from pypinyin import lazy_pinyin, Style
+
+from .symbols import punctuation
+from .tone_sandhi import ToneSandhi
+
+current_file_path = os.path.dirname(__file__)
+pinyin_to_symbol_map = {
+    line.split("\t")[0]: line.strip().split("\t")[1]
+    for line in open(os.path.join(current_file_path, "opencpop-strict.txt")).readlines()
+}
+
+import jieba.posseg as psg
+
+
+rep_map = {
+    "：": ",",
+    "；": ",",
+    "，": ",",
+    "。": ".",
+    "！": "!",
+    "？": "?",
+    "\n": ".",
+    "·": ",",
+    "、": ",",
+    "...": "…",
+    "$": ".",
+    "“": "'",
+    "”": "'",
+    "‘": "'",
+    "’": "'",
+    "（": "'",
+    "）": "'",
+    "(": "'",
+    ")": "'",
+    "《": "'",
+    "》": "'",
+    "【": "'",
+    "】": "'",
+    "[": "'",
+    "]": "'",
+    "—": "-",
+    "～": "-",
+    "~": "-",
+    "「": "'",
+    "」": "'",
+}
+
+tone_modifier = ToneSandhi()
+
+
+def replace_punctuation(text):
+    text = text.replace("嗯", "恩").replace("呣", "母")
+    pattern = re.compile("|".join(re.escape(p) for p in rep_map.keys()))
+
+    replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
+
+    replaced_text = re.sub(
+        r"[^\u4e00-\u9fa5" + "".join(punctuation) + r"]+", "", replaced_text
+    )
+
+    return replaced_text
+
+
+def g2p(text):
+    pattern = r"(?<=[{0}])\s*".format("".join(punctuation))
+    sentences = [i for i in re.split(pattern, text) if i.strip() != ""]
+    phones, tones, word2ph = _g2p(sentences)
+    assert sum(word2ph) == len(phones)
+    assert len(word2ph) == len(text)  # Sometimes it will crash,you can add a try-catch.
+    phones = ["_"] + phones + ["_"]
+    tones = [0] + tones + [0]
+    word2ph = [1] + word2ph + [1]
+    return phones, tones, word2ph
+
+
+def _get_initials_finals(word):
+    initials = []
+    finals = []
+    orig_initials = lazy_pinyin(word, neutral_tone_with_five=True, style=Style.INITIALS)
+    orig_finals = lazy_pinyin(
+        word, neutral_tone_with_five=True, style=Style.FINALS_TONE3
+    )
+    for c, v in zip(orig_initials, orig_finals):
+        initials.append(c)
+        finals.append(v)
+    return initials, finals
+
+
+def _g2p(segments):
+    phones_list = []
+    tones_list = []
+    word2ph = []
+    for seg in segments:
+        # Replace all English words in the sentence
+        seg = re.sub("[a-zA-Z]+", "", seg)
+        seg_cut = psg.lcut(seg)
+        initials = []
+        finals = []
+        seg_cut = tone_modifier.pre_merge_for_modify(seg_cut)
+        for word, pos in seg_cut:
+            if pos == "eng":
+                continue
+            sub_initials, sub_finals = _get_initials_finals(word)
+            sub_finals = tone_modifier.modified_tone(word, pos, sub_finals)
+            initials.append(sub_initials)
+            finals.append(sub_finals)
+
+            # assert len(sub_initials) == len(sub_finals) == len(word)
+        initials = sum(initials, [])
+        finals = sum(finals, [])
+        #
+        for c, v in zip(initials, finals):
+            raw_pinyin = c + v
+            # NOTE: post process for pypinyin outputs
+            # we discriminate i, ii and iii
+            if c == v:
+                assert c in punctuation
+                phone = [c]
+                tone = "0"
+                word2ph.append(1)
+            else:
+                v_without_tone = v[:-1]
+                tone = v[-1]
+
+                pinyin = c + v_without_tone
+                assert tone in "12345"
+
+                if c:
+                    # 多音节
+                    v_rep_map = {
+                        "uei": "ui",
+                        "iou": "iu",
+                        "uen": "un",
+                    }
+                    if v_without_tone in v_rep_map.keys():
+                        pinyin = c + v_rep_map[v_without_tone]
+                else:
+                    # 单音节
+                    pinyin_rep_map = {
+                        "ing": "ying",
+                        "i": "yi",
+                        "in": "yin",
+                        "u": "wu",
+                    }
+                    if pinyin in pinyin_rep_map.keys():
+                        pinyin = pinyin_rep_map[pinyin]
+                    else:
+                        single_rep_map = {
+                            "v": "yu",
+                            "e": "e",
+                            "i": "y",
+                            "u": "w",
+                        }
+                        if pinyin[0] in single_rep_map.keys():
+                            pinyin = single_rep_map[pinyin[0]] + pinyin[1:]
+
+                assert pinyin in pinyin_to_symbol_map.keys(), (pinyin, seg, raw_pinyin)
+                phone = pinyin_to_symbol_map[pinyin].split(" ")
+                word2ph.append(len(phone))
+
+            phones_list += phone
+            tones_list += [int(tone)] * len(phone)
+    return phones_list, tones_list, word2ph
+
+
+def text_normalize(text):
+    numbers = re.findall(r"\d+(?:\.?\d+)?", text)
+    for number in numbers:
+        text = text.replace(number, cn2an.an2cn(number), 1)
+    text = replace_punctuation(text)
+    return text
+
+
+def get_bert_feature(text, word2ph):
+    from text import chinese_bert
+
+    return chinese_bert.get_bert_feature(text, word2ph)
+
+
+if __name__ == "__main__":
+    from text.chinese_bert import get_bert_feature
+
+    text = "啊！但是《原神》是由,米哈\游自主，  [研发]的一款全.新开放世界.冒险游戏"
+    text = text_normalize(text)
+    print(text)
+    phones, tones, word2ph = g2p(text)
+    bert = get_bert_feature(text, word2ph)
+
+    print(phones, tones, word2ph, bert.shape)
+
+
+# # 示例用法
+# text = "这是一个示例文本：,你好！这是一个测试...."
+# print(g2p_paddle(text))  # 输出: 这是一个示例文本你好这是一个测试