tsukumi 2833fb5eeb Fix: The conversion script for English BERT to Fast Tokenizer was incorrect, so tokenization was not performed correctly

2024-12-19 11:10:33 +09:00

..

added_tokens.json

[pre-commit.ci] pre-commit autoupdate (#32)

2023-09-19 10:21:34 +08:00

config.json

add bert

2023-07-21 12:22:49 +08:00

special_tokens_map.json

[pre-commit.ci] pre-commit autoupdate (#32)

2023-09-19 10:21:34 +08:00

tokenizer_config.json

[pre-commit.ci] pre-commit autoupdate (#32)

2023-09-19 10:21:34 +08:00

tokenizer.json

Fix: The conversion script for English BERT to Fast Tokenizer was incorrect, so tokenization was not performed correctly

2024-12-19 11:10:33 +09:00

vocab.txt

add bert

2023-07-21 12:22:49 +08:00