tsukumi 2833fb5eeb Fix: The conversion script for English BERT to Fast Tokenizer was incorrect, so tokenization was not performed correctly

2024-12-19 11:10:33 +09:00

..

config.json

Dev no emo (#123)

2023-10-28 10:36:21 +08:00

generator_config.json

Dev no emo (#123)

2023-10-28 10:36:21 +08:00

tokenizer_config.json

Dev no emo (#123)

2023-10-28 10:36:21 +08:00

tokenizer.json

Fix: The conversion script for English BERT to Fast Tokenizer was incorrect, so tokenization was not performed correctly

2024-12-19 11:10:33 +09:00