add trained BBPE Tokenizer model

2025-04-30 17:25:53 +08:00
parent a68b346429
commit 9e604a0339
6 changed files with 126549 additions and 2 deletions
@@ -1,2 +1,3 @@
 # USAGE
-使用sentencepiece和huggingface transformers分别进行tokenizer的训练
+使用sentencepiece和huggingface transformers分别进行tokenizer的训练
+support BPE and BBPE