add trained BBPE Tokenizer model

2025-04-30 17:25:53 +08:00
parent 76be55cbdf
commit 71b968f782
6 changed files with 126549 additions and 2 deletions
@@ -1,2 +1,3 @@
 # USAGE
-使用sentencepiece和huggingface transformers分别进行tokenizer的训练
+使用sentencepiece和huggingface transformers分别进行tokenizer的训练
+support BPE and BBPE