add tokenizer_code readme

2025-04-29 18:33:35 +08:00
parent b5d90ba346
commit 45ffd1a891
4 changed files with 2 additions and 75652 deletions
@@ -0,0 +1,2 @@
+# USAGE
+使用sentencepiece和huggingface transformers分别进行tokenizer的训练