add tokenizer_code readme

2025-04-29 18:33:35 +08:00
parent 2798c7bc68
commit c945aad67c
4 changed files with 2 additions and 75652 deletions
@@ -0,0 +1,2 @@
+# USAGE
+使用sentencepiece和huggingface transformers分别进行tokenizer的训练