admin/LLM-from-scratch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update tokenizer code

Browse Source
This commit is contained in:
coffee
2025-05-07 11:24:51 +08:00
parent cf47360d01
commit c5ab40c2df
10 changed files with 189 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src_tokenizer/README.md
Executable
+3
View File
@@ -0,0 +1,3 @@
# USAGE
使用sentencepiece和huggingface transformers分别进行tokenizer的训练
support BPE and BBPE