admin/LLM-from-scratch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
754ba76140d1323a6b0e955fadc405c12e1ac0d4
LLM-from-scratch/Tokenizer_code/README.md
T
coffee 71b968f782 add trained BBPE Tokenizer model
2025-04-30 17:25:53 +08:00

105 B
Raw Blame History

USAGE

使用sentencepiece和huggingface transformers分别进行tokenizer的训练 support BPE and BBPE

Reference in New Issue View Git Blame Copy Permalink