admin/LLM-from-scratch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
165f61b8b1214ea3b276355192ca32abf378f00a
LLM-from-scratch/src_tokenizer
T
History
coffee e8438a6b42 update tokenizer code
2025-05-07 11:24:51 +08:00
..
hf_tokenizer_code
update tokenizer code
2025-05-07 11:24:51 +08:00
spm_tokenizer_code
update tokenizer code
2025-05-07 11:24:51 +08:00
README.md
update tokenizer code
2025-05-07 11:24:51 +08:00

README.md

USAGE

使用sentencepiece和huggingface transformers分别进行tokenizer的训练 support BPE and BBPE

Reference in New Issue View Git Blame Copy Permalink