admin/LLM-from-scratch

Files

T

History

coffee f9e0ae5825 [2025.06.06] update tokenizer train code

2025-06-06 16:06:36 +08:00

..

hf_tokenizer_code

[2025.06.06] update tokenizer train code

2025-06-06 16:06:36 +08:00

spm_tokenizer_code

[2025.06.05] update tokenizer train code, use the style of Qwen

2025-06-05 18:09:18 +08:00

README.md

update tokenizer code

2025-05-07 11:24:51 +08:00

README.md

USAGE

使用sentencepiece和huggingface transformers分别进行tokenizer的训练 support BPE and BBPE