admin/LLM-from-scratch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ece74ce72ac3d09678c3ed8e4058596488c04a39
LLM-from-scratch/utils/spm_train_tokenizer.sh
T
coffee 1b809eb64c add tokenizer code
2025-04-29 18:21:17 +08:00

1 line
101 B
Bash
Raw Blame History

spm_train --input=corpus/wikipedia.txt --model_prefix=tokenizer_zh --vocab_size=32000 --num_threads=8
Reference in New Issue View Git Blame Copy Permalink