admin/LLM-from-scratch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
13c1edeb05cf83f141e12a0b9b9444a687a92e4a
LLM-from-scratch/utils/spm_train_tokenizer.sh
T
coffee 78a009f1c8 update utils
2025-05-07 11:28:08 +08:00

1 line
101 B
Bash
Executable File
Raw Blame History

spm_train --input=corpus/wikipedia.txt --model_prefix=tokenizer_zh --vocab_size=32000 --num_threads=8
Reference in New Issue View Git Blame Copy Permalink