admin/LLM-from-scratch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add tokenizer code

Browse Source
This commit is contained in:
coffee
2025-04-29 18:21:17 +08:00
parent df6dbd8702
commit 1b809eb64c
43 changed files with 546364 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tokenizer_file/hf_tokenizer/special_tokens_map.json
+7
View File
@@ -0,0 +1,7 @@
{
"cls_token": "[CLS]",
"mask_token": "[MASK]",
"pad_token": "[PAD]",
"sep_token": "[SEP]",
"unk_token": "[UNK]"
}