admin/LLM-from-scratch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tokenizer support BBPE

Browse Source
This commit is contained in:
coffee
2025-04-30 16:38:18 +08:00
parent 96c99c8e1a
commit c5c5a560df
3 changed files with 10 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+5 -2
View File
@@ -3,10 +3,13 @@ Dolly is a custom LLM built from scratch
## 🗓️ 更新日志
### 2025-04-29
#### 2025-04-30
- 📝 添加Tokenizer的BBPE方式训练
#### 2025-04-29
- 📝 添加tokenizer构建代码，支持sentencepiece和transfomers方式，支持从文本构建和从已有的tokenzier构建。
### 2025-04-24
#### 2025-04-24
- ✅ 测试从transformers构建自定义的LLM模型结构。
## 致谢