admin/LLM-from-scratch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add MoE support

Browse Source
This commit is contained in:
coffee
2025-05-22 17:52:24 +08:00
parent a229a11b0e
commit 6152ab5a35
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+1 -1
View File
@@ -5,7 +5,7 @@ LLM-Dolly is a custom LLM built from scratch
## 🗓️ 更新日志 ## 🗓️ 更新日志
#### 2025-05-22 #### 2025-05-22
- 📝 添加支持MoE模型，训练资源占用不稳定，Experts=8, experts_per_tok=4,GPU显存从60%-94%跳动,单个GPU利用率0%-100%跳动。 - 📝 添加支持MoE模型，训练资源占用不稳定，测试模型Experts=8, experts_per_tok=4,GPU显存从60%-94%跳动,单个GPU利用率0%-100%跳动。
#### 2025-05-20 #### 2025-05-20
- 📝 添加支持jsonl文件训练. - 📝 添加支持jsonl文件训练.