[25.05.22] add MoE support

2025-05-22 17:45:47 +08:00
parent 28fe25780e
commit a229a11b0e
3 changed files with 10 additions and 5 deletions
@@ -4,6 +4,9 @@ LLM-Dolly is a custom LLM built from scratch

 ## 🗓️ 更新日志

+#### 2025-05-22
+- 📝 添加支持MoE模型，训练资源占用不稳定，Experts=8, experts_per_tok=4,GPU显存从60%-94%跳动,单个GPU利用率0%-100%跳动。
+
 #### 2025-05-20
 - 📝 添加支持jsonl文件训练.