[2025.06.04]update tokenizer train code and style

2025-06-04 17:17:19 +08:00
parent 13c1edeb05
commit b61572a8a1
1 changed files with 3 additions and 0 deletions
@@ -4,6 +4,9 @@ LLM-Dolly is a custom LLM built from scratch

 ## 🗓️ 更新日志

+#### 2025-06-04
+- 📝 调整tokenizer的格式和训练方式，和qwen2的tokenizer风格保持一致
+
 #### 2025-05-26
 - 📝 增加LLM文档，涵盖LLM全周期技术点。