admin/LLM-from-scratch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
49 Commits 1 Branch 0 Tags
f3760009efcc355f455644dc38b87db33a3692ff
T
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
coffee f3760009ef update RMSNorm code comments
2025-05-09 18:30:06 +08:00
corpus
add corpus files and update ignore
2025-04-29 18:35:28 +08:00
src_llm
update RMSNorm code comments
2025-05-09 18:30:06 +08:00
src_tokenizer
update tokenizer code
2025-05-07 11:24:51 +08:00
test
add dolly test
2025-05-09 17:47:11 +08:00
utils
update utils
2025-05-07 11:28:08 +08:00
weights_tokenizer
update tokenizer weights
2025-05-07 11:25:19 +08:00
.gitignore
update gitignore
2025-05-07 11:19:46 +08:00
LICENSE
update readme
2025-05-07 11:21:18 +08:00
README.md
update README.md
2025-05-09 18:06:46 +08:00
requirements_version.txt
add requirements_version.txt
2025-05-07 11:27:09 +08:00
requirements.txt
update requirements.txt
2025-05-07 11:27:38 +08:00
start from 2025.04.23
add start date
2025-05-07 11:35:41 +08:00

README.md

LLM-Dolly

LLM-Dolly is a custom LLM built from scratch, 持续优化更新中...

🗓️ 更新日志

2025-05-07

  • 📝 使用transformers格式规范modeling和configuration，并设计修改v0.1版modeling_dolly,参数量11.5B。

2025-05-06

  • 📝 实现 configuration_dolly类,以及添加v0.0版modeling_dolly。

2025-04-30

  • 📝 添加Tokenizer的BBPE方式训练。

2025-04-29

  • 📝 添加tokenizer构建代码，支持sentencepiece和transfomers方式，支持从文本构建和从已有的tokenzier构建。

2025-04-24

  • ✅ 测试从transformers构建自定义的LLM模型结构。

致谢

特别感谢以下资源和文章的帮助：

Reference in New Issue View Git Blame Copy Permalink
S
Description
大语言模型（LLM）从零训练流程的最小复现。
Readme 7.8 MiB
Languages
Python 99.6%
Shell 0.4%