admin/LLM-from-scratch

T

coffee 79b90c99fe update README log

2025-04-29 18:49:12 +08:00

add corpus files and update ignore

2025-04-29 18:35:28 +08:00

add tokenizer_code readme

2025-04-29 18:33:35 +08:00

del some tokenizer model file

2025-04-29 18:24:43 +08:00

add utils for data process

2025-04-29 18:28:45 +08:00

.gitignore

add corpus files and update ignore

2025-04-29 18:35:28 +08:00

LICENSE

Initial commit

2025-04-23 22:42:17 +08:00

README.md

update README log

2025-04-29 18:49:12 +08:00

requirements.txt

add tokenizer code

2025-04-29 18:21:17 +08:00

test_huggingface_train.py

test huggingface train

2025-04-24 22:32:31 +08:00

test_huggingface_transformer.py

test huggingface transformers

2025-04-24 22:19:55 +08:00

test.py

add tokenizer code

2025-04-29 18:21:17 +08:00

README.md

dolly

Dolly is a custom LLM built from scratch

🗓️ 更新日志

2025-04-29

📝 添加tokenizer构建代码，支持sentencepiece和transfomers方式，支持从文本构建和从已有的tokenzier构建。

2025-04-24

✅ 测试从transformers构建自定义的LLM模型结构。

致谢

特别感谢以下资源和文章的帮助：