Seeker
66 lines
2.8 KiB
Markdown
66 lines
2.8 KiB
Markdown
**Read this in other languages: [English](README.md), [中文](README_zh.md).**
|
|
# LLM built from scratch
|
|
Continuously optimizing and updating...
|
|
|
|
🗓️ Changelog
|
|
|
|
#### 2025-06-22
|
|
- 📝 Added reward model training, PPO training, GRPO training
|
|
|
|
<details>
|
|
<summary>Click for more details</summary>
|
|
|
|
#### 2025-06-19
|
|
- 📝 Pretokenized corpus and updated cold start code.
|
|
|
|
#### 2025-06-05
|
|
- 📝 Uploaded tokenizer training weights (1.56 chars/token), adjusted tokenizer format and training method to align with the style of the Qwen2 tokenizer.
|
|
|
|
#### 2025-05-26
|
|
- 📝 Added LLM documentation covering full lifecycle technical points of LLMs.
|
|
|
|
#### 2025-05-22
|
|
- 📝 Added support for MoE models. Training resource usage is unstable. Tested model with Experts=8, experts_per_tok=4. GPU memory fluctuated from 60% to 94%, and single GPU utilization fluctuated from 0% to 100%. After reducing the batch_size, memory usage stabilized at around 90%, single GPU utilization stabilized above 90%, occasionally dropping to around 30%.
|
|
|
|
#### 2025-05-20
|
|
- 📝 Added support for training with jsonl files.
|
|
|
|
#### 2025-05-19
|
|
- 📝 Added support for training code using DeepSpeed. Testing showed a maximum batch_size increase of 38% and a training speed increase of 9.6%.
|
|
|
|
#### 2025-05-16
|
|
- 📝 Added pre-training code for dolly_llm. Conducted a pre-training test: model 0.6B, corpus 500M, using 46GB * 4 GPUs.
|
|
|
|
#### 2025-05-14
|
|
- ✅ Released dolly_llm as a pip package for installation.
|
|
|
|
#### 2025-05-09
|
|
- 📝 Optimized code for [RMSNorm](https://arxiv.org/pdf/1910.07467), MLP, and [RoPE](https://arxiv.org/pdf/2104.09864).
|
|
|
|
#### 2025-05-07
|
|
- 📝 Standardized modeling and configuration using the transformers format, and designed/modified modeling_dolly v0.1 with 11.5B parameters.
|
|
|
|
#### 2025-05-06
|
|
- 📝 Implemented the configuration_dolly class and added modeling_dolly v0.0.
|
|
|
|
#### 2025-04-30
|
|
- 📝 Added [BBPE](https://arxiv.org/pdf/1909.03341) method training for the Tokenizer.
|
|
|
|
#### 2025-04-29
|
|
- 📝 Added tokenizer construction code, supporting sentencepiece and transformers' [BPE](https://arxiv.org/pdf/1508.07909). Supports building from text and from existing tokenizers.
|
|
|
|
#### 2025-04-24
|
|
- ✅ Tested constructing custom LLM model architectures from transformers.
|
|
</details>
|
|
|
|
## Acknowledgments
|
|
|
|
Special thanks to the following resources and articles for their assistance:
|
|
|
|
- [Huggingface transformers](https://github.com/huggingface/transformers)
|
|
- [Qwen](https://huggingface.co/Qwen)
|
|
- [Neural Machine Translation of Rare Words with Subword Units](https://arxiv.org/pdf/1508.07909)
|
|
- [Neural Machine Translation with Byte-Level Subwords](https://arxiv.org/pdf/1909.03341)
|
|
- [Root Mean Square Layer Normalization](https://arxiv.org/pdf/1910.07467)
|
|
- [RoPE RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/pdf/2104.09864)
|