 coffee
|
9c1739baf3
|
[2025.06.10] train EN tokenizer use parquet files
|
2025-06-10 17:56:00 +08:00 |
|
|
root
|
eb1c6c0c17
|
[2025.06.10]train en tokenizer
|
2025-06-10 17:36:53 +08:00 |
|
|
coffee
|
c6e171e8ed
|
[2025.06.10] adjust directory structure
|
2025-06-10 14:04:57 +08:00 |
|
|
coffee
|
ef2f7a1bc5
|
[25.06.10] update code
|
2025-06-10 13:58:12 +08:00 |
|
|
coffee
|
f9e0ae5825
|
[2025.06.06] update tokenizer train code
|
2025-06-06 16:06:36 +08:00 |
|
|
coffee
|
628aa8c674
|
[2025.06.05] add tokenzier test
|
2025-06-05 18:35:39 +08:00 |
|
|
coffee
|
edc60be53b
|
[2025.06.05] upload tokenizer weights and update the tokenizer train code
|
2025-06-05 18:33:32 +08:00 |
|
|
coffee
|
dc4bbf43f6
|
[2025.06.05] add tokenzier test
|
2025-06-05 18:20:36 +08:00 |
|
|
coffee
|
0e023c8bc1
|
[2025.06.05] upload tokenizer weights and update the tokenizer train code
|
2025-06-05 18:16:34 +08:00 |
|
|
coffee
|
80517880b5
|
[2025.06.05] upload tokenizer weights and update the tokenizer train code
|
2025-06-05 18:15:28 +08:00 |
|
|
coffee
|
947ad2b87c
|
[2025.06.05] upload tokenizer weights and update the tokenizer train code
|
2025-06-05 18:12:48 +08:00 |
|
|
coffee
|
6a30309848
|
[2025.06.05] update tokenizer train code, use the style of Qwen
|
2025-06-05 18:09:18 +08:00 |
|
|
coffee
|
db4c2658b1
|
[2025.06.05] add BBPE tokenizer weights and update code
|
2025-06-05 18:01:07 +08:00 |
|
|
coffee
|
b61572a8a1
|
[2025.06.04]update tokenizer train code and style
|
2025-06-04 17:17:19 +08:00 |
|
|
coffee
|
13c1edeb05
|
update tokenizer code, train from local files
|
2025-06-04 17:15:09 +08:00 |
|
|
coffee
|
319516e3fb
|
rm llm
|
2025-05-29 17:18:24 +08:00 |
|
|
coffee
|
d8c63e73aa
|
add doc_tokenizer
|
2025-05-29 17:16:48 +08:00 |
|
|
hanluzhi0429@gmail.com
|
ba00ae0c36
|
[25.05.27] add unigramLM
|
2025-05-27 21:41:34 +08:00 |
|
|
hanluzhi0429@gmail.com
|
61b8b42b3c
|
[25.05.26] add documents->tokenizer
|
2025-05-26 22:35:30 +08:00 |
|
|
hanluzhi0429@gmail.com
|
87fc8933ec
|
[25.05.26] add documents->tokenizer
|
2025-05-26 22:31:22 +08:00 |
|
|
coffee
|
6d7bdecac5
|
[25.05.26] add deepspeed stage3 config
|
2025-05-26 17:40:32 +08:00 |
|
|
coffee
|
a7edf87dca
|
[2025.05.22] add MoE support
|
2025-05-22 18:13:01 +08:00 |
|
|
coffee
|
9ae88672f5
|
[2025.05.22] add MoE support
|
2025-05-22 18:11:50 +08:00 |
|
|
coffee
|
6152ab5a35
|
add MoE support
|
2025-05-22 17:52:24 +08:00 |
|
|
coffee
|
a229a11b0e
|
[25.05.22] add MoE support
|
2025-05-22 17:45:47 +08:00 |
|
|
coffee
|
28fe25780e
|
[2025.05.22] add MoE support
|
2025-05-22 16:47:52 +08:00 |
|
|
coffee
|
1d698ba7c3
|
add jsonl support
|
2025-05-20 17:19:09 +08:00 |
|
|
coffee
|
b9b1d4b51d
|
add jsonl support
|
2025-05-20 17:17:49 +08:00 |
|
|
coffee
|
b262507cb8
|
[25.05.20] update code support resume from checkpoints
|
2025-05-20 11:31:35 +08:00 |
|
|
coffee
|
646c182469
|
update README.md
|
2025-05-19 18:01:12 +08:00 |
|
|
coffee
|
4cf94f0d48
|
update requirements
|
2025-05-19 17:59:24 +08:00 |
|
|
coffee
|
82f7c41bdc
|
update requirements_version file
|
2025-05-19 17:53:53 +08:00 |
|
|
coffee
|
11473a798f
|
remove pycache
|
2025-05-19 15:43:02 +08:00 |
|
|
coffee
|
5ca9ee3803
|
remove pycache
|
2025-05-19 15:43:02 +08:00 |
|
|
coffee
|
acdbde74ce
|
add torchrun train script
|
2025-05-19 11:52:16 +08:00 |
|
|
coffee
|
010580981e
|
add torchrun train script
|
2025-05-19 11:52:16 +08:00 |
|
|
coffee
|
98a71bac61
|
25.05.19: add ds train code
|
2025-05-19 11:51:33 +08:00 |
|
|
coffee
|
be180dfb6a
|
25.05.19: add ds train code
|
2025-05-19 11:51:33 +08:00 |
|
|
coffee
|
ddde74e187
|
25.05.19:update pretrain_transformers code
|
2025-05-19 11:50:14 +08:00 |
|
|
coffee
|
c2d9b96acc
|
25.05.19:update pretrain_transformers code
|
2025-05-19 11:50:14 +08:00 |
|
|
coffee
|
0482d6559d
|
update dolly config
|
2025-05-19 09:12:33 +08:00 |
|
|
coffee
|
c36f0911a7
|
update dolly config
|
2025-05-19 09:12:33 +08:00 |
|
|
coffee
|
2db94bfa0f
|
add pretrain code
|
2025-05-19 08:46:46 +08:00 |
|
|
coffee
|
1ee644e8ae
|
add pretrain code
|
2025-05-19 08:46:46 +08:00 |
|
|
coffee
|
7e55276b43
|
25.05.15:update modeling_dolly code and code comments
|
2025-05-15 10:51:57 +08:00 |
|
|
coffee
|
b48ff8a049
|
25.05.15:update modeling_dolly code and code comments
|
2025-05-15 10:51:57 +08:00 |
|
|
coffee
|
d52f33aa4a
|
25.05.15:update modeling_dolly code and code comments
|
2025-05-15 10:46:58 +08:00 |
|
|
coffee
|
630d761f26
|
25.05.15:update modeling_dolly code and code comments
|
2025-05-15 10:46:58 +08:00 |
|
|
coffee
|
48019ee38a
|
25.05.14:update modeling_dolly code and code comments
|
2025-05-14 17:37:20 +08:00 |
|
|
coffee
|
bf0f04bcb3
|
25.05.14:update modeling_dolly code and code comments
|
2025-05-14 17:37:20 +08:00 |
|