admin/LLM-from-scratch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[25.05.20] update code support resume from checkpoints

Browse Source
This commit is contained in:
coffee
2025-05-20 11:31:35 +08:00
parent 646c182469
commit b262507cb8
9 changed files with 70 additions and 172 deletions
Download Patch File Download Diff File Expand all files Collapse all files
utils/dolly_pretrain_deepspeed.sh
+1 -1
View File
@@ -1,2 +1,2 @@
deepspeed --num_gpus=4 --master_port=12345 pretrain_transformers.py
# deepspeed --include="localhost:1,2,3" --master_port=12345 pretrain_transformers.py
# CUDA_VISIBLE_DEVICES=1,2,3 deepspeed --include="localhost:1,2,3" --master_port=12345 src_pretrain/pretrain_deepspeed.py
utils/dolly_pretrain_torch.sh
+2 -1
View File
@@ -1,3 +1,4 @@
python3 -m torch.distributed.launch --nproc_per_node=4 --master_port=12345 src_pretrain/pretrain_transformers.py
# torchrun --nproc_per_node=4 src_pretrain/pretrain_transformers.py
# CUDA_VISIBLE_DEVICES=1,2,3 torchrun --nproc_per_node=3 src_pretrain/pretrain_transformers.py
# CUDA_VISIBLE_DEVICES=1,2,3 torchrun --nproc_per_node=3 src_pretrain/pretrain_transformers.py
# CUDA_VISIBLE_DEVICES=1,2,3 python3 -m torch.distributed.launch --nproc_per_node=4 --master_port=12345 src_pretrain/pretrain_transformers.py