更新并完善分布式训练功能 (#186)

* 修复多机训练问题

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* 更新并完善分布式训练功能

近期融合V2版本代码时发现之前修改的多机功能并不正确，仍会报错，只不过单机多卡情况下local_rank即相当于rank，感知不出
1. 修复train_ms.py中DDP初始化及.cuda绑定到local_rank上
2. 在default_config.yml配置文件中添加env变量 LOCAL_RANK，否则默认情况下会key error
3. 添加run_MnodesAndMgpus.sh，更新分布式相关说明

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

This commit is contained in:

Lvjinhong

2023-11-16 13:46:15 +08:00

committed by

GitHub

parent 1fbddf4202

commit 31de84eb3b

4 changed files with 80 additions and 27 deletions

									
										1

default_config.yml
									
												View File
												
				@@ -66,6 +66,7 @@ train_ms:

				    MASTER_ADDR: "localhost"

				    MASTER_PORT: 10086

				    WORLD_SIZE: 1

				    LOCAL_RANK: 0

				    RANK: 0

				    # 可以填写任意名的环境变量

				    # THE_ENV_VAR_YOU_NEED_TO_USE: "1234567"

更新并完善分布式训练功能 (#186)

1 default_config.yml Unescape Escape View File

1

default_config.yml

View File