杩戞棩锛岄樋閲屼簯鏈哄櫒瀛︿範骞冲彴PAI涓诲鐨勮鏂囥€奧hale: Efficient Giant Model Training over Heterogeneous GPUs銆嬶紝楂樻晥澶фā鍨嬭缁冩鏋禬hale鍏ラ€塙SENIX ATC'22銆俉hale閫氳繃瀵逛笉鍚屽苟琛屽寲绛栫暐杩涜缁熶竴鎶借薄銆佸皝瑁咃紝鍦ㄤ竴濂楀垎甯冨紡璁粌妗嗘灦涓敮鎸佸绉嶅苟琛岀瓥鐣ワ紝骞惰繘琛屾樉瀛樸€佽绠椼€侀€氫俊绛夊叏鏂逛綅鐨勪紭鍖栵紝鏉ユ彁渚涙槗鐢ㄣ€侀珮鏁堢殑鍒嗗竷寮忚缁冩鏋躲€
USENIX Annul Technical Conference (USENIX ATC)锛屾槸璁＄畻鏈虹郴缁熼鍩熷浗闄呴《绾у鏈細璁 (CCF-A)锛岃嚜1992骞翠妇鍔炵涓€灞奤SENIX ATC浼氳浠ユ潵锛岃嚦浠婂凡鎴愬姛涓惧姙30澶氬眾锛屽湪瀛︽湳鍜屽伐涓氱晫閮芥湁宸ㄥぇ鐨勫奖鍝嶅姏銆俇SENIX ATC2022灏嗕簬2022骞7鏈11鏃ュ彫寮€銆傛娆″叆閫夋剰鍛崇潃闃块噷浜戞満鍣ㄥ涔犲钩鍙癙AI鑷爺鐨勬繁搴﹀涔犲垎甯冨紡妯″瀷璁粌绯荤粺杈惧埌浜嗗叏鐞冧笟鐣屽厛杩涙按骞筹紝鑾峰緱浜嗗浗闄呭鑰呯殑璁ゅ彲锛屽睍鐜颁簡涓浗鏈哄櫒瀛︿範绯荤粺鎶€鏈垱鏂板湪鍥介檯涓婄殑绔炰簤鍔涖€
Whale鏄樋閲屼簯鏈哄櫒瀛︿範PAI骞冲彴鑷爺鐨勯珮鏁堛€侀€氱敤銆佺‖浠舵劅鐭ョ殑澶фā鍨嬪垎甯冨紡璁粌妗嗘灦锛岀幇宸插紑婧愶紝寮€婧愬悗鐨勫悕绉版槸EPL(Easy Parallel Library)銆俉hale閫氳繃缁熶竴鐨勭瓥鐣ユ娊璞℃潵瀹炵幇鍚勭鍒嗗竷寮忕瓥鐣ョ殑琛ㄨ揪锛屽苟閫氳繃纭欢鎰熺煡鍜岃嚜鍔ㄨ绠楀浘鏀瑰啓鍙婁紭鍖栧畬鎴愰珮鏁堢殑鍒嗗竷寮忔ā鍨嬪疄鐜般€
闅忕潃妯″瀷鍙傛暟瑙勬ā椋為€熷闀匡紝妯″瀷鏁堟灉涔熷湪閫愭鎻愰珮锛屼絾鍚屾椂涔熶负璁粌妗嗘灦甯︽潵鏇村ぇ鐨勬寫鎴樸€傚綋鍓嶇殑涓氱晫鍒嗗竷寮忚缁冩鏋跺彧鏀寔灏戦噺鐨勫苟琛岀瓥鐣ワ紝缂轰箯涓€涓粺涓€鐨勬娊璞℃潵鏀寔鎵€鏈夌殑骞惰绛栫暐鍙婂叾娣峰悎绛栫暐銆傚悓鏃跺疄鐜板鏉傜殑骞惰绛栫暐闇€瑕佸ぇ閲忕殑妯″瀷浠ｇ爜鏀归€犲拰瀵瑰簳灞傜郴缁熺殑鐞嗚В锛屽ぇ澶у鍔犱簡鐢ㄦ埛鐨勪娇鐢ㄩ毦搴︺€傜敱浜庨泦缇や腑寮傛瀯GPU璁＄畻鑳藉姏鍜屾樉瀛樼殑宸紓锛岄潤鎬佺殑骞惰鍖栫瓥鐣ユ棤娉曞厖鍒嗗埄鐢ㄥ紓鏋勮祫婧愬疄鐜伴珮鏁堣缁冦€
閽堝杩欎簺闂锛學hale鎶借薄骞跺畾涔変簡涓や釜鍒嗗竷寮忓師璇(replicate鍜宻plit) 銆傜敤鎴峰彲浠ラ€氳繃鍦ㄦā鍨嬩笂娣诲姞鍑犺鍘熻鏍囪锛屽嵆鏉ヨ〃杈惧拰杞崲鍚勭骞惰鍖栫瓥鐣ュ強鍏剁粍鍚堬紝鏋佸ぇ闄嶄綆浜嗗垎甯冨紡妗嗘灦鐨勪娇鐢ㄩ棬妲涖€俉hale runtime灏嗙敤鎴风殑鏍囪淇℃伅铻嶅悎鍒拌绠楀浘涓紝鑷姩瀹屾垚妯″瀷鐨勫苟琛屽寲浼樺寲銆傚悓鏃禬hale鎻愪緵浜嗗熀浜庣‖浠舵劅鐭ョ殑鑷姩鍖栧垎甯冨紡骞惰绛栫暐锛屼紭鍖栧湪寮傛瀯GPU闆嗙兢涓婂垎甯冨紡璁粌鎬ц兘銆俉hale鐨勮璁″緢濂藉湴骞宠 浜嗘ā鍨嬬敤鎴风殑骞查鍜岀郴缁熶紭鍖栨満浼氾紝璁╂瘡涓€涓畻娉曞伐绋嬪笀閮借兘杞绘澗楂樻晥璁粌鍒嗗竷寮忓ぇ妯″瀷浠诲姟銆傚€熷姪Whale锛岄樋閲岃揪鎽╅櫌M6妯″瀷浣跨敤512寮燰100 GPU鍦10澶╁唴鍗冲彲瀹屾垚10涓囦嚎澶氭ā鎬侀璁粌妯″瀷銆
Whale璧锋簮浜庨樋閲屽唴閮ㄤ笟鍔★紝鍦ㄩ樋閲屽唴閮ㄥ凡缁忔敮鎸佸浘鍍忋€佹帹鑽愩€佽闊炽€佽棰戙€佽嚜鐒惰瑷€銆佸妯℃€佺瓑涓氬姟鍦烘櫙銆傚苟涓旇兘鏃犳劅鏀寔妯″瀷瑙勬ā妯悜鎵╁睍锛屾渶澶у畬鎴愪簡10涓囦嚎瑙勬ā鐨凪6妯″瀷璁粌銆傚悓鏃禬hale宸茬粡闆嗘垚鍦ㄩ樋閲屼簯鏈哄櫒瀛︿範骞冲彴PAI涓紝澶у鍙互鍦ㄩ樋閲屼簯PAI骞冲彴涓婁娇鐢╓hale銆傛満鍣ㄥ涔犲钩鍙癙AI鏄潰鍚戝紑鍙戣€呭拰浼佷笟鐨凙I宸ョ▼鍖栧钩鍙帮紝鎻愪緵浜嗚鐩栨暟鎹噯澶囥€佹ā鍨嬪紑鍙戙€佹ā鍨嬭缁冦€佹ā鍨嬮儴缃茬殑鍏ㄦ祦绋嬫湇鍔°€
Whale(EPL)寮€婧愬湴鍧€(https://github.com/alibaba/EasyParallelLibrary)锛屾杩庡ぇ瀹舵潵璇曠敤鍜岀粰鍑哄缓璁€
璁烘枃鍚嶇О锛歐hale: Efficient Giant Model Training over Heterogeneous GPUs
璁烘枃浣滆€咃細璐捐搐鑹筹紝姹熶箰锛岀帇鏄傦紝鑲栨枃鑱紝鐭冲瓙楠ワ紝寮犳澃锛屾潕鏄曞厓锛岄檲娴煶锛屾潕姘革紝閮戠ク锛屽垬灏忓媷锛屾灄浼
寮€婧愰摼鎺ワ細https://github.com/alibaba/easyparallellibrary
璁烘枃閾炬帴锛歨ttps://www.usenix.org/conference/atc22/presentation/jia-xianyan
浜哄伐鏅鸿兘鎶€鏈綉 鍊″灏婇噸涓庝繚鎶ょ煡璇嗕骇鏉冦€傚鍙戠幇鏈珯鏂囩珷瀛樺湪鐗堟潈绛夐棶棰橈紝鐑﹁30澶╁唴鎻愪緵鐗堟潈鐤戦棶銆佽韩浠借瘉鏄庛€佺増鏉冭瘉鏄庛€佽仈绯绘柟寮忕瓑鍙戦偖浠惰嚦1851688011@qq.com鎴戜滑灏嗗強鏃舵矡閫氫笌澶勭悊銆傦紒锛棣栭〉 > 鏂伴椈 » 闃块噷浜戞満鍣ㄥ涔犲钩鍙癙AI璁烘枃楂樻晥澶фā鍨嬭缁冩鏋禬hale鍏ラ€塙SENIX ATC'22