mindspore.nn.LambпғҒ

class mindspore.nn.Lamb(params, learning_rate, beta1=0.9, beta2=0.999, vpn梯子 eps=1e-6, weight_decay=0.0)[жәҗд»Јз Ғ]пғҒ

LAMBпјҲLayer-wise Adaptive Moments optimizer for Batching trainingпјҢз”ЁдәҺжү№и®ӯз»ғзҡ„еҲҶеұӮиҮӘйҖӮеә”зҹ©дјҳеҢ–еҷЁпјүз®—жі•зҡ„е®һзҺ°гҖӮ

LAMBжҳҜдёҖз§ҚйҮҮз”ЁеҲҶеұӮиҮӘйҖӮеә”жү№дјҳеҢ–жҠҖжңҜзҡ„дјҳеҢ–з®—жі•гҖӮиҜҰи§Ғи®әж–Ү LARGE BATCH OPTIMIZATION FOR DEEP LEARNING: TRAINING BERT IN 76 MINUTESгҖӮ

LAMBдјҳеҢ–еҷЁж—ЁеңЁдёҚйҷҚдҪҺзІҫеәҰзҡ„жғ…еҶөдёӢеўһеҠ и®ӯз»ғbatch sizeпјҢж”ҜжҢҒиҮӘйҖӮеә”йҖҗе…ғзҙ жӣҙж–°е’ҢзІҫзЎ®зҡ„еҲҶеұӮж ЎжӯЈгҖӮ

еҸӮж•°жӣҙж–°еҰӮдёӢпјҡ

\[\begin{split}\begin{array}{l} vpn free &\newline &\hline \\ vpn梯子 免费 免费的vpn梯子 &\textbf{Parameters}: \: vpn梯子 免费 1^{\text {st }}\text {moment vector} \: m , \: 2^{\text vpn永久免费梯子 {nd}} \: vpn free \text{moment vector} vpn梯子 免费 \: v , \\ &\hspace{5mm}\text{learning rate } \left\{ \gamma_{t}\right\}_{t=1}^{T} , \: \text {exponential decay rates for the moment estimates} \: \beta_{1} \: \beta_{2} , \\ &\hspace{5mm}\text{scaling 免费的vpn梯子 function } \phi vpn梯子 免费 \\ &\textbf{Init}: \boldsymbol{m}_{0} vpn梯子 \leftarrow 0, \: \boldsymbol{v}_{0} \leftarrow 0 \\[-1.ex] &\newline &\hline \\ vpn梯子 免费的vpn梯子 &\textbf{for} \text { t=1 vpn梯子 to vpn free T } \textbf{do} \\ &\hspace{5mm}\text{Draw b samples vpn梯子 免费 } S_{t} \text{ from } \mathbb{P} \text{ . } \\ &\hspace{5mm}\text{Compute } g_{t}=\frac{1}{\left|\mathcal{S}_{t}\right|} \sum_{s_{t} \in \mathcal{S}_{t}} vpn梯子 \nabla vpn free \ell\left(x_{t}, s_{t}\right) . \\ &\hspace{5mm}\boldsymbol{m}_{t} \leftarrow \beta_{1} \boldsymbol{m}_{t-1}+\left(1-\beta_{1}\right) \boldsymbol{g}_{t} \\ vpn梯子 免费 &\hspace{5mm}\boldsymbol{v}_{t} \leftarrow \beta_{2} \boldsymbol{v}_{t-1}+\left(1-\beta_{2}\right) \boldsymbol{g}_{t}^{2} \\ &\hspace{5mm}\hat{\boldsymbol{m}}_{t} \leftarrow \boldsymbol{m}_{t} /\left(1-\beta_{1}^{t}\right) \\ &\hspace{5mm}\hat{\boldsymbol{v}}_{t} \leftarrow \boldsymbol{v}_{t} /\left(1-\beta_{2}^{t}\right) \\ vpn梯子 免费 &\hspace{5mm}\text{Compute ratio } \boldsymbol{r}_{t}=\hat{\boldsymbol{m}}_{t} vpn free /(\sqrt{\hat{\boldsymbol{v}}_{t}}+\epsilon) \\ &\hspace{5mm}\boldsymbol{w}_{t+1}^{(i)}=\boldsymbol{w}_{t}^{(i)}- \gamma_{t} \frac{\boldsymbol{\phi}\left(\left\|\boldsymbol{w}_{t}^{(i)}\right\|\right)} {\left\|\boldsymbol{r}_{t}^{(i)}+\lambda \boldsymbol{w}_{t}^{(i)}\right\|}\left(\boldsymbol{r}_{t}^{(i)}+ \lambda \boldsymbol{w}_{t}^{(i)}\right) \\ &\textbf{end for} \\[-1.ex] &\newline 免费的vpn梯子 &\hline \\[-1.ex] vpn梯子 免费 &\textbf{return} \: \boldsymbol{w}_{t+1}\\[-1.ex] &\newline &\hline \\[-1.ex] \end{array}\end{split}\]

е…¶дёӯпјҢ \(m\) 代表第дёҖдёӘеҠЁйҮҸзҹ©йҳө moment1 vpn梯子 免费 пјҢ\(v\) 代表第дәҢдёӘеҠЁйҮҸзҹ©йҳө moment2 пјҢ\(g\) д»ЈиЎЁжўҜеәҰ gradients пјҢ\(\gamma\) д»ЈиЎЁеӯҰд№ зҺҮ 免费的vpn梯子 learning_rateпјҢ\(\beta_1, \beta_2\) д»ЈиЎЁиЎ°еҮҸйҖҹзҺҮ beta1 е’Ң beta2 пјҢ\(t\) д»ЈиЎЁеҪ“еүҚstepпјҢ\(beta_1^t\) е’Ң \(beta_2^t\) д»ЈиЎЁ beta1 е’Ң beta2 зҡ„tж¬Ўж–№ пјҢ \(w\) д»ЈиЎЁ params пјҢ \(\epsilon\) д»ЈиЎЁ epsпјҢ \(\lambda\) иЎЁзӨәLAMBжқғйҮҚиЎ°еҮҸзҺҮпјҢеҜ№еә”weight_decayгҖӮ

иӯҰе‘Ҡ

  • иҜҘжҺҘеҸЈе·ІеәҹејғпјҢ2.9.0пјҲдёҚеҗ«пјүд№ӢеҗҺзүҲжң¬е°ҶеҲ йҷӨгҖӮ

  • LambдјҳеҢ–еҷЁзҡ„жӣҙж–°иҝҮзЁӢдёҚжҳҜе®Ңе…Ёelement-wiseзҡ„пјҢеҲҶеёғејҸ并иЎҢдёӢеҜ№жқғйҮҚиҝӣиЎҢеҲҮеҲҶеҸҜиғҪдјҡеҪұе“Қжӣҙж–°з»“жһңгҖӮ

иҜҙжҳҺ

  • еңЁеҸӮж•°жңӘеҲҶз»„ж—¶пјҢдјҳеҢ–еҷЁй…ҚзҪ®зҡ„ weight_decay еә”з”ЁдәҺеҗҚз§°дёҚеҗ«"beta"жҲ–"gamma"зҡ„зҪ‘з»ңеҸӮж•°гҖӮ

  • з”ЁжҲ·еҸҜд»ҘеҲҶз»„и°ғж•ҙжқғйҮҚиЎ°еҮҸзӯ–з•ҘгҖӮеҲҶз»„ж—¶пјҢжҜҸз»„зҪ‘з»ңеҸӮж•°еқҮеҸҜй…ҚзҪ® weight_decay гҖӮиӢҘжңӘй…ҚзҪ®пјҢеҲҷиҜҘз»„зҪ‘з»ңеҸӮж•°дҪҝз”ЁдјҳеҢ–еҷЁдёӯй…ҚзҪ®зҡ„ weight_decay гҖӮ

дјҳеҢ–еҷЁе’Ңж··еҗҲзІҫеәҰд№Ӣй—ҙйҖҡеёёжІЎжңүиҒ”зі»гҖӮдҪҶжҳҜпјҢеҪ“дҪҝз”Ё FixedLossScaleManager дё” FixedLossScaleManager дёӯзҡ„ drop_overflow_update и®ҫзҪ®дёәFalseж—¶пјҢдјҳеҢ–еҷЁйңҖиҰҒи®ҫзҪ®'loss_scale'гҖӮ

з”ұдәҺжӯӨдјҳеҢ–еҷЁжІЎжңү loss_scale зҡ„еҸӮж•°пјҢеӣ жӯӨйңҖиҰҒйҖҡиҝҮе…¶д»–ж–№ејҸеӨ„зҗҶ loss_scale гҖӮ

еҸӮж•°пјҡ

  • params (Union[list[Parameter], list[dict]]) - еҝ…йЎ»жҳҜ Parameter з»„жҲҗзҡ„еҲ—иЎЁжҲ–еӯ—е…ёз»„жҲҗзҡ„еҲ—иЎЁгҖӮеҪ“еҲ—иЎЁе…ғзҙ жҳҜеӯ—е…ёж—¶пјҢеӯ—е…ёзҡ„й”®еҸҜд»ҘжҳҜ"params"гҖҒ"lr"гҖҒ"weight_decay"гҖҒ"grad_centralization"е’Ң"order_params"пјҡ

    • params - еҝ…еЎ«гҖӮеҪ“еүҚз»„еҲ«зҡ„жқғйҮҚпјҢиҜҘеҖјеҝ…йЎ»жҳҜ Parameter еҲ—иЎЁгҖӮ

    • lr - еҸҜйҖүгҖӮеҰӮжһңй”®дёӯеӯҳеңЁ"lr"пјҢеҲҷдҪҝз”ЁеҜ№еә”зҡ„еҖјдҪңдёәеӯҰд№ зҺҮгҖӮеҰӮжһңжІЎжңүпјҢеҲҷдҪҝз”ЁдјҳеҢ–еҷЁдёӯзҡ„еҸӮж•° learning_rate дҪңдёәеӯҰд№ зҺҮгҖӮж”ҜжҢҒеӣәе®ҡе’ҢеҠЁжҖҒеӯҰд№ зҺҮгҖӮ

    • weight_decay - еҸҜйҖүгҖӮеҰӮжһңй”®дёӯеӯҳеңЁ"weight_decayвҖқпјҢеҲҷдҪҝз”ЁеҜ№еә”зҡ„еҖјдҪңдёәжқғйҮҚиЎ°еҮҸеҖјгҖӮеҰӮжһңжІЎжңүпјҢеҲҷдҪҝз”ЁдјҳеҢ–еҷЁдёӯй…ҚзҪ®зҡ„ weight_decay дҪңдёәжқғйҮҚиЎ°еҮҸеҖјгҖӮ еҖјеҫ—жіЁж„Ҹзҡ„жҳҜпјҢ weight_decay еҸҜд»ҘжҳҜеёёйҮҸпјҢд№ҹеҸҜд»ҘжҳҜCellзұ»еһӢгҖӮCellзұ»еһӢзҡ„weight decayз”ЁдәҺе®һзҺ°еҠЁжҖҒweight decayз®—жі•гҖӮеҠЁжҖҒжқғйҮҚиЎ°еҮҸе’ҢеҠЁжҖҒеӯҰд№ зҺҮзӣёдјјпјҢ з”ЁжҲ·йңҖиҰҒиҮӘе®ҡд№үдёҖдёӘиҫ“е…Ҙдёәglobal stepзҡ„weight_decay_scheduleгҖӮеңЁи®ӯз»ғзҡ„иҝҮзЁӢдёӯпјҢдјҳеҢ–еҷЁдјҡи°ғз”ЁWeightDecayScheduleзҡ„е®һдҫӢжқҘиҺ·еҸ–еҪ“еүҚstepзҡ„weight decayеҖјгҖӮ

    • grad_centralization - еҸҜйҖүгҖӮеҰӮжһңй”®дёӯеӯҳеңЁ"grad_centralization"пјҢеҲҷдҪҝз”ЁеҜ№еә”зҡ„еҖјпјҢиҜҘеҖјеҝ…йЎ»дёәеёғе°”зұ»еһӢгҖӮеҰӮжһңжІЎжңүпјҢеҲҷи®Өдёә grad_centralization дёәFalseгҖӮиҜҘеҸӮж•°д»…йҖӮз”ЁдәҺеҚ·з§ҜеұӮгҖӮ

    • order_params - еҸҜйҖүгҖӮеҖјзҡ„йЎәеәҸжҳҜеҸӮж•°жӣҙж–°зҡ„йЎәеәҸгҖӮеҪ“дҪҝз”ЁеҸӮж•°еҲҶз»„еҠҹиғҪж—¶пјҢйҖҡеёёдҪҝз”ЁиҜҘй…ҚзҪ®йЎ№дҝқжҢҒ parameters зҡ„йЎәеәҸд»ҘжҸҗеҚҮжҖ§иғҪгҖӮеҰӮжһңй”®дёӯеӯҳеңЁ"order_params"пјҢеҲҷдјҡеҝҪз•ҘиҜҘз»„й…ҚзҪ®дёӯзҡ„е…¶д»–й”®гҖӮ"order_params"дёӯзҡ„еҸӮж•°еҝ…йЎ»еңЁжҹҗдёҖз»„ params еҸӮж•°дёӯгҖӮ

  • learning_rate 免费的vpn梯子 (Union[float, int, Tensor, Iterable, LearningRateSchedule]) -

    • float - еӣәе®ҡзҡ„еӯҰд№ зҺҮгҖӮеҝ…йЎ»еӨ§дәҺзӯүдәҺйӣ¶гҖӮ

    • int - еӣәе®ҡзҡ„еӯҰд№ зҺҮгҖӮеҝ…йЎ»еӨ§дәҺзӯүдәҺйӣ¶гҖӮж•ҙж•°зұ»еһӢдјҡиў«иҪ¬жҚўдёәжө®зӮ№ж•°гҖӮ

    • Tensor - еҸҜд»ҘжҳҜж ҮйҮҸжҲ–дёҖз»ҙеҗ‘йҮҸгҖӮж ҮйҮҸжҳҜеӣәе®ҡзҡ„еӯҰд№ зҺҮгҖӮдёҖз»ҙеҗ‘йҮҸжҳҜеҠЁжҖҒзҡ„еӯҰд№ зҺҮпјҢ第iжӯҘе°ҶеҸ–еҗ‘йҮҸдёӯ第iдёӘеҖјдҪңдёәеӯҰд№ зҺҮгҖӮ

    • Iterable - еҠЁжҖҒзҡ„еӯҰд№ зҺҮгҖӮ第iжӯҘе°ҶеҸ–иҝӯд»ЈеҷЁз¬¬iдёӘеҖјдҪңдёәеӯҰд№ зҺҮгҖӮ

    • LearningRateSchedule - еҠЁжҖҒзҡ„еӯҰд№ зҺҮгҖӮеңЁи®ӯз»ғиҝҮзЁӢдёӯпјҢдјҳеҢ–еҷЁе°ҶдҪҝз”ЁжӯҘж•°пјҲstepпјүдҪңдёәиҫ“е…ҘпјҢи°ғз”Ё LearningRateSchedule е®һдҫӢжқҘи®Ўз®—еҪ“еүҚеӯҰд№ зҺҮгҖӮ

  • beta1 (float) - 第дёҖзҹ©зҡ„жҢҮж•°иЎ°еҮҸзҺҮгҖӮеҸӮж•°иҢғеӣҙпјҲ0.0,1.0пјүгҖӮй»ҳи®ӨеҖјпјҡ 0.9 гҖӮ

  • beta2 (float) - 第дәҢзҹ©зҡ„жҢҮж•°иЎ°еҮҸзҺҮгҖӮеҸӮж•°иҢғеӣҙпјҲ0.0,1.0пјүгҖӮй»ҳи®ӨеҖјпјҡ 0.999 гҖӮ

  • eps (float) - е°Ҷж·»еҠ еҲ°еҲҶжҜҚдёӯпјҢд»ҘжҸҗй«ҳж•°еҖјзЁіе®ҡжҖ§гҖӮеҝ…йЎ»еӨ§дәҺ0гҖӮй»ҳи®ӨеҖјпјҡ 1e-6 гҖӮ

  • weight_decay (Union[float, int, Cell]) - жқғйҮҚиЎ°еҮҸпјҲL2 vpn永久免费梯子 penaltyпјүгҖӮй»ҳи®ӨеҖјпјҡ 0.0 гҖӮ

    • float: еӣәе®ҡеҖјпјҢеҝ…йЎ»еӨ§дәҺжҲ–иҖ…зӯүдәҺ0гҖӮ

    • int: еӣәе®ҡеҖјпјҢеҝ…йЎ»еӨ§дәҺжҲ–иҖ…зӯүдәҺ0пјҢдјҡиў«иҪ¬жҚўжҲҗfloatгҖӮ

    • Cell: еҠЁжҖҒweight decayгҖӮеңЁи®ӯз»ғиҝҮзЁӢдёӯпјҢдјҳеҢ–еҷЁдјҡдҪҝз”ЁжӯҘж•°пјҲstepпјүдҪңдёәиҫ“е…ҘпјҢи°ғз”ЁиҜҘCellе®һдҫӢжқҘи®Ўз®—еҪ“еүҚweight decayеҖјгҖӮ

иҫ“е…Ҙпјҡ

  • gradients (tuple[Tensor]) - params зҡ„жўҜеәҰпјҢshapeдёҺ vpn永久免费梯子 params зӣёеҗҢгҖӮ

иҫ“еҮәпјҡ

tuple[bool]пјҢжүҖжңүе…ғзҙ йғҪдёәTrueгҖӮ

ејӮеёёпјҡ

  • TypeError - learning_rate дёҚжҳҜintгҖҒfloatгҖҒTensorгҖҒIterableжҲ–LearningRateScheduleгҖӮ

  • TypeError - parameters зҡ„е…ғзҙ дёҚжҳҜParameterжҲ–dictгҖӮ

  • TypeError - beta1гҖҒbeta2 vpn梯子 жҲ– eps дёҚжҳҜfloatгҖӮ

  • TypeError - weight_decay дёҚжҳҜfloatжҲ–intгҖӮ

  • ValueError - eps е°ҸдәҺзӯүдәҺ0гҖӮ

  • ValueError vpn free - beta1гҖҒbeta2 дёҚеңЁпјҲ0.0,1.0пјүиҢғеӣҙеҶ…гҖӮ

  • ValueError - weight_decay е°ҸдәҺ0гҖӮ

ж”ҜжҢҒе№іеҸ°пјҡ

Ascend GPU

ж ·дҫӢпјҡ

>>> vpn梯子 import mindspore as ms
>>> from mindspore vpn free import nn
>>>
>>> # Define the network structure of LeNet5. Refer to
>>> # https://atomgit.com/mindspore/docs/blob/r2.9.0/docs/mindspore/code/lenet.py
>>> net = LeNet5()
>>> #1) All parameters use the same learning rate and weight decay
>>> optim = nn.Lamb(params=net.trainable_params(), learning_rate=0.1)
>>>
>>> #2) Use parameter groups vpn永久免费梯子 and set different vpn梯子 免费 values
>>> poly_decay_lr = nn.PolynomialDecayLR(learning_rate=0.1, end_learning_rate=0.01,
...    vpn梯子 免费  vpn梯子 免费            免费的vpn梯子                 免费的vpn梯子                     decay_steps=4, power = 0.5)
>>> conv_params = list(filter(lambda x: 'conv' in x.name, net.trainable_params()))
>>> no_conv_params = list(filter(lambda x: 'conv' not in x.name, net.trainable_params()))
>>> group_params = [{'params': conv_params, 'weight_decay': 0.01, 'grad_centralization':True},
...                 vpn free {'params': no_conv_params, 'lr': poly_decay_lr},
... vpn梯子 免费        vpn永久免费梯子          {'order_params': net.trainable_params(0.01)}]
>>> optim = nn.Lamb(group_params, learning_rate=0.1, weight_decay=0.0)
>>> # The conv_params's parameters will use default learning rate of 0.1 vpn永久免费梯子 and weight decay of 0.01 and grad
>>> # centralization of True.
>>> # The no_conv_params's parameters will use dynamic learning rate of poly decay learning rate and default
>>> # weight decay of 0.0 and grad centralization of False.
>>> # The final parameters order in vpn梯子 vpn永久免费梯子 which the optimizer will be followed vpn梯子 免费 is the value of 'order_params'.
>>>
>>> vpn梯子 免费 loss = nn.SoftmaxCrossEntropyWithLogits()
>>> model = ms.train.Model(net, loss_fn=loss, optimizer=optim)