# AdamW implementation

**URL:** <https://discuss.huggingface.co/t/adamw-implementation/8426>\
**Category:** Beginners\
**Created:** [July 16, 2021, 11:14am UTC](https://discuss.huggingface.co/t/adamw-implementation/8426 "2021-07-16T11:14:57Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yuti](https://avatars.discourse-cdn.com/v4/letter/y/cdc98d/32.png) [@Yuti](https://discuss.huggingface.co/u/Yuti)\
**Post date:** [July 16, 2021, 11:14am UTC](https://discuss.huggingface.co/t/adamw-implementation/8426/1 "2021-07-16T11:14:57Z")

</div>

Hi,  
I was looking at the 🤗 [implementation of the AdamW optimizer](https://huggingface.co/transformers/_modules/transformers/optimization.html#AdamW) and I didn’t understand why you put the weight decay at the end.

Shouldn’t you swap between this line:  
`p.data.addcdiv_(exp_avg, denom, value=-step_size)`

and the weight decay part?  
Thanks.

The AdamW algorithm from the “DECOUPLED WEIGHT DECAY REGULARIZATION” paper & The relevant source code for transformers.AdamW:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/7/7926bf3548661da1102fa39fc4b5cdb6dd12f64d.png)

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [July 16, 2021, 2:39pm UTC](https://discuss.huggingface.co/t/adamw-implementation/8426/2 "2021-07-16T14:39:28Z")

</div>

The two lines substract an independent thing to the model parameters, so executing them in any order will give the same results.

---

<div class="post-metadata">

**Author:** ![Yuti](https://avatars.discourse-cdn.com/v4/letter/y/cdc98d/32.png) [@Yuti](https://discuss.huggingface.co/u/Yuti)\
**Post date:** [July 17, 2021, 10:39am UTC](https://discuss.huggingface.co/t/adamw-implementation/8426/3 "2021-07-17T10:39:48Z")

</div>

Thanks for the fast reply 🙂 .

I’m not sure why they are independent.  
The weight decay line is adding the parameter multiplied by `-lr*wd` to the parameter itself, so it depends on the parameter’s data.

The first line is adding a value to the parameter’s data i.e. changing it.  
So the first line is affecting the value to be added in the second line which makes them dependent.

In the following image `x` denotes the parameter’s data, `a` denotes the value added in line 1 and `b` replaces `lr*wd`

![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/1/1da9dd7347712c2e6654e582ea3ee2843ff3af30.png)

So the updated `x` is different in the two versions.

Thanks again.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [July 17, 2021, 2:08pm UTC](https://discuss.huggingface.co/t/adamw-implementation/8426/4 "2021-07-17T14:08:09Z")

</div>

Ah yes you’re right, it is indeed in the wrong place and should be put first, thanks for the lengthy explanation, I was a bit too tired when I read your post first 🙂

However there is a line `# Add weight decay at the end (fixed version)` in the code introduced by @thomwolf so there is probably a reason this is put in second.

---

<div class="post-metadata">

**Author:** ![Yuti](https://avatars.discourse-cdn.com/v4/letter/y/cdc98d/32.png) [@Yuti](https://discuss.huggingface.co/u/Yuti)\
**Post date:** [July 17, 2021, 6:29pm UTC](https://discuss.huggingface.co/t/adamw-implementation/8426/5 "2021-07-17T18:29:50Z")

</div>

Oh that’s totally fine 🙃 thanks for your replies!  
I’ll wait for @thomwolf’s response.