# Weight decay rate in create optimizer tensorflow

**URL:** <https://discuss.huggingface.co/t/weight-decay-rate-in-create-optimizer-tensorflow/16547>\
**Category:** Intermediate\
**Created:** [April 6, 2022, 5:12pm UTC](https://discuss.huggingface.co/t/weight-decay-rate-in-create-optimizer-tensorflow/16547 "2022-04-06T17:12:45Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gozdi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gozdi/32/4221_2.png) [@Gozdi](https://discuss.huggingface.co/u/Gozdi)\
**Post date:** [April 6, 2022, 5:12pm UTC](https://discuss.huggingface.co/t/weight-decay-rate-in-create-optimizer-tensorflow/16547/1 "2022-04-06T17:12:45Z")

</div>

Hi how dose weight decay rate affect learning rate?  
From the documentation i got that lr\_schedule is created like that and then it is passed to AdamWeightDecay as learning\_rate argument

```auto
lr_schedule = tf.keras.optimizers.schedules.PolynomialDecay(
        initial_learning_rate=init_lr,
        decay_steps=num_train_steps - num_warmup_steps,
        end_learning_rate=init_lr * min_lr_ratio,
    )

```

it doesn’t use weight decay rate and if I were to plot this schedule i would be just a straight line going from init\_lr to end\_lr

weight decay rate is used later in AdamWeightDecay class

```auto
def _decay_weights_op(self, var, learning_rate, apply_state):
        do_decay = self._do_use_weight_decay(var.name)
        if do_decay:
            return var.assign_sub(
                learning_rate * var * apply_state[(var.device, var.dtype.base_dtype)]["weight_decay_rate"],
                use_locking=self._use_locking,
            )
        return tf.no_op()

```

so is weight\_decay\_rate just another scalar that scales learning rate for particular training step??

for example if my lr\_schedule created by tensorflow would look like this `[0.1, 0.09, 0.08, 0.07, 0.06, 0.05, 0.04, 0.03, 0.02, 0.01]`

and weight\_decay\_rate=0.01 then final learning rate would look like this `[0.001, 0.0009, 0.0008, 0.0007, 0.0006, 0.0005, 0.0004, 0.0003, 0.0002, 0.0001]`???  
Am i getting this right?