# Continue LM pretraining with run\_mlm - loss function clarification

**URL:** <https://discuss.huggingface.co/t/continue-lm-pretraining-with-run-mlm-loss-function-clarification/15700>\
**Category:** 🤗Transformers\
**Created:** [March 14, 2022, 6:40pm UTC](https://discuss.huggingface.co/t/continue-lm-pretraining-with-run-mlm-loss-function-clarification/15700 "2022-03-14T18:40:14Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![dalia](https://avatars.discourse-cdn.com/v4/letter/d/a698b9/32.png) [@dalia](https://discuss.huggingface.co/u/dalia)\
**Post date:** [March 14, 2022, 6:40pm UTC](https://discuss.huggingface.co/t/continue-lm-pretraining-with-run-mlm-loss-function-clarification/15700/1 "2022-03-14T18:40:14Z")

</div>

I’m trying to use Huggingface’s tensorflow run\_mlm.py script to continue pretraining a bert model, and didn’t understand the following: in the above script, the model is loaded using from\_pretrained and then compiled with a dummy\_loss function before running model.fit(…). The dummy\_loss function defined in the script ignores y\_true and simply returns the mean of y\_pred. Is this loss function overridden somehow? I can’t understand how the script actually continues pretraining if this is the only loss function used.