# Cannot Resume Training

**URL:** <https://discuss.huggingface.co/t/cannot-resume-training/2823>\
**Category:** Beginners\
**Created:** [December 14, 2020, 9:55pm UTC](https://discuss.huggingface.co/t/cannot-resume-training/2823 "2020-12-14T21:55:03Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![david-waterworth](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@david-waterworth](https://discuss.huggingface.co/u/david-waterworth)\
**Post date:** [December 14, 2020, 9:55pm UTC](https://discuss.huggingface.co/t/cannot-resume-training/2823/1 "2020-12-14T21:55:03Z")

</div>

I’m trying to resume training using a checkpoint with RobertaForMaskedLM

I’m using the same script I trained except at the last stage I call `trainer.train("checkpoint-200000")`, i.e. the model is created as usual from config and the tokenizer loaded from disk.

The trainer prints the message below, but then the loss isn’t consistent with the original training. At this stage in the original training the loss was around 0.6, on resume it drops to 0.0004 and then 7.85

This makes me not trust reloading the trained model as I have no confidence I’ve correctly trained and serialised it. What could I be doing wrong?

> \*\*\*\*\* Running training \*\*\*\*\*  
> Num examples = 1061833  
> Num Epochs = 50  
> Instantaneous batch size per device = 256  
> Total train batch size (w. parallel, distributed & accumulation) = 256  
> Gradient Accumulation steps = 1  
> Total optimization steps = 207400  
> Continuing training from checkpoint, will skip to saved global\_step  
> Continuing training from epoch 48  
> Continuing training from global step 200000  
> Will skip the first 896 batches in the first epoch

> {‘loss’: 0.00040710282906264845, ‘learning\_rate’: 1.7815814850530375e-06, ‘epoch’: 48.21841851494696}  
> {‘loss’: 7.853060150146485, ‘learning\_rate’: 1.7791706846673097e-06, ‘epoch’: 48.22082931533269}  
> {‘loss’: 7.491885375976563, ‘learning\_rate’: 1.7767598842815817e-06, ‘epoch’: 48.22324011571842}

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [December 15, 2020, 12:30am UTC](https://discuss.huggingface.co/t/cannot-resume-training/2823/2 "2020-12-15T00:30:22Z")

</div>

The loss you were at isn’t saved so you can’t trust what it tells you when training resumes. First it’s not an average of all the losses since the beginning of the training, just since you restarted. Then the first time it’s logged, it’s divided by the wrong number (a number way too big) which is why you have that low loss.

I’ll look at this tomorrow and see if we can have the same losses printed in a full training and a resumed training.