# Run\_mlm.py cuda error memory after resuming a training

**URL:** <https://discuss.huggingface.co/t/run-mlm-py-cuda-error-memory-after-resuming-a-training/5549>\
**Category:** 🤗Transformers\
**Created:** [April 16, 2021, 9:54am UTC](https://discuss.huggingface.co/t/run-mlm-py-cuda-error-memory-after-resuming-a-training/5549 "2021-04-16T09:54:11Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![IamAdiSri](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/iamadisri/32/20434_2.png) [@IamAdiSri](https://discuss.huggingface.co/u/IamAdiSri)\
**Post date:** [April 17, 2021, 4:09am UTC](https://discuss.huggingface.co/t/run-mlm-py-cuda-error-memory-after-resuming-a-training/5549/2 "2021-04-17T04:09:58Z")

</div>

I’m running into the same issue but with the mBART model. For some reason, running training from scratch with the `Seq2SeqTrainer` works just fine, but resuming from checkpoint exceeds the memory limit, and produces a CUDA ‘out of memory’ error.

I think it might be related to [this](https://github.com/huggingface/transformers/issues/3730) issue on the GitHub repository.

@sshleifer I think this is another issue with training large models, as we discussed [here](https://discuss.huggingface.co/t/pruning-a-model-embedding-matrix-for-memory-efficiency/5502) although this just seems to be a bug in the trainer.

---

_[View the full topic](https://discuss.huggingface.co/t/run-mlm-py-cuda-error-memory-after-resuming-a-training/5549)._
