# Run\_mlm.py cuda error memory after resuming a training

**URL:** <https://discuss.huggingface.co/t/run-mlm-py-cuda-error-memory-after-resuming-a-training/5549>\
**Category:** 🤗Transformers\
**Created:** [April 16, 2021, 9:54am UTC](https://discuss.huggingface.co/t/run-mlm-py-cuda-error-memory-after-resuming-a-training/5549 "2021-04-16T09:54:11Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shiro](https://avatars.discourse-cdn.com/v4/letter/s/7bcc69/32.png) [@Shiro](https://discuss.huggingface.co/u/Shiro)\
**Post date:** [April 16, 2021, 9:54am UTC](https://discuss.huggingface.co/t/run-mlm-py-cuda-error-memory-after-resuming-a-training/5549/1 "2021-04-16T09:54:11Z")

</div>

Hello,

I am trying to pretrain an XLM Roberta model, but I got some issue. I have trained the model for few thousand steps and get the checkpoint. Then I wanted to continue the pre-training from the checkpoint, but got a memory error “CUDA error memory” after few steps. I wonder if there is a leak of something like that ? The memory used during the 1st pretraining used around 15.X GB / 16.2 GB so I quite don’t understand what’s going on.

---

<div class="post-metadata">

**Author:** ![IamAdiSri](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/iamadisri/32/20434_2.png) [@IamAdiSri](https://discuss.huggingface.co/u/IamAdiSri)\
**Post date:** [April 17, 2021, 4:09am UTC](https://discuss.huggingface.co/t/run-mlm-py-cuda-error-memory-after-resuming-a-training/5549/2 "2021-04-17T04:09:58Z")

</div>

I’m running into the same issue but with the mBART model. For some reason, running training from scratch with the `Seq2SeqTrainer` works just fine, but resuming from checkpoint exceeds the memory limit, and produces a CUDA ‘out of memory’ error.

I think it might be related to [this](https://github.com/huggingface/transformers/issues/3730) issue on the GitHub repository.

@sshleifer I think this is another issue with training large models, as we discussed [here](https://discuss.huggingface.co/t/pruning-a-model-embedding-matrix-for-memory-efficiency/5502) although this just seems to be a bug in the trainer.

---

<div class="post-metadata">

**Author:** ![IamAdiSri](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/iamadisri/32/20434_2.png) [@IamAdiSri](https://discuss.huggingface.co/u/IamAdiSri)\
**Post date:** [April 20, 2021, 12:09pm UTC](https://discuss.huggingface.co/t/run-mlm-py-cuda-error-memory-after-resuming-a-training/5549/3 "2021-04-20T12:09:48Z")

</div>

This fix by @sgugger in a [recent commit](https://github.com/huggingface/transformers/pull/11318) resolved the issue for me. The fix is not in the official release yet, but you can install the bleeding edge version of the Transformers library from source using `pip install git+https://github.com/huggingface/transformers` which pulls it from GitHub.

---

<div class="post-metadata">

**Author:** ![xusky](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/xusky/32/1008_2.png) [@xusky](https://discuss.huggingface.co/u/xusky)\
**Post date:** [April 21, 2021, 2:34pm UTC](https://discuss.huggingface.co/t/run-mlm-py-cuda-error-memory-after-resuming-a-training/5549/4 "2021-04-21T14:34:21Z")

</div>

having the same issue here with GPT-2 large, gonna try what @IamAdiSri suggested

---

<div class="post-metadata">

**Author:** ![xusky](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/xusky/32/1008_2.png) [@xusky](https://discuss.huggingface.co/u/xusky)\
**Post date:** [April 21, 2021, 3:10pm UTC](https://discuss.huggingface.co/t/run-mlm-py-cuda-error-memory-after-resuming-a-training/5549/5 "2021-04-21T15:10:48Z")

</div>

Can confirm that installing the bleeding edge version of transformers as suggested by @IamAdiSri fixed the issue.
