# "No space left on device" when using HuggingFace + SageMaker

**URL:** <https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406>\
**Category:** Amazon SageMaker\
**Created:** [April 10, 2021, 12:07pm UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406 "2021-04-10T12:07:55Z")\
**Posts on this page:** 1\
**Showing post:** 37

<div class="post-metadata">

**Author:** ![Vinayaks117](https://avatars.discourse-cdn.com/v4/letter/v/ebca7d/32.png) [@Vinayaks117](https://discuss.huggingface.co/u/Vinayaks117)\
**Post date:** [April 27, 2022, 6:47am UTC](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406/37 "2022-04-27T06:47:33Z")

</div>

> [@philschmid](#):
>
> P.S. we are trying to help you for free being unpolite has no place here!

@philschmid Sorry, I never meant in that way. Thanks.

Hi All,

My working solution for space issue.

If we don’t want to save all the checkpoints then we can go with an option below.

1. Increase the “volume\_size”, need to set this parameter in HuggingFace estimator.
2. Set the “save\_total\_limit” parameter in TrainingArguments.  
Ex: save\_total\_limit = 2

Which means it will save only 2 checkpoints: best checkpoint and last checkpoint (to make sure we can resume training from it)

[Reference](https://discuss.huggingface.co/t/save-only-best-model-in-trainer/8442/4)

If we want to save all the checkpoints and use it for future analysis then we can go with an option below.

1. Increase the `volume_size`, need to set this parameter in HuggingFace estimator.
2. Need to use `checkpointing`, which saves all checkpoints in `/opt/ml/checkpoints` which is in sync to a s3 bucket defined in the `HuggingFace` estimator.

Set `output_dir` parameter in hyperparameters → ‘output\_dir’:“/opt/ml/checkpoints”  
Set `checkpoint_s3_uri` parameter in `HuggingFace` estimator → checkpoint\_s3\_uri=“s3://sm-pipelines/checkpoints”  
Set `output_dir` parameter in TrainingArguments which allows us to save the checkpoints in “/opt/ml/checkpoints” directory which is in sync with s3 bucket → output\_dir=args.output\_dir

Hope this helps.

---

_[View the full topic](https://discuss.huggingface.co/t/no-space-left-on-device-when-using-huggingface-sagemaker/5406)._
