# Segmentation fault with gradient\_checkpointing on multiGPU

**URL:** <https://discuss.huggingface.co/t/segmentation-fault-with-gradient-checkpointing-on-multigpu/51496>\
**Category:** Beginners\
**Created:** [August 21, 2023, 9:13am UTC](https://discuss.huggingface.co/t/segmentation-fault-with-gradient-checkpointing-on-multigpu/51496 "2023-08-21T09:13:14Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![mirix](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mirix/32/16327_2.png) [@mirix](https://discuss.huggingface.co/u/mirix)\
**Post date:** [August 21, 2023, 9:13am UTC](https://discuss.huggingface.co/t/segmentation-fault-with-gradient-checkpointing-on-multigpu/51496/1 "2023-08-21T09:13:14Z")

</div>

I am trying to train a wav2vec2 model on my own dataset by following [this template](https://huggingface.co/blog/fine-tune-wav2vec2-english).

I have two issues:

1. The model does not seem to be learning much. I have tried different learning rates and I see differences, but not good enough.

2. If I set gradient\_checkpointing=True the training segfaults (core dumped) when CUDA\_VISIBLE\_DEVICES is set to more than one GPU (single node). With just one GPU it is OK, no matter which one. And if gradient\_checkpointing is not set, the training can take advantage of multiple GPUs. Is this a known issue/feature? Are there any extra options that need to be set?

I am running Cuda 12.1 with the latest driver and the nightly developing version on Pytorch on two RTX 4090.

---

<div class="post-metadata">

**Author:** ![rhasan](https://avatars.discourse-cdn.com/v4/letter/r/8e7dd6/32.png) [@rhasan](https://discuss.huggingface.co/u/rhasan)\
**Post date:** [September 5, 2024, 5:24am UTC](https://discuss.huggingface.co/t/segmentation-fault-with-gradient-checkpointing-on-multigpu/51496/2 "2024-09-05T05:24:19Z")

</div>

Hi, did you get it solved by any chance?

Unfortunately, I’m also getting a Seg fault with multi-GPU. Turning off gradient checkpointing is not solving the issue either.  
I’m using AMD GPU with the following packages in Python 3.12.3:  
pytorch-triton-rocm 3.0.0  
torch 2.4.0+rocm6.1  
transformers 4.44.2  
datasets 2.21.0
