# Load a single GPU checkpoint to 2 GPUS (deepspeed)

**URL:** <https://discuss.huggingface.co/t/load-a-single-gpu-checkpoint-to-2-gpus-deepspeed/19806>\
**Category:** Intermediate\
**Created:** [June 29, 2022, 6:33pm UTC](https://discuss.huggingface.co/t/load-a-single-gpu-checkpoint-to-2-gpus-deepspeed/19806 "2022-06-29T18:33:51Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ierezell](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ierezell/32/2517_2.png) [@ierezell](https://discuss.huggingface.co/u/ierezell)\
**Post date:** [June 29, 2022, 6:33pm UTC](https://discuss.huggingface.co/t/load-a-single-gpu-checkpoint-to-2-gpus-deepspeed/19806/1 "2022-06-29T18:33:51Z")

</div>

Hello 🙂

When training a model on one GPU with deep speed, I was able to save the model and to convert the weights to a `pytorch_model.bin` with the given `zero_to_fp32.py` script and train again.  
(see here : [[Solved] Cannot restart training from deepspeed checkpoint](https://discuss.huggingface.co/t/solved-cannot-restart-training-from-deepspeed-checkpoint/19786))

However, I’m not able to load again this model and train it this time on two gpus :

```auto
deepspeed.runtime.zero.utils.ZeRORuntimeException: The checkpoint being loaded used a DP world size of 1 but the current world size is 2. Automatic adjustment of ZeRO's optimizer state partitioning with a new world size is not currently supported.

```

The error seems clear but is there any way to load the model (CPU or GPU) like for inference and _then_ split it among the two gpus ?

Thanks in advance,  
Have a great day
