# Tensor shape mismatch error when doing an allgather in distributed training with FSDP

**URL:** <https://discuss.huggingface.co/t/tensor-shape-mismatch-error-when-doing-an-allgather-in-distributed-training-with-fsdp/142433>\
**Category:** 🤗Transformers\
**Created:** [February 23, 2025, 3:23am UTC](https://discuss.huggingface.co/t/tensor-shape-mismatch-error-when-doing-an-allgather-in-distributed-training-with-fsdp/142433 "2025-02-23T03:23:34Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![tcm03](https://avatars.discourse-cdn.com/v4/letter/t/c0e974/32.png) [@tcm03](https://discuss.huggingface.co/u/tcm03)\
**Post date:** [February 23, 2025, 3:23am UTC](https://discuss.huggingface.co/t/tensor-shape-mismatch-error-when-doing-an-allgather-in-distributed-training-with-fsdp/142433/1 "2025-02-23T03:23:34Z")

</div>

Hi,

I’m finetuning a multimodal LLM and during this process, I encounter the following error when attempting to save the checkpoint. More particularly, I can save the model normally but when the optimizer states are saved, the following error occurs:

```auto
RuntimeError: Detected mismatch between collectives on ranks. Rank 0 is running collective: CollectiveFingerPrint(SequenceNumber=549, OpType=ALLGATHER, TensorShape=[0], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt))), but Rank 2 is running collective: CollectiveFingerPrint(SequenceNumber=549, OpType=ALLGATHER, TensorShape=[183971584], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt))).Collectives differ in the following aspects: Tensor Tensor shapes: 0vs 183971584

```

This is the full traceback of tensor shape mismatch when saving fsdp optimizer states:

```auto
File "/media02/nthuy/miniconda3/envs/thesis_longvu/lib/python3.10/site-packages/transformers/trainer.py", line 2356, in _inner_training_loop
File "/media02/nthuy/miniconda3/envs/thesis_longvu/lib/python3.10/site-packages/transformers/trainer.py", line 2807, in _maybe_log_save_evaluate
File "/media02/nthuy/miniconda3/envs/thesis_longvu/lib/python3.10/site-packages/transformers/trainer.py", line 2890, in _save_checkpoint
File "/media02/nthuy/miniconda3/envs/thesis_longvu/lib/python3.10/site-packages/transformers/trainer.py", line 3001, in _save_optimizer_and_scheduler
File "/media02/nthuy/miniconda3/envs/thesis_longvu/lib/python3.10/site-packages/accelerate/utils/fsdp_utils.py", line 185, in save_fsdp_optimizer
File "/media02/nthuy/miniconda3/envs/thesis_longvu/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1828, in optim_state_dict
File "/media02/nthuy/miniconda3/envs/thesis_longvu/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 1253, in _optim_state_dict_impl
File "/media02/nthuy/miniconda3/envs/thesis_longvu/lib/python3.10/site-packages/torch/distributed/fsdp/_optim_utils.py", line 1396, in _optim_state_dict
File "/media02/nthuy/miniconda3/envs/thesis_longvu/lib/python3.10/site-packages/torch/distributed/fsdp/_optim_utils.py", line 1657, in _gather_orig_param_state
File "/media02/nthuy/miniconda3/envs/thesis_longvu/lib/python3.10/site-packages/torch/distributed/fsdp/_optim_utils.py", line 1593, in _all_gather_optim_state

```

The packages version is:

```auto
torch==2.1.2
numpy==1.26.4
transformers==4.43.1

```

This is the full details I trace the error in functions listed in the traceback. It’s quite long but it leads to what I believe a possible reason for error above and my question below:

> **Summary**
>
> I began digging into the codebase, starting with `torch.distributed.fsdp`, to find out the cause as follows:
> 
> - First of all, the error stems from this function: [pytorch/torch/distributed/fsdp/\_optim\_utils.py at v2.1.2 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/blob/v2.1.2/torch/distributed/fsdp/_optim_utils.py#L1513)  
> It happens that when execution enters this line
> 
> ```auto
> work = dist.all_gather(
> tensors, local_state, group=fsdp_state.process_group, async_op=True
> )
> 
> ```
> 
> `tensors` is a list where certain elements are 0 and others are non-zero, as raised by the error above. When I printed out `object_state.tensors`:
> 
> ```auto
> 2025-02-22 17:02:42,065 - root - DEBUG - rank 0, object_state.tensors: {}, name: exp_avg, info: None
> 2025-02-22 17:02:42,065 - root - DEBUG - rank 1, object_state.tensors: {}, name: exp_avg, info: None
> 2025-02-22 17:02:42,066 - root - DEBUG - rank 2, object_state.tensors: {'exp_avg': _PosDimTensorInfo(shape=torch.Size([183971584]), dtype=torch.float32), 'exp_avg_sq': _PosDimTensorInfo(shape=torch.Size([183971584]), dtype=torch.float32)}, name: exp_avg, info: _PosDimTensorInfo(shape=torch.Size([183971584]), dtype=torch.float32)                                                                                            
> 2025-02-22 17:02:42,066 - root - DEBUG - rank 3, object_state.tensors: {'exp_avg': _PosDimTensorInfo(shape=torch.Size([210030848]), dtype=torch.float32), 'exp_avg_sq': _PosDimTensorInfo(shape=torch.Size([210030848]), dtype=torch.float32)}, name: exp_avg, info: _PosDimTensorInfo(shape=torch.Size([210030848]), dtype=torch.float32)
> 
> ```
> 
> It can be seen that on the 2 ranks 0 and 1, the tensors are empty. Since these `object_state.tensors` are gathered in `object_list` from processes in the process group via:
> 
> ```auto
> dist.all_gather_object(object_list, processed_state, group=fsdp_state.process_group)
> 
> ```
> 
> It seems that the `processed_state` for ranks 0 and 1 is empty (`StateInfo({}, {}, {})`). And this is caused by empty `optim_state` if you look at the for loop in the beginning of `_all_gather_optim_state()`.
> 
> ```auto
> 2025-02-22 17:02:42,060 - root - DEBUG - @tcm: In _all_gather_optim_state(): optim_state: {}
> 
> ```
> 
> - After that, I look into the second function in the traceback: [pytorch/torch/distributed/fsdp/\_optim\_utils.py at v2.1.2 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/blob/v2.1.2/torch/distributed/fsdp/_optim_utils.py#L1611)  
> The `optim_state` dictionary in this function is still empty.
> 
> ```auto
> 2025-02-22 17:02:42,060 - root - DEBUG - @tcm: In _gather_orig_param_state(): optim_state: {}
> 
> ```
> 
> - I continued looking into the next function in the traceback: [pytorch/torch/distributed/fsdp/\_optim\_utils.py at v2.1.2 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/blob/v2.1.2/torch/distributed/fsdp/_optim_utils.py#L1269)  
> In this function, I focused on the following part:
> 
> ```auto
> # Iterate in rank 0's flat parameter ID order to ensure aligned all-gathers
> # across ranks
> for optim_state_key in all_optim_state_keys:
> param_key: Union[str, int, None] = optim_state_key_to_param_key.get(
> optim_state_key, None
> )
> 
> if param_key is None:
> assert use_orig_params, (
> "If use_orig_params is False, we must be able to find the "
> f"corresponding param id. {optim_state_key} {param_key}"
> )
> if not optim_state_key.is_fsdp_managed:
> continue
> 
> if optim_state_key.is_fsdp_managed:
> # If there are multiple unflat_param_names (not use_orig_params),
> # they share the same FSDPParamInfo. So the first unflat_param_name
> # is sufficient to fetch the FSDPParamInfo.
> fqn = optim_state_key.unflat_param_names[0]
> fsdp_param_info = fqn_to_fsdp_param_info[fqn]
> if use_orig_params:
> state = (
> {} if param_key is None else optim_state_dict["state"][param_key]
> )
> unflat_state = [
> _gather_orig_param_state(
> fsdp_param_info,
> fqn,
> state,
> shard_state,
> )
> ]
> 
> ```
> 
> The problem is that `param_key` is `None` which leads to empty ‘state’ when passed to `_gather_orig_param_state()`:
> 
> ```auto
> state = ({} if param_key is None else optim_state_dict["state"][param_key])
> 
> ```
> 
> The `param_key` is `None` because `optim_state_key_to_param_key` dictionary is empty:
> 
> ```auto
> 2025-02-22 17:02:42,046 - root - DEBUG - @tcm: In _optim_state_dict(): optim_state_key_to_param_key: {} # rank 0 or 1
> 2025-02-22 17:02:42,046 - root - DEBUG - @tcm: In _optim_state_dict(): optim_state_key_to_param_key: {_OptimStateKey(unflat_param_names=('lm_head.weight',), is_fsdp_managed=True): 2} # rank 2 or 3
> 2025-02-22 17:02:42,046 - root - DEBUG - @tcm: In _optim_state_dict(): optim_state_key_to_param_key: {_OptimStateKey(unflat_param_names=('model.mm_projector.0.weight',), is_fsdp_managed=True): 0, _OptimStateKey(unflat_param_names=('model.mm_projector.2.weight',), is_fsdp_managed=True): 1, _OptimStateKey(unflat_param_names=('model.mm_projector.0.bias',), is_fsdp_managed=True): 3, _OptimStateKey(unflat_param_names=('model.mm_projector.2.bias',), is_fsdp_managed=True): 4} # rank 2 or 3
> 
> ```
> 
> To understand why the dict `optim_state_key_to_param_key` is empty, I looked into the function: [pytorch/torch/distributed/fsdp/\_optim\_utils.py at v2.1.2 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/blob/v2.1.2/torch/distributed/fsdp/_optim_utils.py#L1161)  
> Here, if we look at the for loop in the beginning:
> 
> ```auto
> for param_key, param in param_key_to_param.items():
> # Do not include parameters without state to avoid empty mappings
> # just like in normal `torch.optim.Optimizer.state_dict()`
> if param_key not in optim_state_dict["state"]:
> continue
> 
> ```
> 
> `optim_state_dict["state"]` is empty so the iteration is skipped, causing `optim_state_key_to_param_key` to not be updated.
> 
> ```auto
> 2025-02-22 17:02:42,041 - root - DEBUG - @tcm: In _map_param_key_to_optim_keys(): optim_state_dict["state"]: {} # in empty ranks such as 0
> 2025-02-22 17:02:28,436 - root - DEBUG - @tcm: In _map_param_key_to_optim_keys(): optim_state_dict["state"]: {0: {'step': tensor(1.), 'exp_avg': tensor([-6.2440e-07, 6.8065e-07, -2.2726e-06, ..., 3.1220e-07, 1.6088e-06, -1.5047e-07], device='cuda:1'), 'exp_avg_sq': tensor([3.8987e-14, 4.6328e-14, 5.1646e-13, ..., 9.7467e-15, 2.5882e-13, 2.2642e-15], device='cuda:1')}...
> 
> ```
> 
> So the problem is `optim_state_dict` being empty when passed into `_optim_state_dict()`.
> 
> - If we go further in the traceback:  
> `_optim_state_dict_impl()`: [pytorch/torch/distributed/fsdp/fully\_sharded\_data\_parallel.py at v2.1.2 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/blob/v2.1.2/torch/distributed/fsdp/fully_sharded_data_parallel.py#1209)  
> `_optim_state_dict()`: [pytorch/torch/distributed/fsdp/fully\_sharded\_data\_parallel.py at v2.1.2 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/blob/v2.1.2/torch/distributed/fsdp/fully_sharded_data_parallel.py#1741)  
> We see that `optim_state_dict` is initially `None`:
> 
> ```auto
> 2025-02-22 17:02:28,304 - root - DEBUG - @tcm: In FSDP.optim_state_dict(): optim_state_dict: None
> 
> ```
> 
> and initialized through:
> 
> ```auto
> if optim_state_dict is None:
> optim_state_dict = optim.state_dict()
> 
> ```
> 
> - Now, when I go to transformers `trainer.py` and look at the following function:  
> [transformers/src/transformers/trainer.py at v4.43.1 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/v4.43.1/src/transformers/trainer.py#L2969)  
> we can see that when saving the fsdp optimizer, no `optim_state_dict` is passed:
> 
> ```auto
> save_fsdp_optimizer(self.accelerator.state.fsdp_plugin, self.accelerator, self.optimizer, self.model, output_dir)
> 
> ```
> 
> So in `torch.distributed.fsdp._optim_utils.py`, the following function is used to initialize `optim_state_dict`: [pytorch/torch/optim/optimizer.py at main · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/blob/main/torch/optim/optimizer.py#L636)

To start with, this is the architecture of the model that I’m trying to fine-tune:

```auto
FullyShardedDataParallel(
  (_fsdp_wrapped_module): CambrianLlamaForCausalLM(
    (model): CambrianLlamaModel(
      (embed_tokens): Embedding(128256, 3072)
      (layers): ModuleList(
        (0-27): 28 x FullyShardedDataParallel(
          (_fsdp_wrapped_module): LlamaDecoderLayer(
            (self_attn): LlamaSdpaAttention(
              (q_proj): Linear(in_features=3072, out_features=3072, bias=False)
              (k_proj): Linear(in_features=3072, out_features=1024, bias=False)
              (v_proj): Linear(in_features=3072, out_features=1024, bias=False)
              (o_proj): Linear(in_features=3072, out_features=3072, bias=False)
              (rotary_emb): LlamaRotaryEmbedding()
            )
            (mlp): LlamaMLP(
              (gate_proj): Linear(in_features=3072, out_features=8192, bias=False)
              (up_proj): Linear(in_features=3072, out_features=8192, bias=False)
              (down_proj): Linear(in_features=8192, out_features=3072, bias=False)
              (act_fn): SiLU()
            )
            (input_layernorm): LlamaRMSNorm()
            (post_attention_layernorm): LlamaRMSNorm()
          )
        )
      )
      (norm): LlamaRMSNorm()
      (rotary_emb): LlamaRotaryEmbedding()
      (mm_projector): Sequential(
        (0): Linear(in_features=1024, out_features=3072, bias=True)
        (1): GELU(approximate='none')
        (2): Linear(in_features=3072, out_features=3072, bias=True)
      )
      (mm_projector_aux_0): Sequential(
        (0): Linear(in_features=1152, out_features=1024, bias=True)
        (1): GELU(approximate='none')
        (2): Linear(in_features=1024, out_features=1024, bias=True)
        (3): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
      )
      (mm_projector_aux_1): Sequential(
        (0): Linear(in_features=1536, out_features=1024, bias=True)
        (1): GELU(approximate='none')
        (2): Linear(in_features=1024, out_features=1024, bias=True)
        (3): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
      )
      (vision_sampler_0): VisionTokenSampler(
        (layers): ModuleList(
          (0-2): 3 x VisionCrossAttentionLayer(
            (proj_context): Linear(in_features=1024, out_features=1024, bias=False)
            (proj_in): Linear(in_features=2048, out_features=1024, bias=False)
            (proj_out): MLP(
              (linear_1): Linear(in_features=1024, out_features=1024, bias=False)
              (act): GELU(approximate='none')
              (linear_2): Linear(in_features=1024, out_features=1024, bias=False)
            )
            (norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
            (cross_attn): MultiKVCrossAttention(
              (q_proj): Sequential(
                (0): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
                (1): Linear(in_features=1024, out_features=1024, bias=False)
              )
              (k_proj_0): Sequential(
                (0): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
                (1): Linear(in_features=1024, out_features=1024, bias=False)
              )
              (v_proj_0): Sequential(
                (0): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
                (1): Linear(in_features=1024, out_features=1024, bias=False)
              )
              (k_proj_1): Sequential(
                (0): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
                (1): Linear(in_features=1024, out_features=1024, bias=False)
              )
              (v_proj_1): Sequential(
                (0): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
                (1): Linear(in_features=1024, out_features=1024, bias=False)
              )
              (o_proj): Linear(in_features=1024, out_features=1024, bias=False)
            )
          )
        )
      )
      (lm_head): Linear(in_features=3072, out_features=128256, bias=False)
    )
  )
)

```

There are two FSDP instances: the entire model and the `LlamaDecoderLayer` layer. In my fine-tuning script, this is how I configure FSDP options when used in `Trainer`:

```auto
--fsdp "full_shard auto_wrap" \
--fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \

```

Since the error is related to saving FSDP optimizer states, I would like to provide the following method in `LLaVATrainer` class, which is a subclass from Huggingface `Trainer`:

```auto
class LLaVATrainer(Trainer):
    def create_optimizer(self):
        """
        Setup the optimizer.

        We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the
        Trainer's init through `optimizers`, or subclass and override this method in a subclass.
        """
        # pyre-fixme[16]: `Trainer` has no attribute `model`.
        opt_model = self.model
        # if self.args.unfreeze_mm_vision_tower:
        # opt_model.get_model().vision_tower_aux_list = nn.ModuleList(opt_model.get_vision_tower_aux_list())
        # self.param_to_name = map_params_to_module_names([opt_model])
        # pyre-fixme[16]: `Trainer` has no attribute `optimizer`.
        if self.optimizer is None:
            decay_parameters = get_parameter_names(opt_model, ALL_LAYERNORM_LAYERS)
            decay_parameters = [name for name in decay_parameters if "bias" not in name]
            # pyre-fixme[16]: `Trainer` has no attribute `mm_projector_lr`.
            assert not (self.args.mm_projector_lr and self.args.mm_vision_sampler_lr)
                optimizer_grouped_parameters = [
                    {
                        "params": [
                            p
                            for n, p in opt_model.named_parameters()
                            if (n in decay_parameters and p.requires_grad)
                        ],
                        "weight_decay": self.args.weight_decay,
                    },
                    {
                        "params": [
                            p
                            for n, p in opt_model.named_parameters()
                            if (n not in decay_parameters and p.requires_grad)
                        ],
                        "weight_decay": 0.0,
                    },
                ]
optimizer_cls, optimizer_kwargs = Trainer.get_optimizer_cls_and_kwargs(
                self.args
            )

            self.optimizer = optimizer_cls(
                optimizer_grouped_parameters, **optimizer_kwargs
            )
        return self.optimizer

```

In the model codebase, the `create_optimizer()` method above creates param groups and other params. When I print out the steps in this `create_optimizer()` method, the output is as follows:

```auto
2025-02-22 17:01:20,658 - root - DEBUG - @tcm: In LLaVATrainer: n = _fsdp_wrapped_module.lm_head.weight, p = Parameter containing: tensor([], device='cuda:0', dtype=torch.bfloat16, requires_grad=True)
2025-02-22 17:01:20,695 - root - DEBUG - @tcm: In LLaVATrainer: n = _fsdp_wrapped_module.lm_head.weight, p = Parameter containing:
tensor([], device='cuda:1', dtype=torch.bfloat16, requires_grad=True)
...
2025-02-22 17:01:20,690 - root - DEBUG - @tcm: In LLaVATrainer: n = _fsdp_wrapped_module.lm_head.weight, p = Parameter containing: tensor([0.0103, 0.0090, 0.0134, ..., 0.0049, -0.0025, -0.0052], device='cuda:2', dtype=torch.bfloat16, requires_grad=True)
2025-02-22 17:01:20,691 - root - DEBUG - @tcm: In LLaVATrainer: n = _fsdp_wrapped_module.lm_head.weight, p = Parameter containing: tensor([-0.0099, -0.0302, -0.0054, ..., -0.0038, -0.0027, -0.0015],
       device='cuda:3', dtype=torch.bfloat16, requires_grad=True)

```

As can be seen, on ranks 0 and 1, the `LM head` layer wrapped in FSDP unit has no params in the tensor, but they are present on ranks 3 and 4. So I think the error might stem from this fsdp sharding where the same layer `lm head` is sharded on ranks 2 and 3 but empty on ranks 0 and 1. Therefore, **my question is** :

I would like to ask why FSDP in `Trainer` shards a layer such that it’s empty on certain ranks, possibly leading to the error above?

I’m been trying my best to find out the root cause and fix this error but the codebase is large and complex, so I am seeking help from the community.

Thanks in advance.

P/S: Since the error and the way I approach it is quite technically deep, please allow me to ping @sgugger and maybe several other experts on this problem! Thank you!

---

<div class="post-metadata">

**Author:** ![tcm03](https://avatars.discourse-cdn.com/v4/letter/t/c0e974/32.png) [@tcm03](https://discuss.huggingface.co/u/tcm03)\
**Post date:** [February 23, 2025, 4:55pm UTC](https://discuss.huggingface.co/t/tensor-shape-mismatch-error-when-doing-an-allgather-in-distributed-training-with-fsdp/142433/2 "2025-02-23T16:55:58Z")

</div>

Hi. I think this post is not going to have anybody comment so if somebody knows where I could ask this with potential help, please tell me, I would appreciate it. Thanks!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 24, 2025, 6:13am UTC](https://discuss.huggingface.co/t/tensor-shape-mismatch-error-when-doing-an-allgather-in-distributed-training-with-fsdp/142433/3 "2025-02-24T06:13:25Z")

</div>

I think there is a possibility of a bug in the Transformers or accelerate libraries, but I thought it might be a problem with the PyTorch specifications. In any case, raising the issue in the github issue section of one of the libraries will certainly be conveyed to the developers.

> <https://github.com/Lightning-AI/pytorch-lightning/issues/16815#issuecomment-1558572282>
>
> \### Bug description
> 
> I'm using \`fsdp\` distributed training, and I noticed when… I resume training from a ckpt with a different number of workers, the optimize step will fail
> 
> \### How to reproduce the bug
> 
> \_No response\_
> 
> \### Error messages and logs
> 
> RuntimeError: The size of tensor a (56124554) must match the size of tensor b (14031139) at non-singleton dimension 0
> 
> The checkpoint is trained with a node with 6 cards. Then when resuming with 4 nodes (a total of 24 cards), it gives this error.
> 
> \*\*Note that 56124554 / 4 = 14031138.5 ~= 14031139\*\*
> 
> \<details\>
> \<summary\>Full Traceback\</summary\>
> \`\`\`
> Traceback (most recent call last):
> File "/opt/code-fetcher/yuxlu-test.git/yuxlu-test/src/yuxlutest/main.py", line 168, in \<module\>
> main()
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/linkedin/clix/\_clix.py", line 659, in decorated
> f.callback.invoke\_listeners(e)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/linkedin/clix/\_clix.py", line 230, in invoke\_listeners
> raise value
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/linkedin/clix/\_clix.py", line 656, in decorated
> return f(\*args, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/linkedin/clix/ext/zipapp.py", line 41, in \_\_call\_\_
> super().\_\_call\_\_(\*args, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/click/core.py", line 1130, in \_\_call\_\_
> return self.main(\*args, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
> rv = self.invoke(ctx)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
> return ctx.invoke(self.callback, \*\*ctx.params)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
> return \_\_callback(\*args, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/linkedin/clix/\_clix.py", line 212, in \_\_call\_\_
> self.invoke\_listeners(e)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/linkedin/clix/\_clix.py", line 230, in invoke\_listeners
> raise value
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/linkedin/clix/\_clix.py", line 206, in \_\_call\_\_
> return self.callback(self, \*args, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/\_\_init\_\_.py", line 345, in wrapper
> return f(\*args, \*\*kwargs)
> File "/opt/code-fetcher/yuxlu-test.git/yuxlu-test/src/yuxlutest/main.py", line 152, in main
> trainer.fit(mnist\_model, datamodule=datamodule, ckpt\_path=ckpt\_path)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/trainer/trainer.py", line 603, in fit
> call.\_call\_and\_handle\_interrupt(
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/trainer/call.py", line 38, in \_call\_and\_handle\_interrupt
> return trainer\_fn(\*args, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/trainer/trainer.py", line 645, in \_fit\_impl
> self.\_run(model, ckpt\_path=self.ckpt\_path)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/trainer/trainer.py", line 1098, in \_run
> results = self.\_run\_stage()
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/trainer/trainer.py", line 1177, in \_run\_stage
> self.\_run\_train()
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/trainer/trainer.py", line 1200, in \_run\_train
> self.fit\_loop.run()
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/loops/loop.py", line 199, in run
> self.advance(\*args, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/loops/fit\_loop.py", line 267, in advance
> self.\_outputs = self.epoch\_loop.run(self.\_data\_fetcher)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/loops/loop.py", line 199, in run
> self.advance(\*args, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/loops/epoch/training\_epoch\_loop.py", line 214, in advance
> batch\_output = self.batch\_loop.run(kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/loops/loop.py", line 199, in run
> self.advance(\*args, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/loops/batch/training\_batch\_loop.py", line 88, in advance
> outputs = self.optimizer\_loop.run(optimizers, kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/loops/loop.py", line 199, in run
> self.advance(\*args, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/loops/optimization/optimizer\_loop.py", line 200, in advance
> result = self.\_run\_optimization(kwargs, self.\_optimizers\[self.optim\_progress.optimizer\_position\])
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/loops/optimization/optimizer\_loop.py", line 247, in \_run\_optimization
> self.\_optimizer\_step(optimizer, opt\_idx, kwargs.get("batch\_idx", 0), closure)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/loops/optimization/optimizer\_loop.py", line 357, in \_optimizer\_step
> self.trainer.\_call\_lightning\_module\_hook(
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/trainer/trainer.py", line 1342, in \_call\_lightning\_module\_hook
> output = fn(\*args, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/core/module.py", line 1661, in optimizer\_step
> optimizer.step(closure=optimizer\_closure)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/core/optimizer.py", line 169, in step
> step\_output = self.\_strategy.optimizer\_step(self.\_optimizer, self.\_optimizer\_idx, closure, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/strategies/ddp.py", line 281, in optimizer\_step
> optimizer\_output = super().optimizer\_step(optimizer, opt\_idx, closure, model, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/strategies/strategy.py", line 234, in optimizer\_step
> return self.precision\_plugin.optimizer\_step(
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/pytorch\_lightning/plugins/precision/native\_amp.py", line 98, in optimizer\_step
> step\_output = self.scaler.step(optimizer, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/fairscale/optim/grad\_scaler.py", line 317, in step
> retval = self.\_maybe\_opt\_step(optimizer, optimizer\_state, \*args, \*\*kwargs) # type: ignore
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/torch/cuda/amp/grad\_scaler.py", line 285, in \_maybe\_opt\_step
> retval = optimizer.step(\*args, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/torch/optim/lr\_scheduler.py", line 65, in wrapper
> return wrapped(\*args, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/torch/optim/optimizer.py", line 113, in wrapper
> return func(\*args, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/torch/autograd/grad\_mode.py", line 27, in decorate\_context
> return func(\*args, \*\*kwargs)
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/torch/optim/adam.py", line 157, in step
> adam(params\_with\_grad,
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/torch/optim/adam.py", line 213, in adam
> func(params,
> File "/home/jobuser/build/yuxlu-test/environments/development-venv/lib/python3.10/site-packages/torch/optim/adam.py", line 262, in \_single\_tensor\_adam
> exp\_avg.mul\_(beta1).add\_(grad, alpha=1 - beta1)
> RuntimeError: The size of tensor a (56124554) must match the size of tensor b (14031139) at non-singleton dimension 0
> \`\`\`
> \</details\>
> 
> \### Environment
> 
> \<details\>
> \<summary\>Current environment\</summary\>
> 
> trainer argument:
> \`\`\`python
> trainer = pl.Trainer(
> default\_root\_dir=config\["root\_dir"\],
> accelerator="gpu",
> devices=torch.cuda.device\_count()
> if torch.cuda.is\_available()
> else None, # limiting got iPython runs
> max\_epochs=config\["max\_epochs"\],
> logger=TensorBoardLogger(
> save\_dir=config\["root\_dir"\],
> name="",
> version=run\_name, # current time
> default\_hp\_metric=True,
> max\_queue=200,
> ),
> log\_every\_n\_steps=1,
> strategy=strategy, # DDPFullyShardedStrategy
> num\_nodes=int(os.getenv("WORLD\_SIZE", "1")) // torch.cuda.device\_count(),
> callbacks=\[ModelCheckpoint(
> dirpath=config\["root\_dir"\] + "/" + run\_name + "/checkpoints",
> save\_top\_k=-1, # We have plenty of disks. Save all.
> mode="max",
> monitor="val/loss",
> filename="epoch={epoch}-step={step}-val\_loss={val/loss:.5f}",
> every\_n\_train\_steps=100,
> auto\_insert\_metric\_name=False,
> save\_last=True,
> )\],
> precision=16,
> accumulate\_grad\_batches=config\["accumulate\_grad\_batches"\],
> val\_check\_interval=500,
> max\_steps=config\["max\_steps"\],
> )
> 
> \`\`\`
> 
> \`\`\`
> \#- Running environment of LightningApp (e.g. local, cloud): cloud
> 
> other information:
> \* CUDA:
> - GPU:
> - Tesla V100-PCIE-32GB
> - Tesla V100-PCIE-32GB
> - Tesla V100-PCIE-32GB
> - Tesla V100-PCIE-32GB
> - Tesla V100-PCIE-32GB
> - Tesla V100-PCIE-32GB
> - available: True
> - version: 10.2
> \* Lightning:
> - lightning-utilities: 0.5.0
> - lion-pytorch: 0.0.6
> - pytorch-lightning: 1.8.6
> - torch: 1.12.1
> - torchmetrics: 0.7.2
> \* Packages:
> - absl-py: 0.15.0
> - accept-types: 0.4.1
> - acl-python-api: 15.0.0
> - aiodns: 2.0.0
> - aiohttp: 3.8.1
> - aiosignal: 1.2.0
> - ansicolors: 1.1.8
> - anyio: 2.2.0
> - apipkg: 1.5
> - appdirs: 1.4.4
> - appnope: 0.1.3
> - argon2-cffi: 20.1.0
> - async-generator: 1.10
> - async-timeout: 4.0.1
> - asynctest: 0.13.0
> - atomicwrites: 1.4.0
> - atpublic: 1.0
> - attrs: 21.4.0
> - avro-json-serializer: 0.5
> - avro-python3: 1.8.2
> - azure-common: 1.1.23
> - azure-core: 1.11.0
> - azure-identity: 1.4.0
> - azure-keyvault-secrets: 4.0.0
> - babel: 2.9.1
> - backcall: 0.2.0
> - backoff: 1.3.1
> - backports.entry-points-selectable: 1.0.4
> - bleach: 3.2.1
> - cachecontrol: 0.12.11
> - cached-property: 1.5.1
> - cachetools: 5.2.0
> - cachy: 0.3.0
> - certifi: 2022.6.15
> - cffi: 1.15.0
> - charset-normalizer: 2.1.0
> - cleo: 0.8.1
> - click: 8.1.3
> - clikit: 0.6.2
> - colorama: 0.4.3
> - construct: 2.8.11
> - contextvars: 2.4
> - coverage: 6.3.2
> - crashtest: 0.3.1
> - cryptography: 2.9.2
> - cython: 0.29.32
> - datasets: 2.5.1
> - decorator: 4.4.2
> - defusedxml: 0.6.0
> - deprecation: 2.0.7
> - dill: 0.3.5.1
> - diskcache: 5.3.0
> - distgradle: 2.1.6
> - distlib: 0.3.1
> - dnspython: 2.1.0.1
> - docutils: 0.17.1
> - entrypoints: 0.3.0.2
> - execnet: 1.7.1
> - fairscale: 0.4.13
> - filelock: 3.8.0
> - flake8: 4.0.1.1
> - flake8-annotations-complexity: 0.0.6.2
> - flake8-bugbear: 20.1.4
> - flake8-builtins: 1.4.2
> - flake8-pie: 0.5.0.1
> - flit: 1.3.0.1
> - flit-core: 3.7.1
> - flufl.lock: 5.0.1
> - frozenlist: 1.2.0
> - fsspec: 2022.11.0
> - gitdb: 4.0.9
> - gitpython: 3.1.27
> - google-auth: 2.10.0
> - google-auth-oauthlib: 0.4.6
> - googleapis-common-protos: 1.54.0
> - grpcio: 1.47.0
> - halo: 0.0.30
> - html5lib: 1.0.1
> - huggingface-hub: 0.10.1
> - idna: 3.3
> - idna-ssl: 1.1.0
> - immutables: 0.9
> - importlib-metadata: 4.12.0
> - importlib-resources: 5.1.2
> - iniconfig: 1.1.1
> - intreehooks: 1.0
> - ipykernel: 5.5.3
> - ipython: 7.19.0
> - ipython-genutils: 0.2.0
> - isodate: 0.6.0
> - jedi: 0.18.0
> - jeepney: 0.4.3.2
> - jinja2: 3.0.3
> - json5: 0.9.5
> - jsonschema: 3.2.0
> - jupyter-client: 6.1.12
> - jupyter-core: 4.7.1
> - jupyter-packaging: 0.9.2
> - jupyter-server: 1.6.1
> - jupyterlab: 3.2.9
> - jupyterlab-pygments: 0.1.2
> - jupyterlab-server: 2.4.0
> - keyring: 21.4.0
> - kms-python-api: 3.1.13
> - ldap3: 2.9.1
> - lightning-utilities: 0.5.0
> - lion-pytorch: 0.0.6
> - log-symbols: 0.0.14
> - lxml: 4.8.0
> - markdown: 3.4.1
> - markupsafe: 2.1.1
> - mccabe: 0.7.0
> - mistune: 0.8.4
> - more-itertools: 8.0.2
> - msal: 1.15.0
> - msal-extensions: 0.2.2
> - msgpack: 1.0.4
> - msrest: 0.6.10
> - multidict: 5.2.0
> - multiprocess: 0.70.11.1
> - mypy: 0.812
> - mypy-extensions: 0.4.3
> - nbclassic: 0.2.7
> - nbclient: 0.5.0
> - nbconvert: 6.0.7
> - nbformat: 5.1.3
> - nest-asyncio: 1.4.0
> - netifaces: 0.10.9
> - notebook: 6.3.0
> - numpy: 1.23.0
> - oauthlib: 3.1.0
> - oldest-supported-numpy: 2022.1.30
> - packaging: 21.3
> - pandas: 1.3.5
> - pandocfilters: 1.4.2
> - parsimonious: 0.8.1
> - parso: 0.8.1
> - pastel: 0.2.1
> - pbkdf2: 1.3
> - pbr: 5.6.0
> - pexpect: 4.8.0
> - pickleshare: 0.7.5
> - pip: 21.3.1
> - pkginfo: 1.6.1
> - pluggy: 1.0.0
> - poetry: 1.1.4.1
> - poetry-core: 1.0.8
> - portalocker: 1.7.1
> - prometheus-client: 0.10.1
> - prompt-toolkit: 3.0.14
> - protobuf: 3.19.4
> - psutil: 5.9.0
> - ptyprocess: 0.7.0
> - py: 1.11.0
> - pyarrow: 8.0.0
> - pyasn1: 0.4.8
> - pyasn1-modules: 0.2.8
> - pycares: 3.0.0
> - pycodestyle: 2.8.0
> - pycparser: 2.20
> - pydeprecate: 0.3.1
> - pyflakes: 2.4.0
> - pygments: 2.10.0
> - pyjwt: 1.7.1
> - pylev: 1.3.0
> - pyopenssl: 19.1.0
> - pyparsing: 3.0.6
> - pyrsistent: 0.14.11
> - pytest: 7.1.2
> - pytest-cov: 3.0.0.1
> - pytest-datadir: 1.3.1
> - pytest-forked: 1.1.3
> - pytest-html: 3.1.1
> - pytest-metadata: 1.11.0
> - pytest-runner: 5.3.1
> - pytest-xdist: 2.5.0
> - python-dateutil: 2.8.2
> - python-seco-range: 9.1.3
> - pytoml: 0.1.21
> - pytorch-lightning: 1.8.6
> - pytz: 2021.3
> - pyyaml: 6.0
> - pyzmq: 22.0.3
> - regex: 2022.7.25
> - requests: 2.28.1
> - requests-oauthlib: 1.3.0
> - requests-toolbelt: 0.9.1
> - responses: 0.18.0
> - retry: 0.9.2.1
> - rsa: 4.9
> - secretstorage: 3.1.2.1
> - send2trash: 1.5.0
> - setuptools: 59.6.0
> - setuptools-git: 1.2
> - setuptools-scm: 6.4.2
> - setuptools-scm-git-archive: 1.1
> - shellingham: 1.3.1
> - simplejson: 3.17.2
> - six: 1.16.0
> - smmap: 5.0.0
> - sniffio: 1.2.0
> - spinners: 0.0.24
> - statsd: 3.3.0
> - tblib: 1.4.0
> - tbvaccine: 0.3.1
> - tensorboard: 2.10.0
> - tensorboard-data-server: 0.6.1
> - tensorboard-plugin-wit: 1.8.1
> - tensorboardx: 2.2
> - termcolor: 1.1.0
> - terminado: 0.9.4
> - testpath: 0.4.4
> - tokenizers: 0.11.6
> - toml: 0.10.2
> - tomli: 2.0.1
> - tomlkit: 0.7.2
> - tools-events-api: 6.0.10
> - torch: 1.12.1
> - torchmetrics: 0.7.2
> - tornado: 6.1
> - tqdm: 4.64.1
> - traitlets: 5.0.5
> - transformers: 4.23.1
> - types-futures: 3.3.1
> - types-protobuf: 3.18.3
> - typing-extensions: 4.4.0
> - urllib3: 1.26.9
> - virtualenv: 20.1.0
> - wcwidth: 0.2.5
> - weaver-grpc-python-api: 0.0.2
> - weaver-python-api: 4.0.0
> - webencodings: 0.5.1
> - werkzeug: 2.2.2
> - wheel: 0.37.0
> - xxhash: 3.0.0
> - yarl: 1.7.2
> - zipp: 3.8.0
> \* System:
> - OS: Linux
> - architecture:
> - 64bit
> - ELF
> - processor: x86\_64
> - python: 3.10.2
> - version: #1 SMP Wed Apr 27 20:34:34 UTC 2022
> \`\`\`
> 
> \</details\>
> 
> 
> \### More info
> 
> This problem makes elastic training impossible since \`torchrun\` will restart all workers when a new worker joins, and all of them will fail on this tensor size mismatch.
> 
> cc @awaelchli @tchaton @justusschock

> <https://github.com/pytorch/pytorch/issues/141799>
>
> \### 🐛 Describe the bug
> 
> When saving checkpoints of FSDP models with local shards…, the \`get\_state\_dict\` throws not implemented error.
> 
> The code below reproduces the error on my machine:
> 
> \`\`\`python
> import torch
> import torch.nn as nn
> import torch.optim as optim
> import torch.distributed as dist
> 
> import functools
> from torch.distributed.fsdp import (
> FullyShardedDataParallel as FSDP,
> StateDictType,
> ShardingStrategy,
> )
> from torch.distributed.checkpoint.state\_dict import get\_state\_dict
> from torch.distributed.fsdp.wrap import size\_based\_auto\_wrap\_policy
> 
> 
> class Model(nn.Module):
> def \_\_init\_\_(self, size: int):
> super().\_\_init\_\_()
> self.layer1 = nn.Linear(size // 4, 1, bias=False)
> self.layer2 = nn.Linear(1, size // 4, bias=False)
> self.layer3 = nn.Linear(size // 4, 1, bias=False)
> self.layer4 = nn.Linear(1, size // 4, bias=False)
> 
> def forward(self, x):
> x = self.layer1(x)
> x = self.layer2(x)
> x = self.layer3(x)
> x = self.layer4(x)
> return x
> 
> 
> def get\_auto\_wrap\_policy():
> return functools.partial(
> size\_based\_auto\_wrap\_policy,
> min\_num\_params=100\_000\_000, # 自动包装大于100M参数的层
> force\_leaf\_modules=set(),
> exclude\_wrap\_modules=set()
> )
> 
> 
> if \_\_name\_\_ == "\_\_main\_\_":
> dist.init\_process\_group(backend="nccl")
> rank = dist.get\_rank()
> world\_size = dist.get\_world\_size()
> torch.cuda.set\_device(rank)
> 
> num\_devices = 8
> shard\_size = 4
> 
> shard\_groups = tuple(
> dist.new\_group(backend="nccl", ranks=list(range(group, group + shard\_size)))
> for group in range(0, num\_devices, shard\_size)
> )
> current\_shard\_group = shard\_groups\[rank // shard\_size\]
> 
> replicate\_groups = tuple(
> dist.new\_group(backend="nccl", ranks=list(range(sz, world\_size, shard\_size)))
> for sz in range(shard\_size)
> )
> current\_replicate\_group = replicate\_groups\[rank % shard\_size\]
> 
> size\_in\_GB = 4
> model\_size = size\_in\_GB \* 1024 \* 1024 \* 1024 // 4
> if rank % shard\_size == 0:
> with torch.device("cuda"):
> model = Model(model\_size)
> model\_init\_fn = None
> else:
> with torch.device("meta"):
> model = Model(model\_size)
> model\_init\_fn = lambda x: x.to\_empty(
> device=torch.cuda.current\_device(), recurse=False
> )
> 
> 
> model = FSDP(
> model,
> process\_group=(
> current\_shard\_group,
> current\_replicate\_group
> ),
> sharding\_strategy=ShardingStrategy.HYBRID\_SHARD,
> param\_init\_fn=model\_init\_fn,
> auto\_wrap\_policy=get\_auto\_wrap\_policy(),
> sync\_module\_states=True,
> )
> 
> # This is ok, yet it gives deprecated warning
> # with FSDP.state\_dict\_type(model, StateDictType.FULL\_STATE\_DICT):
> # state\_dict = model.state\_dict()
> 
> optimizer = optim.Adam(model.parameters(), lr=0.001)
> # This will raise error
> state\_dict, optimizer = get\_state\_dict(model, optimizer)
> 
> if rank == 0:
> from safetensors.torch import save\_file
> save\_file(state\_dict, f"./model.safetensors")
> 
> for pg in shard\_groups:
> dist.destroy\_process\_group(pg)
> for pg in replicate\_groups:
> dist.destroy\_process\_group(pg)
> dist.destroy\_process\_group()
> \`\`\`
> 
> The error message:
> 
> \`\`\`shell
> \[rank5\]: Traceback (most recent call last):
> \[rank5\]: File "/workspace/inference/scripts/test\_implement/tests/test\_fsdp\_save\_checkpoint.py", line 94, in \<module\>
> \[rank5\]: state\_dict, optimizer = get\_state\_dict(model, optimizer)
> \[rank5\]: File "/home/user/miniconda3/envs/inferenc/lib/python3.10/site-packages/torch/distributed/checkpoint/state\_dict.py", line 1109, in get\_state\_dict
> \[rank5\]: model\_state\_dict = \_get\_model\_state\_dict(model, info)
> \[rank5\]: File "/home/user/miniconda3/envs/inference/lib/python3.10/site-packages/torch/distributed/checkpoint/state\_dict.py", line 520, in \_get\_model\_state\_dict
> \[rank5\]: if torch.is\_tensor(p) and p.is\_meta:
> \[rank5\]: File "/home/user/miniconda3/envs/inference/lib/python3.10/site-packages/typing\_extensions.py", line 2853, in wrapper
> \[rank5\]: return arg(\*args, \*\*kwargs)
> \[rank5\]: File "/home/user/miniconda3/envs/inference/lib/python3.10/site-packages/torch/distributed/\_shard/sharded\_tensor/api.py", line 1140, in \_\_torch\_function\_\_
> \[rank5\]: return dispatch(st\_instance, func)
> \[rank5\]: File "/home/user/miniconda3/envs/inference/lib/python3.10/site-packages/torch/distributed/\_shard/sharded\_tensor/api.py", line 1122, in dispatch
> \[rank5\]: return \_SHARDED\_OPS\[func\](types, args, kwargs, st.\_process\_group)
> \[rank5\]: File "/home/user/miniconda3/envs/inference/lib/python3.10/site-packages/torch/distributed/\_shard/op\_registry\_utils.py", line 33, in wrapper
> \[rank5\]: return wrapped\_func(types, args, kwargs, process\_group)
> \[rank5\]: File "/home/user/miniconda3/envs/inference/lib/python3.10/site-packages/torch/distributed/\_shard/sharded\_tensor/\_ops/tensor\_ops.py", line 57, in st\_is\_meta
> \[rank5\]: return args\[0\].local\_tensor().is\_meta
> \[rank5\]: File "/home/user/miniconda3/envs/inference/lib/python3.10/site-packages/torch/distributed/\_shard/sharded\_tensor/api.py", line 1099, in local\_tensor
> \[rank5\]: raise NotImplementedError("Only single local shard is supported.")
> \[rank5\]: NotImplementedError: Only single local shard is supported.
> \`\`\`
> 
> Using the conventional way of saving checkpoints throws deprecated warning.
> 
> Also, the current interface of \`get\_state\_dict\` forces me to pass an optimizer, which I think should support to be \`None\`.
> 
> \### Versions
> 
> PyTorch version: 2.4.0+cu121
> Is debug build: False
> CUDA used to build PyTorch: 12.1
> ROCM used to build PyTorch: N/A
> 
> OS: Ubuntu 22.04 LTS (x86\_64)
> GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
> Clang version: Could not collect
> CMake version: version 3.22.1
> Libc version: glibc-2.35
> 
> Python version: 3.10.15 (main, Oct 3 2024, 07:27:34) \[GCC 11.2.0\] (64-bit runtime)
> Python platform: Linux-5.4.0-153-generic-x86\_64-with-glibc2.35
> Is CUDA available: True
> CUDA runtime version: 12.2.140
> CUDA\_MODULE\_LOADING set to: LAZY
> GPU models and configuration: 
> GPU 0: NVIDIA A800-SXM4-80GB
> GPU 1: NVIDIA A800-SXM4-80GB
> GPU 2: NVIDIA A800-SXM4-80GB
> GPU 3: NVIDIA A800-SXM4-80GB
> GPU 4: NVIDIA A800-SXM4-80GB
> GPU 5: NVIDIA A800-SXM4-80GB
> GPU 6: NVIDIA A800-SXM4-80GB
> GPU 7: NVIDIA A800-SXM4-80GB
> 
> Nvidia driver version: 535.161.08
> cuDNN version: Could not collect
> HIP runtime version: N/A
> MIOpen runtime version: N/A
> Is XNNPACK available: True
> 
> CPU:
> Architecture: x86\_64
> CPU op-mode(s): 32-bit, 64-bit
> Address sizes: 46 bits physical, 57 bits virtual
> Byte Order: Little Endian
> CPU(s): 128
> On-line CPU(s) list: 0-95
> Off-line CPU(s) list: 96-127
> Vendor ID: GenuineIntel
> Model name: Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz
> CPU family: 6
> Model: 106
> Thread(s) per core: 2
> Core(s) per socket: 32
> Socket(s): 2
> Stepping: 6
> Frequency boost: enabled
> CPU max MHz: 3400.0000
> CPU min MHz: 800.0000
> BogoMIPS: 5200.00
> Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant\_tsc art arch\_perfmon pebs bts rep\_good nopl xtopology nonstop\_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds\_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4\_1 sse4\_2 x2apic movbe popcnt tsc\_deadline\_timer aes xsave avx f16c rdrand lahf\_lm abm 3dnowprefetch cpuid\_fault epb cat\_l3 invpcid\_single ssbd mba ibrs ibpb stibp ibrs\_enhanced tpr\_shadow vnmi flexpriority ept vpid ept\_ad fsgsbase tsc\_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt\_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel\_pt avx512cd sha\_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm\_llc cqm\_occup\_llc cqm\_mbm\_total cqm\_mbm\_local wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke avx512\_vbmi2 gfni vaes vpclmulqdq avx512\_vnni avx512\_bitalg tme avx512\_vpopcntdq rdpid md\_clear pconfig flush\_l1d arch\_capabilities
> Virtualization: VT-x
> L1d cache: 3 MiB (64 instances)
> L1i cache: 2 MiB (64 instances)
> L2 cache: 80 MiB (64 instances)
> L3 cache: 96 MiB (2 instances)
> NUMA node(s): 2
> NUMA node0 CPU(s): 0-31,64-95
> NUMA node1 CPU(s): 32-63,96-127
> Vulnerability Itlb multihit: Not affected
> Vulnerability L1tf: Not affected
> Vulnerability Mds: Not affected
> Vulnerability Meltdown: Not affected
> Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT vulnerable
> Vulnerability Retbleed: Not affected
> Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
> Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and \_\_user pointer sanitization
> Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS\_FW, STIBP conditional, RSB filling, PBRSB-eIBRS SW sequence
> Vulnerability Srbds: Not affected
> Vulnerability Tsx async abort: Not affected
> 
> Versions of relevant libraries:
> \[pip3\] flake8==7.1.1
> \[pip3\] flashinfer==0.1.6+cu121torch2.4
> \[pip3\] mypy-extensions==1.0.0
> \[pip3\] numpy==1.26.4
> \[pip3\] nvidia-cublas-cu12==12.1.3.1
> \[pip3\] nvidia-cuda-cupti-cu12==12.1.105
> \[pip3\] nvidia-cuda-nvrtc-cu12==12.1.105
> \[pip3\] nvidia-cuda-runtime-cu12==12.1.105
> \[pip3\] nvidia-cudnn-cu12==9.1.0.70
> \[pip3\] nvidia-cufft-cu12==11.0.2.54
> \[pip3\] nvidia-curand-cu12==10.3.2.106
> \[pip3\] nvidia-cusolver-cu12==11.4.5.107
> \[pip3\] nvidia-cusparse-cu12==12.1.0.106
> \[pip3\] nvidia-nccl-cu12==2.20.5
> \[pip3\] nvidia-nvjitlink-cu12==12.6.85
> \[pip3\] nvidia-nvtx-cu12==12.1.105
> \[pip3\] torch==2.4.0
> \[pip3\] torchao==0.6.1
> \[pip3\] torchmetrics==1.5.1
> \[pip3\] torchvision==0.19.0
> \[pip3\] triton==3.0.0
> \[conda\] flashinfer 0.1.6+cu121torch2.4 pypi\_0 pypi
> \[conda\] nvidia-cublas-cu12 12.1.3.1 pypi\_0 pypi
> \[conda\] nvidia-cuda-cupti-cu12 12.1.105 pypi\_0 pypi
> \[conda\] nvidia-cuda-nvrtc-cu12 12.1.105 pypi\_0 pypi
> \[conda\] nvidia-cuda-runtime-cu12 12.1.105 pypi\_0 pypi
> \[conda\] nvidia-cudnn-cu12 9.1.0.70 pypi\_0 pypi
> \[conda\] nvidia-cufft-cu12 11.0.2.54 pypi\_0 pypi
> \[conda\] nvidia-curand-cu12 10.3.2.106 pypi\_0 pypi
> \[conda\] nvidia-cusolver-cu12 11.4.5.107 pypi\_0 pypi
> \[conda\] nvidia-cusparse-cu12 12.1.0.106 pypi\_0 pypi
> \[conda\] nvidia-nccl-cu12 2.20.5 pypi\_0 pypi
> \[conda\] nvidia-nvjitlink-cu12 12.6.85 pypi\_0 pypi
> \[conda\] nvidia-nvtx-cu12 12.1.105 pypi\_0 pypi
> \[conda\] torch 2.4.0 pypi\_0 pypi
> \[conda\] torchao 0.6.1 pypi\_0 pypi
> \[conda\] triton 3.0.0 pypi\_0 pypi
> 
> cc @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o @LucasLLC @pradeepfn

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 24, 2025, 7:33am UTC](https://discuss.huggingface.co/t/tensor-shape-mismatch-error-when-doing-an-allgather-in-distributed-training-with-fsdp/142433/4 "2025-02-24T07:33:14Z")

</div>

I think FSDP is in charge of the accelerate library (or PyTorch itself), but the function that was being executed when the error occurred is from the Transformers library, so I think there is no problem with the Transformers library github.

> **[huggingface/transformers](https://github.com/huggingface/transformers/issues)**
>
> 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX. - huggingface/transformers

> **[huggingface/accelerate](https://github.com/huggingface/accelerate/issues)**
>
> 🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed suppo...

---

<div class="post-metadata">

**Author:** ![tcm03](https://avatars.discourse-cdn.com/v4/letter/t/c0e974/32.png) [@tcm03](https://discuss.huggingface.co/u/tcm03)\
**Post date:** [February 24, 2025, 7:53am UTC](https://discuss.huggingface.co/t/tensor-shape-mismatch-error-when-doing-an-allgather-in-distributed-training-with-fsdp/142433/5 "2025-02-24T07:53:41Z")

</div>

Thank you for your direction.  
I’m not sure how many factors are contributing to the error above, but I’ve just found out that a possible reason is `self.state` in `torch.optim.Optimizer`, as can be seen here: [pytorch/torch/optim/optimizer.py at v2.1.2 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/blob/v2.1.2/torch/optim/optimizer.py#L518)

`self.state` stores the optimizer’s states so when I print it out among ranks, there’s one rank where `self.state` is missing:

```auto
2025-02-24 04:16:18,691 - root - DEBUG - @tcm: In optim.Optimizer.state_dict(): self.state=defaultdict(<class 'dict'>, {})

```

For other ranks, `self.state` is available:

```auto
2025-02-24 04:16:07,667 - root - DEBUG - @tcm: In optim.Optimizer.state_dict(): self.state=defaultdict(<class 'dict'>, {Parameter containing: tensor([-0.0099, -0.0302, -0.0054, ..., -0.0038, -0.0027, -0.0015],
       device='cuda:3', requires_grad=True): {'step': tensor(1.), 'exp_avg': tensor([-2.1631e-11, -6.3862e-11, -3.7596e-11, ..., -1.6094e-12, 7.6394e-12, -6.6093e-12], device='cuda:3'), 'exp_avg_sq': tensor([4.6788e-23, 4.0783e-22, 1.4135e-22, ..., 2.5902e-25, 5.8360e-24, 4.3683e-24], device='cuda:3')}})
2025-02-24 04:16:07,667 - root - DEBUG - @tcm: In optim.Optimizer.state_dict(): self.state=defaultdict(<class 'dict'>, {Parameter containing: tensor([0.0103, 0.0090, 0.0134, ..., 0.0049, -0.0025, -0.0052],
       device='cuda:2', requires_grad=True): {'step': tensor(1.), 'exp_avg': tensor([-1.9513e-08, 5.0979e-09, 1.5206e-08, ..., -2.0051e-10, 9.6136e-11, -1.6412e-10], device='cuda:2'), 'exp_avg_sq': tensor([3.8075e-17, 2.5989e-18, 2.3122e-17, ..., 4.0205e-21, 9.2421e-22, 2.6935e-21], device='cuda:2')}})

```

Since the optimizer I used is `AdamW`, when I look into `adamw.py` I see that for state to be initialized non-empty, it has to go through the first `step()` operation: [pytorch/torch/optim/adamw.py at v2.1.2 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/blob/v2.1.2/torch/optim/adamw.py#L149)  
where `self._init_group` will do the initialization. However, I’m really quite unsure the mechanism behind initializing and manipulating optim states.
