# DistributedSampler with Accelerate

**URL:** <https://discuss.huggingface.co/t/distributedsampler-with-accelerate/158787>\
**Category:** 🤗Accelerate\
**Created:** [June 10, 2025, 3:02pm UTC](https://discuss.huggingface.co/t/distributedsampler-with-accelerate/158787 "2025-06-10T15:02:30Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![zaryabmakram](https://avatars.discourse-cdn.com/v4/letter/z/a9a28c/32.png) [@zaryabmakram](https://discuss.huggingface.co/u/zaryabmakram)\
**Post date:** [June 10, 2025, 3:02pm UTC](https://discuss.huggingface.co/t/distributedsampler-with-accelerate/158787/1 "2025-06-10T15:02:30Z")

</div>

I have noticed that using `torch.utils.data.distributed.DistributedSampler` as the sampler in the dataloader, accelerate changes its length after calling `accelerate.prepare()`.

Following is how Im setting up the dataloader:

```auto
# build data loader
sampler = torch.utils.data.distributed.DistributedSampler(
    train_dataset, 
    shuffle=True,
)

self.train_loader = torch.utils.data.DataLoader(
    train_dataset,
    shuffle=(sampler is None),
    sampler=sampler,
    batch_size=10,
    drop_last=True,
    num_workers=8,
    pin_memory=True,
    persistent_workers=True,
    prefetch_factor=2,
)

...

# prepare it with accelerate
self.model, self.optimizer, self.train_loader, self.val_loader = self.accelerator.prepare(
    self.model, 
    self.optimizer, 
    self.train_loader, 
    self.val_loader,
)

```

Following is lengths of the dataset Im getting:

### Before Calling accelerator.prepare

- len(self.train\_loader) = 12 (which is expected)

### Before Calling accelerator.prepare

- len(self.train\_loader) = 1

Can someone suggest what the issue could be?

Accelerate version being used is `1.5.2`

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 10, 2025, 4:15pm UTC](https://discuss.huggingface.co/t/distributedsampler-with-accelerate/158787/2 "2025-06-10T16:15:45Z")

</div>

This unsolved issue may have similar symptoms…

> <https://github.com/huggingface/accelerate/issues/3520>
>
> \### System Info
> 
> \`\`\`Shell
> \- \`Accelerate\` version: 1.2.0
> \- Platform: Linux-5.15.0…-116-generic-x86\_64-with-glibc2.31
> \- \`accelerate\` bash location: /home/zhoust/miniconda3/envs/toolkit/bin/accelerate
> \- Python version: 3.12.2
> \- Numpy version: 2.2.0
> \- PyTorch version (GPU?): 2.5.1+cu124 (True)
> \- PyTorch XPU available: False
> \- PyTorch NPU available: False
> \- PyTorch MLU available: False
> \- PyTorch MUSA available: False
> \- System RAM: 1007.50 GB
> \- GPU type: NVIDIA RTX A6000
> \- \`Accelerate\` default config:
> - compute\_environment: LOCAL\_MACHINE
> - distributed\_type: MULTI\_GPU
> - mixed\_precision: bf16
> - use\_cpu: False
> - debug: False
> - num\_processes: 2
> - machine\_rank: 0
> - num\_machines: 1
> - gpu\_ids: 0,1
> - rdzv\_backend: static
> - same\_network: True
> - main\_training\_function: main
> - enable\_cpu\_affinity: False
> - downcast\_bf16: no
> - tpu\_use\_cluster: False
> - tpu\_use\_sudo: False
> - tpu\_env: \[\]
> \`\`\`
> 
> \### Information
> 
> \- \[\] The official example scripts
> \- \[x\] My own modified scripts
> 
> \### Tasks
> 
> \- \[\] One of the scripts in the examples/ folder of Accelerate or an officially supported \`no\_trainer\` script in the \`examples\` folder of the \`transformers\` repo (such as \`run\_no\_trainer\_glue.py\`)
> \- \[x\] My own task or dataset (give details below)
> 
> \### Reproduction
> 
> \`\`\`python
> from accelerate import Accelerator
> from torch.utils.data import DataLoader, Dataset
> from torch.utils.data.distributed import DistributedSampler
> import torch
> import numpy as np
> 
> \# Dummy dataset
> class DummyDataset(Dataset):
> def \_\_init\_\_(self, length=1000):
> self.data = np.arange(length)
> 
> def \_\_len\_\_(self):
> return len(self.data)
> 
> def \_\_getitem\_\_(self, idx):
> return {
> "input": torch.tensor(self.data\[idx\], dtype=torch.float32)
> }
> 
> \# Configs
> BATCH\_SIZE = 32
> TOTAL\_SAMPLES = 128
> GRAD\_ACC\_STEPS = 2
> 
> \# Initialize Accelerator
> accelerator = Accelerator(gradient\_accumulation\_steps=GRAD\_ACC\_STEPS)
> 
> \# Dataset and Dataloader
> dataset = DummyDataset(length=TOTAL\_SAMPLES)
> dataloader = DataLoader(dataset, batch\_size=BATCH\_SIZE, sampler=DistributedSampler(dataset, num\_replicas=accelerator.num\_processes, shuffle=True))
> 
> accelerator.print(f"Dataloader length (this process): {len(dataloader)}")
> \# Prepare with accelerator
> dataloader = accelerator.prepare(dataloader)
> 
> \# Info logging
> accelerator.print("=" \* 50)
> accelerator.print(f"Device: {accelerator.device}")
> accelerator.print(f"Mixed Precision: {accelerator.mixed\_precision}")
> accelerator.print(f"Num processes (world size): {accelerator.num\_processes}")
> accelerator.print(f"Gradient accumulation steps: {GRAD\_ACC\_STEPS}")
> accelerator.print(f"Batch size per process: {BATCH\_SIZE}")
> accelerator.print(f"Total dataset size: {TOTAL\_SAMPLES}")
> accelerator.print(f"Dataloader length (this process): {len(dataloader)}")
> accelerator.print(f"Global steps per epoch: {len(dataloader) \* accelerator.num\_processes}")
> accelerator.print(f"Effective batch size: {BATCH\_SIZE \* accelerator.num\_processes \* GRAD\_ACC\_STEPS}")
> accelerator.print("=" \* 50)
> 
> \# Iterate through dataloader
> for i, batch in enumerate(dataloader):
> with accelerator.accumulate(None):
> # Simulate training step
> input\_data = batch\["input"\]
> print(f'\[{accelerator.device}\] {input\_data.detach().cpu().numpy().tolist()}')
> accelerator.print(f"\[Batch {i}\] batch\['input'\].shape: {batch\['input'\].shape}, dataloader length: {len(dataloader)}")
> 
> \`\`\`
> 
> \### Expected behavior
> 
> The output should be as follows:
> \`\`\`
> Dataloader length (this process): 2
> ==================================================
> Device: cuda:0
> Mixed Precision: bf16
> Num processes (world size): 2
> Gradient accumulation steps: 2
> Batch size per process: 32
> Total dataset size: 128
> Dataloader length (this process): 1
> Global steps per epoch: 2
> Effective batch size: 128
> ==================================================
> \[cuda:0\] \[44.0, 121.0, 31.0, 71.0, 105.0, 15.0, 48.0, 30.0, 104.0, 72.0, 118.0, 53.0, 18.0, 33.0, 10.0, 101.0, 115.0, 35.0, 74.0, 37.0, 114.0, 41.0, 99.0, 26.0, 110.0, 66.0, 65.0, 23.0, 79.0, 36.0, 63.0, 52.0\]
> \[Batch 0\] batch\['input'\].shape: torch.Size(\[32\]), dataloader length: 1
> \[cuda:1\] \[56.0, 46.0, 17.0, 1.0, 54.0, 87.0, 73.0, 112.0, 55.0, 19.0, 103.0, 59.0, 20.0, 108.0, 96.0, 78.0, 61.0, 64.0, 113.0, 89.0, 91.0, 62.0, 42.0, 24.0, 93.0, 100.0, 80.0, 85.0, 43.0, 9.0, 7.0, 83.0\]
> \`\`\`
> 
> 
> \### Issues 
> 
> \- When initiated with distributed sampler, data was split into two shards ( 2 process).
> \- After the \`accelerator.prepare\`, the length of dataloader got split again.
> \- In the end, only \*\*\_half of the samples\_\*\* where iterated in one loop of dataloader, as shown in last three lines.