# How to use split\_dataset\_by\_node and shuffle on iterable dataset

**URL:** <https://discuss.huggingface.co/t/how-to-use-split-dataset-by-node-and-shuffle-on-iterable-dataset/102660>\
**Category:** 🤗Datasets\
**Created:** [August 16, 2024, 3:46pm UTC](https://discuss.huggingface.co/t/how-to-use-split-dataset-by-node-and-shuffle-on-iterable-dataset/102660 "2024-08-16T15:46:56Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![irow](https://avatars.discourse-cdn.com/v4/letter/i/a4c791/32.png) [@irow](https://discuss.huggingface.co/u/irow)\
**Post date:** [August 16, 2024, 3:46pm UTC](https://discuss.huggingface.co/t/how-to-use-split-dataset-by-node-and-shuffle-on-iterable-dataset/102660/1 "2024-08-16T15:46:56Z")

</div>

I’m currently training a multi-gpu single node model. I am using an IterableDataset in streaming mode. However, after one epoch it hangs then crashes after a timeout. Doing extensive googling I’ve come to believe that it’s the way I’m using `split_dataset_by_node` and `shuffle`. I believe the GPU’s are getting different sizes of batches.

Here’s my full dataloading/dataset code.

```python
 def load_train_objs(dataset_path):
     dataset_path_dict = {
        "train": train_files,
        "val": val_files,
    }

    dataset = load_dataset("parquet", data_files=dataset_path_dict, streaming=True)
    dataset = dataset.shuffle(seed=seed)
    ...
    return dataset

def prepare_data_loader(dataset, batch_size, rank, world_size):
    train_ds = dataset["train"]
    train_ds = split_dataset_by_node(train_ds, rank=rank, world_size=world_size)

    val_ds = dataset["val"]
    val_ds = split_dataset_by_node(val_ds, rank=rank, world_size=world_size)

    train_dl = DataLoader(
        train_ds,
        batch_size,
        num_workers=8,
        drop_last=True,
        collate_fn=collate_fn,
    )

    val_dl = DataLoader(val_ds, batch_size, drop_last=False, collate_fn=collate_fn)

    return train_dl, val_dl

```

I’m not sure what at what point I need to use shuffle?  
Should it be where it currently is or should it be after  
`train_ds = dataset["train"]`  
or even after  
`train_ds = split_dataset_by_node(train_ds, rank=rank, world_size=world_size)`

Any help would be greatly appreciated 🙂

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [August 19, 2024, 1:24pm UTC](https://discuss.huggingface.co/t/how-to-use-split-dataset-by-node-and-shuffle-on-iterable-dataset/102660/2 "2024-08-19T13:24:06Z")

</div>

Hi ! `split_dataset_by_node` may not return datasets with the same size, so your training may time out waiting when one node runs out of examples

---

<div class="post-metadata">

**Author:** ![dhruveshpatel](https://avatars.discourse-cdn.com/v4/letter/d/d07c76/32.png) [@dhruveshpatel](https://discuss.huggingface.co/u/dhruveshpatel)\
**Post date:** [February 16, 2025, 1:42am UTC](https://discuss.huggingface.co/t/how-to-use-split-dataset-by-node-and-shuffle-on-iterable-dataset/102660/3 "2025-02-16T01:42:12Z")

</div>

When I use `.shuffle()` after `split_dataset_by_node`, I see same examples on different ranks in the same epoch. I could not find any documentation for this scenario.

Update: I tried using shuffle before split\_dataset\_by\_node and got the same results. The examples are repeated in the same epoch across ranks.

Update: The issue was that I was giving rank dependent seed to shuffle. If I give the same seed to shuffle on all ranks, then the problem goes away.

---

<div class="post-metadata">

**Author:** ![benstokes](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/benstokes/32/32007_2.png) [@benstokes](https://discuss.huggingface.co/u/benstokes)\
**Post date:** [February 17, 2025, 3:28am UTC](https://discuss.huggingface.co/t/how-to-use-split-dataset-by-node-and-shuffle-on-iterable-dataset/102660/4 "2025-02-17T03:28:33Z")

</div>

Thanks for sharing. It helps me a lot[.](https://www.miocado-net.uk/)

---

<div class="post-metadata">

**Author:** ![Zoe0427](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/zoe0427/32/53729_2.png) [@Zoe0427](https://discuss.huggingface.co/u/Zoe0427)\
**Post date:** [September 13, 2025, 3:21am UTC](https://discuss.huggingface.co/t/how-to-use-split-dataset-by-node-and-shuffle-on-iterable-dataset/102660/5 "2025-09-13T03:21:08Z")

</div>

Hi! When `dataset.num_shards % world_size == 0`, each node will have the same size, right?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [September 15, 2025, 8:43am UTC](https://discuss.huggingface.co/t/how-to-use-split-dataset-by-node-and-shuffle-on-iterable-dataset/102660/6 "2025-09-15T08:43:02Z")

</div>

> When `dataset.num_shards % world_size == 0`, each node will have the same size, right?

In this case each node has the same number of shards to load. This allows each node to have roughly the same amount of data. But it’s likely that the shards don’t have the exact same number of examples