# Correct way to use multiple workers with interleave\_datasets for iterable datasets

**URL:** <https://discuss.huggingface.co/t/correct-way-to-use-multiple-workers-with-interleave-datasets-for-iterable-datasets/93544>\
**Category:** 🤗Datasets\
**Created:** [June 22, 2024, 10:23pm UTC](https://discuss.huggingface.co/t/correct-way-to-use-multiple-workers-with-interleave-datasets-for-iterable-datasets/93544 "2024-06-22T22:23:54Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![alex-hh](https://avatars.discourse-cdn.com/v4/letter/a/c6cbf5/32.png) [@alex-hh](https://discuss.huggingface.co/u/alex-hh)\
**Post date:** [June 22, 2024, 10:23pm UTC](https://discuss.huggingface.co/t/correct-way-to-use-multiple-workers-with-interleave-datasets-for-iterable-datasets/93544/1 "2024-06-22T22:23:54Z")

</div>

I would like to use multiple workers and interleave datasets with iterable datasets.

What is the best way to do this? I gather from this PR [Implement sharding on merged iterable datasets by Hubert-Bonisseur · Pull Request #5735 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/pull/5735) that the individual datasets should be sharded? How is this achieved if using load\_dataset?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [July 3, 2024, 12:53pm UTC](https://discuss.huggingface.co/t/correct-way-to-use-multiple-workers-with-interleave-datasets-for-iterable-datasets/93544/2 "2024-07-03T12:53:18Z")

</div>

Sharding depends on the dataset, one shard corresponds to one file.

Sharding is quite important to enable the use of num\_workers in in data loaders, so feel free to use datasets that are already sharded or shard a dataset your self, e.g. using:

```python
from datasets import load_dataset

ds = load_dataset(...)
ds.push_to_hub(repo_id, num_shards=...)

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [July 4, 2024, 1:08pm UTC](https://discuss.huggingface.co/t/correct-way-to-use-multiple-workers-with-interleave-datasets-for-iterable-datasets/93544/3 "2024-07-04T13:08:24Z")

</div>

Closed on July 4, 2024, 1:08pm UTC

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.
