# How to load a large hf dataset efficiently?

**URL:** <https://discuss.huggingface.co/t/how-to-load-a-large-hf-dataset-efficiently/69288>\
**Category:** 🤗Datasets\
**Created:** [January 16, 2024, 3:52am UTC](https://discuss.huggingface.co/t/how-to-load-a-large-hf-dataset-efficiently/69288 "2024-01-16T03:52:35Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![aaditya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aaditya/32/20855_2.png) [@aaditya](https://discuss.huggingface.co/u/aaditya)\
**Post date:** [January 16, 2024, 7:22pm UTC](https://discuss.huggingface.co/t/how-to-load-a-large-hf-dataset-efficiently/69288/3 "2024-01-16T19:22:51Z")

</div>

Thank you for the reply, following your suggestion is there any way in hf dataset, split into multiple shards to be loaded in parallel?

---

_[View the full topic](https://discuss.huggingface.co/t/how-to-load-a-large-hf-dataset-efficiently/69288)._
