# Loading data from Datasets takes too much memory

**URL:** <https://discuss.huggingface.co/t/loading-data-from-datasets-takes-too-much-memory/69503>\
**Category:** 🤗Datasets\
**Created:** [January 17, 2024, 2:47pm UTC](https://discuss.huggingface.co/t/loading-data-from-datasets-takes-too-much-memory/69503 "2024-01-17T14:47:46Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dakhoo](https://avatars.discourse-cdn.com/v4/letter/d/ed655f/32.png) [@Dakhoo](https://discuss.huggingface.co/u/Dakhoo)\
**Post date:** [January 17, 2024, 2:47pm UTC](https://discuss.huggingface.co/t/loading-data-from-datasets-takes-too-much-memory/69503/1 "2024-01-17T14:47:46Z")

</div>

I am dealing with audio files and have a code to create HF dataset from pandas

![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/a/6/a6589bf63798223386609c71c42f8116c51f8306.png)

it is quiet fast however, if I do something like `hg_data['train']['audio'][0]` to see it if can load the audio (numpy array) my sagemaker notebook crashes (even with 16g memory). or if I do "import

```auto
IPython.display as ipd
ipd.Audio(data['train']['audio'][0],
          rate=22050)"

```

P.S  
my features is like

```auto
features = Features({
    'tag': ClassLabel(names=unique_genres, id=None),
    'SegID': Value(dtype='int32', id=None),
    'value': Value(dtype='int64', id=None),
    'audio': Value(dtype='string', id=None)
})

_df_train = Dataset.from_pandas(_df_train,
                               features=features)
_df_train = _df_train.cast_column("audio", Audio(sampling_rate=16000))

```

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [January 18, 2024, 12:20pm UTC](https://discuss.huggingface.co/t/loading-data-from-datasets-takes-too-much-memory/69503/2 "2024-01-18T12:20:28Z")

</div>

Hi ! It’s because `hg_data['train']` returns the full list in RAM of all the audio data in the dataset.

So instead of using `data['train']['audio'][0]` you should simply use `data['train'][0]['audio']` (which loads only the first example in memory)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [January 19, 2024, 12:21am UTC](https://discuss.huggingface.co/t/loading-data-from-datasets-takes-too-much-memory/69503/3 "2024-01-19T00:21:24Z")

</div>

Closed on January 19, 2024, 12:21am UTC

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.