# Making datasets work with both streaming=True and streaming=False

**URL:** <https://discuss.huggingface.co/t/making-datasets-work-with-both-streaming-true-and-streaming-false/59845>\
**Category:** 🤗Datasets\
**Created:** [October 25, 2023, 11:11am UTC](https://discuss.huggingface.co/t/making-datasets-work-with-both-streaming-true-and-streaming-false/59845 "2023-10-25T11:11:51Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![augustoperes](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@augustoperes](https://discuss.huggingface.co/u/augustoperes)\
**Post date:** [October 25, 2023, 11:11am UTC](https://discuss.huggingface.co/t/making-datasets-work-with-both-streaming-true-and-streaming-false/59845/1 "2023-10-25T11:11:51Z")

</div>

I have a dataset with the following structure on a google cloud bucket:

```auto
v0.1
    ├── 102042348
    │ ├── edges.npy
    │ ├── nodes.npy
    │ ├── wind_pressures.npy
    │ └── wind_velocities.npy
    ├── 102042349
    │ ├── edges.npy
    │ ├── nodes.npy
    │ ├── wind_pressures.npy
    │ └── wind_velocities.npy
    ├── 102042350
    │ ├── edges.npy
    │ ├── nodes.npy
    │ ├── wind_pressures.npy
    │ └── wind_velocities.npy

```

But I am having troubles creating a `_split_generators` function that works with both `streaming=True` and `streaming=False`. At the moment I have this:

```python
    def _split_generators(self, dl_manager):
        # Download and extract the zip file in the bucket.
        downloaded_dir = dl_manager.download_and_extract(self.bucket_url)

        dirs = [
            os.path.join(downloaded_dir, dir_)
            for dir_ in os.listdir(downloaded_dir)
        ]

        return [
            datasets.SplitGenerator(name=datasets.Split.TRAIN,
                                    gen_kwargs={'sim_dir_paths': dirs}),
        ]

```

Which works with `streaming=True` but fails for `streaming=False` with the error:

```auto
File "/home/augusto/Documents/api-data-generation/.env/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1011, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('https://storage.googleapis.com/download/storage/v1/b/ ********* %2F?alt=media')

```

Interestingly enough, if I zip the dataset and upload it to the bucket the exact same function works for `streaming=False` but not for `streaming=True`.

So my question is: How do I get this to work with both `streaming=False` and `streaming=True` at the same time?