# How does one make dataset.take(512) work with streaming = False with hugging face data set?

**URL:** <https://discuss.huggingface.co/t/how-does-one-make-dataset-take-512-work-with-streaming-false-with-hugging-face-data-set/50770>\
**Category:** Beginners\
**Created:** [August 15, 2023, 12:34am UTC](https://discuss.huggingface.co/t/how-does-one-make-dataset-take-512-work-with-streaming-false-with-hugging-face-data-set/50770 "2023-08-15T00:34:32Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\
**Post date:** [August 15, 2023, 12:34am UTC](https://discuss.huggingface.co/t/how-does-one-make-dataset-take-512-work-with-streaming-false-with-hugging-face-data-set/50770/1 "2023-08-15T00:34:32Z")

</div>

I get the error:

```auto
Exception has occurred: AttributeError (note: full exception trace is shown but execution is paused at: _run_module_as_main)
'Dataset' object has no attribute 'take'
  File "/lfs/ampere1/0/brando9/beyond-scale-language-data-diversity/src/diversity/div_coeff.py", line 499, in experiment_compute_diveristy_coeff_single_dataset_then_combined_datasets_with_domain_weights
    batch = dataset.take(batch_size)
  File "/lfs/ampere1/0/brando9/beyond-scale-language-data-diversity/src/diversity/div_coeff.py", line 552, in <module>
    experiment_compute_diveristy_coeff_single_dataset_then_combined_datasets_with_domain_weights()
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/lfs/ampere1/0/brando9/miniconda/envs/beyond_scale/lib/python3.10/runpy.py", line 196, in _run_module_as_main (Current frame)
    return _run_code(code, main_globals, None,
AttributeError: 'Dataset' object has no attribute 'take'

```

only happens with `streaming = False`. How to fix? I do NOT want to stream the data and **I want .take to work**.

* * *

# idea: convert HF Dataset to HF datasets.iterable\_dataset.IterableDataset

This seem to have worked:

```auto
    print(f'{dataset=}')
    print(f'{type(dataset)=}')
    # datasets.iterable_dataset.IterableDataset
    # datasets.arrow_dataset.Dataset
    dataset = IterableDataset(dataset) if type(dataset) != IterableDataset else dataset # to force dataset.take(batch_size) to work in non-streaming mode
    batch = dataset.take(batch_size)

```

seems to work? Takes long to fetch batch.

* * *

# Idea 2: Collate fn

If the custom collate fns actually worked this would be simple since the collate fn would receive a batch of batch\_size already. See it doesn’t work here: [https://stackoverflow.com/questions/76879872/how-to-use-huggingface-hf-trainer-train-with-custom-collate-function](https://stackoverflow.com/questions/76879872/how-to-use-huggingface-hf-trainer-train-with-custom-collate-function) with trainer. Actually but I don’t want to use the trainer…

* * *

related gitissue: [Allow dataset implement .take · Issue #6150 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/6150)  
so: [https://stackoverflow.com/questions/76902824/how-does-one-make-dataset-take512-work-with-streaming-false-with-hugging-fac](https://stackoverflow.com/questions/76902824/how-does-one-make-dataset-take512-work-with-streaming-false-with-hugging-fac)

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [August 15, 2023, 1:41pm UTC](https://discuss.huggingface.co/t/how-does-one-make-dataset-take-512-work-with-streaming-false-with-hugging-face-data-set/50770/2 "2023-08-15T13:41:06Z")

</div>

Hi ! You can replace .take(512) by .select(range(512))

Take hasn’t been implemented yet but will be easy to add

---

<div class="post-metadata">

**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\
**Post date:** [August 16, 2023, 6:08pm UTC](https://discuss.huggingface.co/t/how-does-one-make-dataset-take-512-work-with-streaming-false-with-hugging-face-data-set/50770/3 "2023-08-16T18:08:32Z")

</div>

@lhoestq can we implement this? So that the code doesn’t crash? Easier to mantain a single code base?

gitissue I made: [Allow dataset implement .take · Issue #6150 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/6150)

---

<div class="post-metadata">

**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\
**Post date:** [August 16, 2023, 6:10pm UTC](https://discuss.huggingface.co/t/how-does-one-make-dataset-take-512-work-with-streaming-false-with-hugging-face-data-set/50770/4 "2023-08-16T18:10:26Z")

</div>

I will clarify with the full code because batch objects are also dataset objects so your answer is slightly ambiguous. Just for clarify, you mean this:

```auto
batch = dataset.select(range(512))

```

right? Thanks btw! 🙂

---

<div class="post-metadata">

**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\
**Post date:** [August 16, 2023, 6:11pm UTC](https://discuss.huggingface.co/t/how-does-one-make-dataset-take-512-work-with-streaming-false-with-hugging-face-data-set/50770/5 "2023-08-16T18:11:03Z")

</div>

will range always give me the same data points? Any random numbers I need to generate?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [August 18, 2023, 2:51pm UTC](https://discuss.huggingface.co/t/how-does-one-make-dataset-take-512-work-with-streaming-false-with-hugging-face-data-set/50770/6 "2023-08-18T14:51:32Z")

</div>

Sure, it simply select the examples from index 0 to 512.
