# Datasets behaving strange when calling filter twice

**URL:** <https://discuss.huggingface.co/t/datasets-behaving-strange-when-calling-filter-twice/10251>\
**Category:** 🤗Datasets\
**Created:** [September 23, 2021, 4:09pm UTC](https://discuss.huggingface.co/t/datasets-behaving-strange-when-calling-filter-twice/10251 "2021-09-23T16:09:27Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mrdrozdov](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mrdrozdov/32/2692_2.png) [@mrdrozdov](https://discuss.huggingface.co/u/mrdrozdov)\
**Post date:** [September 23, 2021, 4:09pm UTC](https://discuss.huggingface.co/t/datasets-behaving-strange-when-calling-filter-twice/10251/1 "2021-09-23T16:09:27Z")

</div>

I think I’m seeing unexpected behavior when calling filter twice in succession. I wonder if it’s related to caching?

Here’s an easy reproduction:

```auto
from datasets import load_dataset

def debug_log(key, d):
    print(key, len(d), set(d['language']))

d = load_dataset('tydiqa', 'primary_task', split='validation')
debug_log('d', d)
d_filter_0 = d.filter(lambda x: x['language'] == 'english')
debug_log('d_filter_0', d_filter_0)
d_filter_1 = d_filter_0.filter(lambda x: x['language'] == 'english')
debug_log('d_filter_1', d_filter_1)
d_filter_2 = d.filter(lambda x: x['language'] == 'english')
debug_log('d_filter_2', d_filter_2)

```

Output (notice that languages for d\_filter\_0 != d\_filter\_1):

```auto
Reusing dataset tydiqa (/Users/adrozdov/.cache/huggingface/datasets/tydiqa/primary_task/1.0.0/b8a6c4c0db10bf5703d7b36645e5dbae821b8c0e902dac9daeecd459a8337148)
d 18670 {'bengali', 'swahili', 'english', 'thai', 'telugu', 'indonesian', 'russian', 'arabic', 'finnish', 'japanese', 'korean'}
Loading cached processed dataset at /Users/adrozdov/.cache/huggingface/datasets/tydiqa/primary_task/1.0.0/b8a6c4c0db10bf5703d7b36645e5dbae821b8c0e902dac9daeecd459a8337148/cache-1c4d9d00950a4a3d.arrow
d_filter_0 1031 {'english'}
Loading cached processed dataset at /Users/adrozdov/.cache/huggingface/datasets/tydiqa/primary_task/1.0.0/b8a6c4c0db10bf5703d7b36645e5dbae821b8c0e902dac9daeecd459a8337148/cache-6c4a6a69baa0ff73.arrow
d_filter_1 1031 {'bengali', 'swahili', 'english', 'thai', 'telugu', 'indonesian', 'russian', 'arabic', 'finnish', 'japanese', 'korean'}
Loading cached processed dataset at /Users/adrozdov/.cache/huggingface/datasets/tydiqa/primary_task/1.0.0/b8a6c4c0db10bf5703d7b36645e5dbae821b8c0e902dac9daeecd459a8337148/cache-1c4d9d00950a4a3d.arrow
d_filter_2 1031 {'english'}

```

It would be great to call filter twice! Basically, can filter the first time by language, and the second time by something else like length.

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [September 23, 2021, 9:35pm UTC](https://discuss.huggingface.co/t/datasets-behaving-strange-when-calling-filter-twice/10251/2 "2021-09-23T21:35:57Z")

</div>

Hey @mrdrozdov instead of chaining multiple filter operations, I think you can use conditionals in the lambda function like:

```auto
dset.filter(lambda x: x['language'] == 'english' and x['language'] == 'german')

```

---

<div class="post-metadata">

**Author:** ![mrdrozdov](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mrdrozdov/32/2692_2.png) [@mrdrozdov](https://discuss.huggingface.co/u/mrdrozdov)\
**Post date:** [September 23, 2021, 10:56pm UTC](https://discuss.huggingface.co/t/datasets-behaving-strange-when-calling-filter-twice/10251/3 "2021-09-23T22:56:25Z")

</div>

Thanks! Yeah as long as I don’t chain filter it seems to work fine. Still seems like an issue though. Maybe should throw an exception if chaining filter?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [October 6, 2021, 9:28am UTC](https://discuss.huggingface.co/t/datasets-behaving-strange-when-calling-filter-twice/10251/4 "2021-10-06T09:28:08Z")

</div>

Hi ! Yes this is a bug in versions 1.12.0 and 1.12.1  
It has recently been fixed on `master` and we’ll do a new release.  
see [Fix filter leaking by lhoestq · Pull Request #3019 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/pull/3019)
