# How can I drop duplicates on datasets module?

**URL:** <https://discuss.huggingface.co/t/how-can-i-drop-duplicates-on-datasets-module/15369>\
**Category:** Beginners\
**Created:** [March 4, 2022, 6:56pm UTC](https://discuss.huggingface.co/t/how-can-i-drop-duplicates-on-datasets-module/15369 "2022-03-04T18:56:30Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![conceptofmind](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/conceptofmind/32/55565_2.png) [@conceptofmind](https://discuss.huggingface.co/u/conceptofmind)\
**Post date:** [July 5, 2022, 1:32am UTC](https://discuss.huggingface.co/t/how-can-i-drop-duplicates-on-datasets-module/15369/4 "2022-07-05T01:32:10Z")

</div>

Hi,

I made some adaptations to the deduplication scripts in the Code Parrot research repository. Here is what a barebones deduplication script would look like with MinHash and LSH: [GitHub - conceptofmind/Huggingface-deduplicate](https://github.com/conceptofmind/Huggingface-deduplicate)

I also opened up a new post regarding a general use case here: [Minhash Deduplication](https://discuss.huggingface.co/t/minhash-deduplication/19992)

Best,

Enrico

---

_[View the full topic](https://discuss.huggingface.co/t/how-can-i-drop-duplicates-on-datasets-module/15369)._
