# Processing Large Dataset for Training GPT2 model

**URL:** <https://discuss.huggingface.co/t/processing-large-dataset-for-training-gpt2-model/35577>\
**Category:** 🤗Datasets\
**Created:** [April 5, 2023, 12:04am UTC](https://discuss.huggingface.co/t/processing-large-dataset-for-training-gpt2-model/35577 "2023-04-05T00:04:18Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![shivamsharma2705](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@shivamsharma2705](https://discuss.huggingface.co/u/shivamsharma2705)\
**Post date:** [April 5, 2023, 12:04am UTC](https://discuss.huggingface.co/t/processing-large-dataset-for-training-gpt2-model/35577/1 "2023-04-05T00:04:19Z")

</div>

I am working with a very big datasource (230M documents) and am trying to train a GPT2 style model using run\_clm.py script with Deepspeed. There is a grouping function in the run\_clm.py script ([transformers/run\_clm.py at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py#L490)) which breaks the data into multiple tokens of max\_sequence length.

Since my data is so big, the total time showing for me is around 10 days., which is way too much for pre-processing a data. Is there a way I can fasten up the process?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [April 6, 2023, 12:06pm UTC](https://discuss.huggingface.co/t/processing-large-dataset-for-training-gpt2-model/35577/2 "2023-04-06T12:06:03Z")

</div>

Hi ! Have you tried increasing `preprocessing_num_workers` ?

---

<div class="post-metadata">

**Author:** ![shivamsharma2705](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@shivamsharma2705](https://discuss.huggingface.co/u/shivamsharma2705)\
**Post date:** [April 6, 2023, 3:15pm UTC](https://discuss.huggingface.co/t/processing-large-dataset-for-training-gpt2-model/35577/3 "2023-04-06T15:15:36Z")

</div>

Yeah, I have the maximum number of processors that my system has. Is there a way to replace this function with something which is faster?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [April 7, 2023, 5:43pm UTC](https://discuss.huggingface.co/t/processing-large-dataset-for-training-gpt2-model/35577/4 "2023-04-07T17:43:38Z")

</div>

This function uses pythons lists, which are known to be slow. Using NumPy is significantly faster since it’s integrated with Apache Arrow - the format used to store the datasets.

Therefore you should manipulate NumPy arrays instead. To do that, use the “numpy” formatting:

```python
tokenized_datasets = tokenized_datasets.with_format("numpy")
lm_datasets = tokenized_datasets.map(...)

```

And you can modify your map function to work with NumPy arrays only, instead of lists.

(it can even be slightly faster using the “arrow” formatting, but it would require to rewrite your map function completely so I wouldn’t suggest it)

---

<div class="post-metadata">

**Author:** ![conceptofmind](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/conceptofmind/32/55565_2.png) [@conceptofmind](https://discuss.huggingface.co/u/conceptofmind)\
**Post date:** [April 12, 2023, 12:32am UTC](https://discuss.huggingface.co/t/processing-large-dataset-for-training-gpt2-model/35577/5 "2023-04-12T00:32:04Z")

</div>

I was also noticing this “issue” as well when trying to map the Pile. The estimated time would have been ~10 days to tokenize on 48 proc. I will try with numpy.