# Why is Trainer single-threaded during "Generating split..."?

**URL:** <https://discuss.huggingface.co/t/why-is-trainer-single-threaded-during-generating-split/82309>\
**Category:** 🤗Transformers\
**Created:** [April 17, 2024, 9:55pm UTC](https://discuss.huggingface.co/t/why-is-trainer-single-threaded-during-generating-split/82309 "2024-04-17T21:55:39Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jlmckins](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@jlmckins](https://discuss.huggingface.co/u/jlmckins)\
**Post date:** [April 17, 2024, 9:55pm UTC](https://discuss.huggingface.co/t/why-is-trainer-single-threaded-during-generating-split/82309/1 "2024-04-17T21:55:39Z")

</div>

I am training a 3B LLM from scratch with 1 million text samples, and it takes 1/2 hour to get through the “Generating split train…”. I get ~500 samples per second maximum in bursts. I’m wanting to scale to 200 Million samples, which says it will take 7 days. If I tokenize and pack the data myself using map(), with num\_proc of 64, and large batches, this can be done in less than 8 hours instead of days.

I see that you can pass in a sharded file if you have a streaming dataset, but using streaming gives other errors. How can I trick it to use multiple workers when Trainer is “Generating split” inside of Trainer or SFTTrainer?
