# Significant performance difference between two shapes using Array2D features

**URL:** <https://discuss.huggingface.co/t/significant-performance-difference-between-two-shapes-using-array2d-features/52305>\
**Category:** 🤗Datasets\
**Created:** [August 26, 2023, 10:25pm UTC](https://discuss.huggingface.co/t/significant-performance-difference-between-two-shapes-using-array2d-features/52305 "2023-08-26T22:25:46Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![galopyz](https://avatars.discourse-cdn.com/v4/letter/g/e9a140/32.png) [@galopyz](https://discuss.huggingface.co/u/galopyz)\
**Post date:** [August 26, 2023, 10:25pm UTC](https://discuss.huggingface.co/t/significant-performance-difference-between-two-shapes-using-array2d-features/52305/1 "2023-08-26T22:25:46Z")

</div>

Hello,

I was following Part 2 of FastAI course, and I was trying Hugging Face datasets with Array2D feature on MNIST Fashion data. I also set the format type as Pytorch to gain performance. Image size is 28 by 28, so I used `Array2D(shape=(28, 28), dtype='float32')`, but it was very slow compared to using `Array2D(shape=(1, 28*28), dtype='float32')` when I created dataloaders and looped through it.

When the shape was (28,28), it took 33 seconds to loop through the dataloader, but it only took 6 seconds for (1, 28). Is this performance difference expected? If yes, then how can I gain more performance when using (28, 28) shape?

Here is a [jupyter notebook code](https://colab.research.google.com/drive/1Q96rww5juilYwqG9KWqMItQ3wqx1MurU?usp=sharing) to see what I did.

Thank you.

---

<div class="post-metadata">

**Author:** ![galopyz](https://avatars.discourse-cdn.com/v4/letter/g/e9a140/32.png) [@galopyz](https://discuss.huggingface.co/u/galopyz)\
**Post date:** [September 4, 2023, 8:16pm UTC](https://discuss.huggingface.co/t/significant-performance-difference-between-two-shapes-using-array2d-features/52305/2 "2023-09-04T20:16:49Z")

</div>

Does anybody have an answer to this question?

Thank you.

---

<div class="post-metadata">

**Author:** ![Bjornedt](https://avatars.discourse-cdn.com/v4/letter/b/91b2a8/32.png) [@Bjornedt](https://discuss.huggingface.co/u/Bjornedt)\
**Post date:** [September 5, 2023, 9:33pm UTC](https://discuss.huggingface.co/t/significant-performance-difference-between-two-shapes-using-array2d-features/52305/3 "2023-09-05T21:33:53Z")

</div>

Hi!

The Hugging Face datasets library uses Arrow under the hood. Arrow is optimized for columnar storage, and one possible explanation for the performance difference could be how Arrow handles multi-dimensional arrays versus one-dimensional arrays. Accessing and manipulating data in a one-dimensional array (shape=(1, 28\*28)) could be more efficient in this context than a two-dimensional array (shape=(28, 28)).

Some ideas what you could try to optimize performance with the (28, 28) shape:

- Reshape the data after loading but before feeding it into your model, so you benefit from the fast loading of the one-dimensional array.
- Check if you have the latest versions of the datasets library and Arrow
- Look into parallel processing or asynchronous data loading to speed up the data feeding process.

Hope this helps!

---

<div class="post-metadata">

**Author:** ![galopyz](https://avatars.discourse-cdn.com/v4/letter/g/e9a140/32.png) [@galopyz](https://discuss.huggingface.co/u/galopyz)\
**Post date:** [September 5, 2023, 10:43pm UTC](https://discuss.huggingface.co/t/significant-performance-difference-between-two-shapes-using-array2d-features/52305/4 "2023-09-05T22:43:41Z")

</div>

I appreciate that.

I left the shape as (1, 28\*28) for dataloaders and switched to (28, 28) shape right before using the batch for training.