# Dataset.map saves list as numpy array instead of as list

**URL:** <https://discuss.huggingface.co/t/dataset-map-saves-list-as-numpy-array-instead-of-as-list/28823>\
**Category:** 🤗Datasets\
**Created:** [January 3, 2023, 3:33am UTC](https://discuss.huggingface.co/t/dataset-map-saves-list-as-numpy-array-instead-of-as-list/28823 "2023-01-03T03:33:58Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![rimusa](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rimusa/32/21498_2.png) [@rimusa](https://discuss.huggingface.co/u/rimusa)\
**Post date:** [January 3, 2023, 3:33am UTC](https://discuss.huggingface.co/t/dataset-map-saves-list-as-numpy-array-instead-of-as-list/28823/1 "2023-01-03T03:33:58Z")

</div>

Hi! I have been trying to have a function that ouputs a list to map it to my dataset. However, when trying to pass the output to a csv or a DataFrame, it appears as a Numpy array instead of as a list. So, if I have the following function:

```
    def fetch_embedding(data):
        text = data["text"]
        out = trainer.predict(text)
        embeddings = out[0][1][-1][:,0,:]
        embeddings = embeddings.tolist()
        return {"embeddings" : embeddings}

```

So, I’m very intentionally passing the torch tensor to a list. Then I map it to the dataset to save these embeddings:

```
dataset = dataset.map(fetch_embedding)

```

We check whether the dataset stored the list as an actual list:

```
df = datset.to_pandas()
A = df.iloc[0].loc["embeddings"]
print(type(A))

```

The output of this is the following:

> \<class ‘numpy.ndarray’\>

Is there any way to actually have the output of the map saved as a list instead of it being passed as a Numpy array?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [January 3, 2023, 10:36am UTC](https://discuss.huggingface.co/t/dataset-map-saves-list-as-numpy-array-instead-of-as-list/28823/2 "2023-01-03T10:36:44Z")

</div>

Hi ! This behavior comes from Arrow:

```python
>>> import pyarrow as pa
>>> df = pa.table({"embeddings": [[0, 1, 2]]}).to_pandas()
>>> df.iloc[0].loc["embeddings"]
array([0, 1, 2])

```

Arrow uses numpy arrays to enable fast zero-copy conversions from Arrow to pandas. Converting to a python list requires to copy the data into new pythons objects, which can be costly in some cases.

You can get a python list using

```python
df["embeddings"] = [arr.tolist() for arr in df["embeddings"]]

```

---

<div class="post-metadata">

**Author:** ![rimusa](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rimusa/32/21498_2.png) [@rimusa](https://discuss.huggingface.co/u/rimusa)\
**Post date:** [January 3, 2023, 5:01pm UTC](https://discuss.huggingface.co/t/dataset-map-saves-list-as-numpy-array-instead-of-as-list/28823/3 "2023-01-03T17:01:55Z")

</div>

That makes a lot of sense! Thanks for the answer 🙂
