# Dataset map return only list instead torch tensors **URL:** \ **Category:** Beginners\ **Created:** [March 16, 2022, 9:56am UTC](https://discuss.huggingface.co/t/dataset-map-return-only-list-instead-torch-tensors/15767 "2022-03-16T09:56:28Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![MPA](https://avatars.discourse-cdn.com/v4/letter/m/ee59a6/32.png) [@MPA](https://discuss.huggingface.co/u/MPA)\ **Post date:** [March 16, 2022, 9:56am UTC](https://discuss.huggingface.co/t/dataset-map-return-only-list-instead-torch-tensors/15767/1 "2022-03-16T09:56:28Z")
when i use the map on Dataset object the 'input\_ids return as list instead of tensors ```auto def tokenize(batch): return tokenizer(batch['text'],padding=True, return_tensors='pt', truncation=True).to(DEVICE) ``` `data.map(tokenize, batched=False, batch_size=None)` → return list `tokenizer(data['text'],padding=True, return_tensors='pt', truncation=True).to(DEVICE)` → return tensor ---
**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\ **Post date:** [March 21, 2022, 12:18pm UTC](https://discuss.huggingface.co/t/dataset-map-return-only-list-instead-torch-tensors/15767/2 "2022-03-21T12:18:12Z")
Hi! `map` ignores tensor formatting while writing a cache file, so to get PyTorch tensors under the `input_ids` column, you need to explicitly call `set_format("pt", columns=["input_ids"], output_all_columns=True)` on the dataset object (after `map`). ---
**Author:** ![murphyk](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/murphyk/32/11889_2.png) [@murphyk](https://discuss.huggingface.co/u/murphyk)\ **Post date:** [September 29, 2022, 10:36pm UTC](https://discuss.huggingface.co/t/dataset-map-return-only-list-instead-torch-tensors/15767/3 "2022-09-29T22:36:57Z")
I got burned by this as well. Maybe the documentation (examples) for map could be updated to explain this? ---
**Author:** ![Davidg707](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/davidg707/32/12471_2.png) [@Davidg707](https://discuss.huggingface.co/u/Davidg707)\ **Post date:** [December 21, 2022, 4:18am UTC](https://discuss.huggingface.co/t/dataset-map-return-only-list-instead-torch-tensors/15767/4 "2022-12-21T04:18:44Z")
I got burned by this too! Definitely seems like an issue in the library that `.map` doesn’t return the objects that the mapped function returns. And I’d just finished the course which has a [whole section](https://huggingface.co/course/chapter8/2?fw=pt) about bugs caused by not setting `return_tensors`, but the examples never mention this issue with `.map`. ---
**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\ **Post date:** [December 22, 2022, 7:49pm UTC](https://discuss.huggingface.co/t/dataset-map-return-only-list-instead-torch-tensors/15767/5 "2022-12-22T19:49:18Z")
Hi! When it comes to tensors, PyArrow (the storage format we use) only understands 1D arrays, so we would have to store (potentially) a significant amount of metadata to be able to restore the types after `map` fully. Also, a `map` transform can return different value types for the same column (e.g. PyTorch tensors or Python lists), which would make this process ambiguous (e.g. should we return mixed types or only one type when indexing the dataset afterwards). So for the sake of simplicity, we return Python objects by default, and for other formats, one can use `with_format`/`with_transform`. ---
**Author:** ![sd3ntato](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@sd3ntato](https://discuss.huggingface.co/u/sd3ntato)\ **Post date:** [June 27, 2023, 11:01am UTC](https://discuss.huggingface.co/t/dataset-map-return-only-list-instead-torch-tensors/15767/6 "2023-06-27T11:01:18Z")
I ended up using a data collator to take the lists and turn them back into tensors ---
**Author:** ![kenfus](https://avatars.discourse-cdn.com/v4/letter/k/71e660/32.png) [@kenfus](https://discuss.huggingface.co/u/kenfus)\ **Post date:** [March 1, 2024, 4:12pm UTC](https://discuss.huggingface.co/t/dataset-map-return-only-list-instead-torch-tensors/15767/7 "2024-03-01T16:12:13Z")
Yep, I also got burned by this. What is the point here to return list of lists when it’s a torch tensor? Inside the map function, it’s a tensor, then after it’s a list? ---
**Author:** ![hammer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/hammer/32/25347_2.png) [@hammer](https://discuss.huggingface.co/u/hammer)\ **Post date:** [April 10, 2024, 3:48pm UTC](https://discuss.huggingface.co/t/dataset-map-return-only-list-instead-torch-tensors/15767/8 "2024-04-10T15:48:41Z")
Also burned by this coercion, would be helpful to note the PyArrow limitation (although [Tensor](https://arrow.apache.org/docs/format/Other.html#tensor-multi-dimensional-array) may be helpful here?) and workarounds in the docs… Edit: looks like there’s some discussion on how to use PyArrow’s Tensor type at [Use pyarrow Tensor dtype · Issue #5272 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/5272) ---
**Author:** ![Pavel6453](https://avatars.discourse-cdn.com/v4/letter/p/53a042/32.png) [@Pavel6453](https://discuss.huggingface.co/u/Pavel6453)\ **Post date:** [March 17, 2025, 12:20pm UTC](https://discuss.huggingface.co/t/dataset-map-return-only-list-instead-torch-tensors/15767/9 "2025-03-17T12:20:36Z")
Damn that’s sneaky! I just spent a whole day debugging this thing. I mean, tokenizer does return pytorch tensors, but not if called within Map, Map just ignores that argument! Well, turns out it was some internal implementation thing that’s supposed to be “expected”. I really wish that it was explicitly stated in the docs, not hidden like that