# Error in Dataset Map Function

**URL:** <https://discuss.huggingface.co/t/error-in-dataset-map-function/28121>\
**Category:** Beginners\
**Created:** [December 19, 2022, 2:50pm UTC](https://discuss.huggingface.co/t/error-in-dataset-map-function/28121 "2022-12-19T14:50:32Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![slyle](https://avatars.discourse-cdn.com/v4/letter/s/e36b37/32.png) [@slyle](https://discuss.huggingface.co/u/slyle)\
**Post date:** [December 19, 2022, 2:50pm UTC](https://discuss.huggingface.co/t/error-in-dataset-map-function/28121/1 "2022-12-19T14:50:32Z")

</div>

Hello all, I have a dataset object train\_ds.

Output:

```auto
Dataset({
    features: ['filepath', 'class', 'fold'],
    num_rows: 6810
})

```

When I attempt to map using a preprocess function this works correctly:

```auto
def preprocess_function(examples):
      examples['newclass'] = examples['class']
      return examples

train_dataset = train_ds.map(
    preprocess_function,
    batch_size=100,
    batched=True,
    num_proc=4,
    load_from_cache_file=False
)

```

```auto
Dataset({
    features: ['filepath', 'class', 'fold', 'newclass'],
    num_rows: 6810
})

```

However, I cannot define any functions outside preprocess\_function it seems, or something is bugging out.

```auto
def preprocess_function(examples):
    examples['audio'] = [torchaudio.load(path) for path in examples['filepath']]
    return examples

```

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/2/2baee22b2284bbeb19add389655609df92e06229.png)

However I get the same error even if I define a function and attempt to use that function inside of preprocess\_function. It’s as if the function “forgets” all other variables and functions in the notebook during the error.

```auto
def test_function(path):
    return path

#this next function should give me a new column with text filepaths named "audio"
def preprocess_function(examples):
    examples['audio'] = [test_function(path) for path in examples['filepath']]
    return examples

train_dataset = train_ds.map(
    preprocess_function,
    batch_size=100,
    batched=True,
    num_proc=4,
    load_from_cache_file=False
)

```

```auto
    769 return self._value
    770 else:
--> 771 raise self._value

NameError: name 'test_function' is not defined

```

Can someone help me identify what’s going on? I am using this as an example but working in VSCode:

> **[Google Colaboratory](https://colab.research.google.com/github/m3hrdadfi/soxan/blob/main/notebooks/Emotion_recognition_in_Greek_speech_using_Wav2Vec2.ipynb#scrollTo=qLk-eM1DFjtE)**

---

<div class="post-metadata">

**Author:** ![slyle](https://avatars.discourse-cdn.com/v4/letter/s/e36b37/32.png) [@slyle](https://discuss.huggingface.co/u/slyle)\
**Post date:** [December 19, 2022, 3:21pm UTC](https://discuss.huggingface.co/t/error-in-dataset-map-function/28121/2 "2022-12-19T15:21:55Z")

</div>

UPDATE: I have narrowed it down to when num\_proc \> 1. Will search the forums now that I have more specific information. If I can fix it before, someone replies, I will post my solution for someone else.

---

<div class="post-metadata">

**Author:** ![slyle](https://avatars.discourse-cdn.com/v4/letter/s/e36b37/32.png) [@slyle](https://discuss.huggingface.co/u/slyle)\
**Post date:** [March 21, 2023, 3:14pm UTC](https://discuss.huggingface.co/t/error-in-dataset-map-function/28121/4 "2023-03-21T15:14:20Z")

</div>

It had to do with num\_proc. Num\_proc values greater than 1 was giving me errors. I changed it to 1 if I recall correctly.

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [March 22, 2023, 2:42pm UTC](https://discuss.huggingface.co/t/error-in-dataset-map-function/28121/6 "2023-03-22T14:42:00Z")

</div>

Hi! If you are using Windows, you need to put multiprocessing calls inside `if __name__ == " __main__"`. To learn why this is required, check [this SO thread](https://stackoverflow.com/questions/20222534/python-multiprocessing-on-windows-if-name-main).