# multiprocess.pool.RemoteTraceback and TypeError: Couldn't cast array of type string to null when loading Hugging Face dataset **URL:** \ **Category:** Beginners\ **Created:** [September 23, 2024, 4:04pm UTC](https://discuss.huggingface.co/t/multiprocess-pool-remotetraceback-and-typeerror-couldnt-cast-array-of-type-string-to-null-when-loading-hugging-face-dataset/108166 "2024-09-23T16:04:35Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\ **Post date:** [September 23, 2024, 4:04pm UTC](https://discuss.huggingface.co/t/multiprocess-pool-remotetraceback-and-typeerror-couldnt-cast-array-of-type-string-to-null-when-loading-hugging-face-dataset/108166/1 "2024-09-23T16:04:35Z")
I’m encountering an error while trying to load and process the GAIR/MathPile dataset using the Hugging Face datasets library. The error seems to occur during type casting in pyarrow within a multiprocessing environment. Below is the code I’m using ```python from datasets import Dataset, load_dataset import os def get_hf_dataset_gair(path: str = '~/data/GAIR/MathPile/train/') -> Dataset: path: str = os.path.expanduser(path) dataset = load_dataset(path, split='train', num_proc=os.cpu_count()) print(dataset[0]) # Preview a single example from the dataset # Remove unnecessary columns all_columns = dataset.column_names all_columns.remove('text') dataset = dataset.remove_columns(all_columns) # Shuffle and select 10k examples dataset = dataset.shuffle(seed=42) dataset = dataset.select(10_000) return dataset # get it get_hf_dataset_gair() ``` Download GAIR ```bash source $AFS/.bashrc conda activate beyond_scale_2 mkdir -p ~/data/GAIR/MathPile huggingface-cli download --resume-download --repo-type dataset GAIR/MathPile --local-dir ~/data/GAIR/MathPile --local-dir-use-symlinks False cd ~/data/GAIR/MathPile/ find . -type f -name "*.gz" -exec gzip -d {} \; ``` I get the following error when running the code: ```auto multiprocess.pool.RemoteTraceback: """ Traceback (most recent call last): File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale_2/lib/python3.11/site-packages/datasets/builder.py", line 1869, in _prepare_split_single writer.write_table(table) ... TypeError: Couldn't cast array of type string to null ``` Here’s the full stack trace: ```bash Traceback (most recent call last): File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale_2/lib/python3.11/site-packages/datasets/builder.py", line 1869, in _prepare_split_single writer.write_table(table) File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale_2/lib/python3.11/site-packages/datasets/arrow_writer.py", line 580, in write_table pa_table = table_cast(pa_table, self._schema) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/lfs/hyperturing1/0/brando9/miniconda/envs/beyond_scale_2/lib/python3.11/site-packages/datasets/table.py", line 2283, in table_cast return cast_table_to_schema(table, schema) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ... TypeError: Couldn't cast array of type string to null ``` It seems to be related to type casting between pyarrow types. I suspect it has something to do with the dataset schema, but I’m not sure how to resolve the error. I’ve verified that the dataset is correctly downloaded, and I’m using the following environment: Hugging Face datasets version: 2.x.x Python 3.11 OS: Linux (running on a server) Multiprocessing is set to use all available CPUs (num\_proc=os.cpu\_count()) Has anyone encountered this issue before, or does anyone have suggestions on how to fix it? - ref: [GAIR/MathPile · Still errors with GAIR loading dataset](https://huggingface.co/datasets/GAIR/MathPile/discussions/3) - ref: [multiprocess.pool.RemoteTraceback and TypeError: Couldn't cast array of type string to null when loading Hugging Face dataset](https://discuss.huggingface.co/t/multiprocess-pool-remotetraceback-and-typeerror-couldnt-cast-array-of-type-string-to-null-when-loading-hugging-face-dataset/108166) - ref: [python - multiprocess.pool.RemoteTraceback and TypeError: Couldn't cast array of type string to null when loading Hugging Face dataset - Stack Overflow](https://stackoverflow.com/questions/79012832/multiprocess-pool-remotetraceback-and-typeerror-couldnt-cast-array-of-type-str) ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [September 24, 2024, 4:56am UTC](https://discuss.huggingface.co/t/multiprocess-pool-remotetraceback-and-typeerror-couldnt-cast-array-of-type-string-to-null-when-loading-hugging-face-dataset/108166/2 "2024-09-24T04:56:56Z")
Is this? > > > \### Describe the bug > > Processing a dataset I alredy uploaded to the Hub (https…://huggingface.co/datasets/tj-solergibert/Europarl-ST) I found that for some splits and some languages (test split, source\_lang = "nl") after applying a map function I get the mentioned error. > > I alredy tried reseting the shorter strings (reset\_cortas function). It only happends with NL, PL, RO and PT. It does not make sense since when processing the other languages I also use the corpus of those that fail and it does not cause any errors. > > I suspect that the error may be in this direction: > > We use cast\_array\_to\_feature to support casting to custom types like Audio and Image # Also, when trying type "string", we don't want to convert integers or floats to "string". # We only do it if trying\_type is False - since this is what the user asks for. > > \### Steps to reproduce the bug > > Here I link a colab notebook to reproduce the error: > > https://colab.research.google.com/drive/1JCrS7FlGfu\_kFqChMrwKZ\_bpabnIMqbP?authuser=1#scrollTo=FBAvlhMxIzpA > > \### Expected behavior > > Data processing does not fail. A correct example can be seen here: https://huggingface.co/datasets/tj-solergibert/Europarl-ST-processed-mt-en > > \### Environment info > > \- \`datasets\` version: 2.9.0 > \- Platform: Linux-5.10.147+-x86\_64-with-glibc2.29 > \- Python version: 3.8.10 > \- PyArrow version: 9.0.0 > \- Pandas version: 1.3.5