# TypeError: Couldn't cast array of type int64 to null

**URL:** <https://discuss.huggingface.co/t/typeerror-couldnt-cast-array-of-type-int64-to-null/138935>\
**Category:** 🤗Datasets\
**Created:** [February 4, 2025, 12:47pm UTC](https://discuss.huggingface.co/t/typeerror-couldnt-cast-array-of-type-int64-to-null/138935 "2025-02-04T12:47:16Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![c-bone](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/c-bone/32/38706_2.png) [@c-bone](https://discuss.huggingface.co/u/c-bone)\
**Post date:** [February 4, 2025, 12:47pm UTC](https://discuss.huggingface.co/t/typeerror-couldnt-cast-array-of-type-int64-to-null/138935/1 "2025-02-04T12:47:16Z")

</div>

Hello everyone!

I am currently training a GPT-2 model from scratch using my own [cif-tokenizer](https://huggingface.co/c-bone/cif-tokenizer). The goal is to be able to generate crystallographic information files using an LLM. Since some of the CIFs have more tokens than context length, I am using strided tokenization with returned overflowing tokens and padded to max context length. This method has worked for all my datasets (different materials but same format) - however, using the [MP-20](https://huggingface.co/datasets/c-bone/MP-20_clean) dataset I am getting a ‘TypeError: Couldn’t cast array of type int64 to null’ when performing the dataset.map( function.

Data type of first example processed in train set:

```auto
Split: train                                                                                                                         
Number of examples: 27136                                                                                                            
Column names: ['Database', 'Material ID', 'Reduced Formula', 'CIF']                                                                  
Features: {'Database': Value(dtype='string', id=None), 'Material ID': Value(dtype='string', id=None), 'Reduced Formula': Value(dtype=
'string', id=None), 'CIF': Value(dtype='string', id=None)}                                                                           
                                                                                                                                     
First example:                                                                                                                       
{'Database': 'MP-20', 'Material ID': 'mp-1221227', 'Reduced Formula': 'Na3MnCoNiO6', 'CIF': "data_Na6Mn2Co2Ni2O12\nloop_\n _atom_type
_symbol\n _atom_type_electronegativity\n _atom_type_radius\n _atom_type_ionic_radius\n Na 0.9300 1.8000 1.1600\n [...] \n"}

```

Dataloading script:

```auto
def tokenize_function(examples, tokenizer, context_length, stride):
    '''
    Tokenize a dataset using a sliding window approach.
    Processes the "CIF" column in a batch of examples.
    '''
    # Add BOS and EOS tokens to each CIF sequence
    bos_token = tokenizer.bos_token
    eos_token = tokenizer.eos_token

    # Tokenize the CIF column with BOS and EOS tokens
    return tokenizer(
        [bos_token + example + eos_token for example in examples["CIF"]],
        truncation=True,
        max_length=context_length,
        padding="max_length",
        stride=stride,
        return_overflowing_tokens=True,
        return_special_tokens_mask=True,
        return_offsets_mapping=False
    )

def load_data(tokenizer, dataset, context_length, stride, dataset_streaming=False):
    

    # Wrap the tokenize function to pass the tokenizer
    def tokenize_wrapper(examples):
        return tokenize_function(examples, tokenizer, context_length, stride)
    
    # Apply tokenization to the dataset
    tokenized_dataset = dataset.map(
        tokenize_wrapper, # Pass the wrapped tokenize function
        batched=True, # Apply to batches of examples rather than indiv
        remove_columns=["CIF"], # Remove the original "CIF" column, bc model only needs tokenized data
        # Only use num_proc if not streaming 
        **({"num_proc": 8} if not dataset_streaming else {})
    )
    
    # Create a data collator for training
    data_collator = DataCollatorForLanguageModeling(
        tokenizer=tokenizer,
        mlm=False # No masked language modeling
    )

    return tokenized_dataset, data_collator

```

Full error:

```auto
Conditionality is not activated                                                                                                                     
Map (num_proc=8): 91%|██████████████████████████████████████████████████████████████████████████████████████▋ | 24744/27136 [00:04<00:00, 52
00.22 examples/s]                                                                                                                                   
multiprocess.pool.RemoteTraceback:                                                                                                                  
"""                                                                                                                                                 
Traceback (most recent call last):                                                                                                                  
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/multiprocess/pool.py", line 125, in worker                      
    result = (True, func(*args, **kwds))                                                                                                            
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 678, in _write_generator_to_qu
eue                                                                                                                                                 
    for i, result in enumerate(func(**kwargs)):                                                                                                     
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3499, in _map_single           
    writer.write_batch(batch)                                                                                                                       
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/datasets/arrow_writer.py", line 605, in write_batch             
    arrays.append(pa.array(typed_sequence))                                                                                                         
  File "pyarrow/array.pxi", line 250, in pyarrow.lib.array                                                                                          
  File "pyarrow/array.pxi", line 114, in pyarrow.lib._handle_arrow_array_protocol                                                                   
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/datasets/arrow_writer.py", line 243, in __arrow_array__         
    out = cast_array_to_feature(                                                                                                                    
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/datasets/table.py", line 1797, in wrapper                       
    return func(array, *args, **kwargs) " 12:44 04-Feb-25
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/datasets/table.py", line 2065, in cast_array_to_feature
    casted_array_values = _c(array.values, feature.feature)                                                                                         
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/datasets/table.py", line 1797, in wrapper
    return func(array, *args, **kwargs)                                   
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/datasets/table.py", line 2102, in cast_array_to_feature
    return array_cast(                                                    
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/datasets/table.py", line 1797, in wrapper
    return func(array, *args, **kwargs)                                   
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/datasets/table.py", line 1948, in array_cast
    raise TypeError(f"Couldn't cast array of type {_short_str(array.type)} to {_short_str(pa_type)}")
TypeError: Couldn't cast array of type int64 to null   

The above exception was the direct cause of the following exception:                                                                                

Traceback (most recent call last):                                        
  File "/home/cyprien/CrystaLLMv2_CG/_train.py", line 194, in <module>                                                                              
    main()                                                                
  File "/home/cyprien/CrystaLLMv2_CG/_train.py", line 96, in main                                                                                   
    tokenized_dataset, data_collator = load_data(                                                                                                   
  File "/home/cyprien/CrystaLLMv2_CG/_dataloader.py", line 106, in load_data                                                                        
    tokenized_dataset = dataset.map(                                      
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/datasets/dataset_dict.py", line 886, in map
    {                                                                     
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/datasets/dataset_dict.py", line 887, in <dictcomp>
    k: dataset.map(                                                       
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 560, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)                                                                              
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3165, in map
    for rank, done, content in iflatmap_unordered(                                                                                                  
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 718, in iflatmap_unordered
    [async_result.get(timeout=0.05) for async_result in async_results]                                                                              
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 718, in <listcomp>
    [async_result.get(timeout=0.05) for async_result in async_results]                                                                              
  File "/home/cyprien/miniconda3/envs/crystallmv2_venv/lib/python3.10/site-packages/multiprocess/pool.py", line 774, in get
    raise self._value                                                     
TypeError: Couldn't cast array of type int64 to null

```

If I adjust the context length to be \>max\_token\_length, I do not get an error. I have also tried specifying explicitly the features as per some of the posts on this forum, but I am unsure if I am doing it correctly.  
I was wondering if anyone knows how I could tackle this problem? I am still quite new to HuggingFace so struggling to grasp what has gone wrong for this dataset. Thank you in advance for the help.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 5, 2025, 11:53am UTC](https://discuss.huggingface.co/t/typeerror-couldnt-cast-array-of-type-int64-to-null/138935/2 "2025-02-05T11:53:41Z")

</div>

I found a similar case, but it’s different…

> <https://stackoverflow.com/questions/79012832/multiprocess-pool-remotetraceback-and-typeerror-couldnt-cast-array-of-type-str>

> [@TypeError: Couldn't cast array of type int64 while mapping the dataset](https://discuss.huggingface.co/t/typeerror-couldnt-cast-array-of-type-int64-while-mapping-the-dataset/22167):
>
> Hai, I am trying to finetune LayoutLMV2 model for document image classification but in preprocessing stage I am getting error. Can some one please help me in fixing the error. dataset = Dataset.from\_pandas(df\_copy) dataset Dataset({ features: ['Image\_File\_Path', 'Label'], num\_rows: 12 }) this is the code # we need to define custom features path = './Image FIle/' features = Features({ 'image': Array3D(dtype="int64", shape=(3, 224, 224)), 'input\_ids': Sequence(feature=Value(d…

---

<div class="post-metadata">

**Author:** ![c-bone](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/c-bone/32/38706_2.png) [@c-bone](https://discuss.huggingface.co/u/c-bone)\
**Post date:** [February 6, 2025, 9:49am UTC](https://discuss.huggingface.co/t/typeerror-couldnt-cast-array-of-type-int64-to-null/138935/3 "2025-02-06T09:49:19Z")

</div>

Hi, thanks for the help. I did see these before posting, but in my case all the entries strings so I don’t have these different datatypes. the only places I would have different datatypes would be in the return overflowing token mappings which I removed, at least to my understanding. So I’m not sure how I could deal with this.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 6, 2025, 1:52pm UTC](https://discuss.huggingface.co/t/typeerror-couldnt-cast-array-of-type-int64-to-null/138935/4 "2025-02-06T13:52:37Z")

</div>

To begin with, Python’s language specification includes **None** but not **null** , so I think that **null** probably comes from pandas or pyarrow.  
In pandas, for example, just having **“”** creates **null**. I think that such data is causing problems when implicit conversion is performed inside the datasets library. On the other hand, **int64** is probably data that has been successfully tokenized.

> <https://github.com/huggingface/datasets/issues/5965>
>
> \### Describe the bug
> 
> When doing a map of a dataset with complex types, sometime…s \`datasets\` is unable to interpret the valid schema of a returned datasets.map() function. This often comes from conflicting types, like when both empty lists and filled lists are competing for the same field value.
> 
> This is prone to happen in batch mapping, when the mapper returns a sequence of null/empty values and other batches are non-null. A workaround is to manually cast the new batch to a pyarrow table (like implemented in this \[workaround\](https://github.com/piercefreeman/lassen/pull/3)) but it feels like this ideally should be solved at the core library level.
> 
> Note that the reproduction case only throws this error if the first datapoint has the empty list. If it is processed later, datasets already detects its representation as list-type and therefore allows the empty list to be provided.
> 
> \### Steps to reproduce the bug
> 
> A trivial reproduction case:
> 
> \`\`\`python
> from typing import Iterator, Any
> import pandas as pd
> from datasets import Dataset
> 
> def batch\_to\_examples(batch: dict\[str, list\[Any\]\]) -\> Iterator\[dict\[str, Any\]\]:
> for i in range(next(iter(lengths))):
> yield {feature: values\[i\] for feature, values in batch.items()}
> 
> def examples\_to\_batch(examples) -\> dict\[str, list\[Any\]\]:
> batch = {}
> 
> for example in examples:
> for feature, value in example.items():
> if feature not in batch:
> batch\[feature\] = \[\]
> batch\[feature\].append(value)
> 
> return batch
> 
> def batch\_process(examples, explicit\_schema: bool):
> new\_examples = \[\]
> for example in batch\_to\_examples(examples):
> new\_examples.append(dict(texts=example\["raw\_text"\].split()))
> return examples\_to\_batch(new\_examples)
> 
> df = pd.DataFrame(
> \[
> {"raw\_text": ""},
> {"raw\_text": "This is a test"},
> {"raw\_text": "This is another test"},
> \]
> )
> 
> dataset = Dataset.from\_pandas(df)
> 
> \# datasets won't be able to typehint a dataset that starts with an empty example.
> with pytest.raises(TypeError, match="Couldn't cast array of type"):
> dataset = dataset.map(
> batch\_process,
> batched=True,
> batch\_size=1,
> num\_proc=1,
> remove\_columns=dataset.column\_names,
> )
> \`\`\`
> 
> This results in crashes like:
> 
> \`\`\`bash
> File "/Users/piercefreeman/Library/Caches/pypoetry/virtualenvs/example-9kBqeSPy-py3.11/lib/python3.11/site-packages/datasets/table.py", line 1819, in wrapper
> return func(array, \*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/Users/piercefreeman/Library/Caches/pypoetry/virtualenvs/example-9kBqeSPy-py3.11/lib/python3.11/site-packages/datasets/table.py", line 2109, in cast\_array\_to\_feature
> return array\_cast(array, feature(), allow\_number\_to\_str=allow\_number\_to\_str)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/Users/piercefreeman/Library/Caches/pypoetry/virtualenvs/example-9kBqeSPy-py3.11/lib/python3.11/site-packages/datasets/table.py", line 1819, in wrapper
> return func(array, \*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/Users/piercefreeman/Library/Caches/pypoetry/virtualenvs/example-9kBqeSPy-py3.11/lib/python3.11/site-packages/datasets/table.py", line 1998, in array\_cast
> raise TypeError(f"Couldn't cast array of type {array.type} to {pa\_type}")
> TypeError: Couldn't cast array of type string to null
> \`\`\`
> 
> \### Expected behavior
> 
> The code should successfully map and create a new dataset without error.
> 
> \### Environment info
> 
> Mac OSX, Linux