# GPT2Tokenizer not working in Kaggle Notebook

**URL:** <https://discuss.huggingface.co/t/gpt2tokenizer-not-working-in-kaggle-notebook/41508>\
**Category:** 🤗Tokenizers\
**Created:** [May 30, 2023, 2:58pm UTC](https://discuss.huggingface.co/t/gpt2tokenizer-not-working-in-kaggle-notebook/41508 "2023-05-30T14:58:06Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![admarcosai](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@admarcosai](https://discuss.huggingface.co/u/admarcosai)\
**Post date:** [May 30, 2023, 2:58pm UTC](https://discuss.huggingface.co/t/gpt2tokenizer-not-working-in-kaggle-notebook/41508/1 "2023-05-30T14:58:06Z")

</div>

Hello I have been trying to tokenize the WMT14 en-de. I first started in a free google colab and the following code worked there; I then switched to Kaggle notebook since it is a better environment but it doesn’t work there for some reason. The error it throws is below:

```auto
from transformers import GPT2Tokenizer
bpe_tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

def tokenization(examples):
    source, target = [], []
    for example in examples:
        trgt = bpe_tokenizer(example['de'])
        src = bpe_tokenizer(example['en'])
        target.append(trgt)
        source.append(src)
        
    return {'de': target,
            'en': source}

train_dataset = dataset_de_en['train'].map(lambda examples: tokenization(examples['translation']), batched=True)
test_dataset = dataset_de_en['test'].map(lambda examples: tokenization(examples['translation']), batched=True)
val_dataset = dataset_de_en['val'].map(lambda examples: tokenization(examples['translation']), batched=True)

```

```auto
ArrowInvalid: Could not convert {'input_ids': [54, 798, 263, 559, 22184, 993, 1326, 4587, 311, 4224, 2150, 82, 525, 72, 1098], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]} with type BatchEncoding: did not recognize Python value type when inferring an Arrow data type

```

Is there an explanation to this?
