# Padding not working when loading a tokenizer trained via the tokenizers library into transformers

**URL:** <https://discuss.huggingface.co/t/padding-not-working-when-loading-a-tokenizer-trained-via-the-tokenizers-library-into-transformers/42326>\
**Category:** 🤗Transformers\
**Created:** [June 6, 2023, 11:19pm UTC](https://discuss.huggingface.co/t/padding-not-working-when-loading-a-tokenizer-trained-via-the-tokenizers-library-into-transformers/42326 "2023-06-06T23:19:39Z")\
**Posts on this page:** 1\
**Showing post:** 1

<div class="post-metadata">

**Author:** ![abhinavp](https://avatars.discourse-cdn.com/v4/letter/a/e19adc/32.png) [@abhinavp](https://discuss.huggingface.co/u/abhinavp)\
**Post date:** [June 6, 2023, 11:19pm UTC](https://discuss.huggingface.co/t/padding-not-working-when-loading-a-tokenizer-trained-via-the-tokenizers-library-into-transformers/42326/1 "2023-06-06T23:19:39Z")

</div>

Hi,

I trained a simple WhitespaceSplit/WordLevel tokenizer using the `tokenizers` library. I added padding by calling `enable_padding(pad_token="<pad>")` on the Tokenizer instance. Then I saved it to a JSON file and then loaded it into transformers using [the instructions here](https://huggingface.co/docs/transformers/fast_tokenizers):

```auto
fast_tokenizer = PreTrainedTokenizerFast(tokenizer_file="tokenizer.json")

```

When using the `tokenizers.Tokenizer` object directly, `encode` correctly adds the padding tokens. However, if I try padding when tokenizing using the `PreTrainedTokenizerFast` instance, I get the exception:

```auto
ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.

```

Sure enough, if I follow the instructions and add the pad token as a special token, it works. But I want the tokenizer to work _out of the box_, exactly as the equivalent `tokenizer.Tokenizer` instance does, including in terms of padding behavior.

Why is this not the case? Why do I have to enable padding for the `tokenizer.Tokenizer` instance, and then _again_ for the `PreTrainedTokenizerFast` instance? Am I doing something wrong or missing something?

---

_[View the full topic](https://discuss.huggingface.co/t/padding-not-working-when-loading-a-tokenizer-trained-via-the-tokenizers-library-into-transformers/42326)._
