# Vocabulary count mismatch when loading the previously created tokenizer

**URL:** <https://discuss.huggingface.co/t/vocabulary-count-mismatch-when-loading-the-previously-created-tokenizer/68355>\
**Category:** 🤗Transformers\
**Created:** [January 8, 2024, 10:16am UTC](https://discuss.huggingface.co/t/vocabulary-count-mismatch-when-loading-the-previously-created-tokenizer/68355 "2024-01-08T10:16:06Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![dizzyme09](https://avatars.discourse-cdn.com/v4/letter/d/3da27b/32.png) [@dizzyme09](https://discuss.huggingface.co/u/dizzyme09)\
**Post date:** [January 8, 2024, 10:16am UTC](https://discuss.huggingface.co/t/vocabulary-count-mismatch-when-loading-the-previously-created-tokenizer/68355/1 "2024-01-08T10:16:06Z")

</div>

I am currently fine-tuning the Wav2Vec2 XLS-R model. I have created a tokenizer that is stored in one of the huggingface repos. Here is how I store my tokenizer in huggingface:

```auto
from transformers import Wav2Vec2CTCTokenizer

tokenizer = Wav2Vec2CTCTokenizer.from_pretrained("./", unk_token="[UNK]", pad_token="[PAD]", word_delimiter_token="|")
tokenizer.push_to_hub(repo_name)

```

At a certain time, I want to do the same task using the tokenizer. I load the tokenizer with the following code:

```auto
tokenizer = Wav2Vec2CTCTokenizer.from_pretrained(repo_name)

```

with that code, all the files needed for the tokenizer are loaded, such as `vocab.json` , `tokenizer_config.json` , and `special_tokens_map.json` . All the vocabularies I should have are 30, according to what’s in `vocab.json` .  
But, here’s something strange. when I load the tokenizer, I get two additional tokens, which are already in my vocabulary. From the image is the tokenizer object information that I loaded on the repo.  
how did this happen? did I load the tokenizer incorrectly?

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/3/1/319510183698f238520e3e9e45815954acd042aa.png)
