# Adding tokens, but tokenizer doesn't use them **URL:** \ **Category:** 🤗Tokenizers\ **Created:** [March 25, 2024, 11:18am UTC](https://discuss.huggingface.co/t/adding-tokens-but-tokenizer-doesnt-use-them/78674 "2024-03-25T11:18:58Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![WonderYear1905](https://avatars.discourse-cdn.com/v4/letter/w/c89c15/32.png) [@WonderYear1905](https://discuss.huggingface.co/u/WonderYear1905)\ **Post date:** [March 25, 2024, 11:18am UTC](https://discuss.huggingface.co/t/adding-tokens-but-tokenizer-doesnt-use-them/78674/1 "2024-03-25T11:18:58Z")
Hi, Im trying to add tokens to a pretrained tokenizer. First I initialized the tokenizer: > tokenizer = AutoTokenizer.from\_pretrained(“mistralai/Mistral-7B-Instruct-v0.2”) Next I created a training iterator and trained a new tokenizer: > training\_corpus = get\_training\_corpus() > new\_tokenizer = tokenizer.train\_new\_from\_iterator(training\_corpus,vocab\_size=10000)#,10000) Taking the diff: > tokens\_to\_add = list(set(new\_tokenizer.vocab.keys())- set(tokenizer.vocab.keys())) > output = tokenizer.add\_tokens(tokens\_to\_add) tokenizer is updated, I can see its now in the correct size (i.e original size=32000+ added tokens), I can also see the new token under `added_tokens_decoder` and `added_tokens_encoder`. Everything seems great. But, when Im trying to tokenize my input data: > tokenizer.tokenize(x, return\_tensors=“pt”) The tokenizer just doesnt use the new tokens. Any idea what Im doing wrong?! Thanks! ---
**Author:** ![Butanium](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/butanium/32/49190_2.png) [@Butanium](https://discuss.huggingface.co/u/Butanium)\ **Post date:** [August 14, 2024, 11:57am UTC](https://discuss.huggingface.co/t/adding-tokens-but-tokenizer-doesnt-use-them/78674/2 "2024-08-14T11:57:57Z")
Just experienceed the same issue with llama2-7b, did you find a solution to that? In my case adding `normalized=False` fixed the issue I had on my single example.