# Does tokenizer changed during model training

**URL:** <https://discuss.huggingface.co/t/does-tokenizer-changed-during-model-training/19213>\
**Category:** Beginners\
**Created:** [June 15, 2022, 7:01pm UTC](https://discuss.huggingface.co/t/does-tokenizer-changed-during-model-training/19213 "2022-06-15T19:01:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Talha](https://avatars.discourse-cdn.com/v4/letter/t/a5b964/32.png) [@Talha](https://discuss.huggingface.co/u/Talha)\
**Post date:** [June 15, 2022, 7:01pm UTC](https://discuss.huggingface.co/t/does-tokenizer-changed-during-model-training/19213/1 "2022-06-15T19:01:15Z")

</div>

I want to know during model training, does tokenizer alter it self, (i mean does it train) or it remain the same as original one.  
For example i used the  
`AutoTokenizer.from_pretrained(model_name,local_files_only=False)`  
after model training, is there any change to tokenizer?

---

<div class="post-metadata">

**Author:** ![Slinae](https://avatars.discourse-cdn.com/v4/letter/s/a183cd/32.png) [@Slinae](https://discuss.huggingface.co/u/Slinae)\
**Post date:** [June 16, 2022, 1:44am UTC](https://discuss.huggingface.co/t/does-tokenizer-changed-during-model-training/19213/2 "2022-06-16T01:44:22Z")

</div>

Hi, Talha,

Tokenizer id does not change. For example, the token\_id of `[['sep']]` is 102. It will be 102 through the training. But the embedding hidden state will be changed.

---

<div class="post-metadata">

**Author:** ![ShieldHero](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shieldhero/32/4451_2.png) [@ShieldHero](https://discuss.huggingface.co/u/ShieldHero)\
**Post date:** [August 11, 2022, 1:47pm UTC](https://discuss.huggingface.co/t/does-tokenizer-changed-during-model-training/19213/3 "2022-08-11T13:47:47Z")

</div>

One doubt. So lets say I have trained two models of “bert-base-uncased” to do different classification tasks(same number of classes).  
So, now, can I use the tokenizer of one model1 to tokenize the other one? Will the result be different compared to using two different tokenizers?
