# T5 tokenizer vs t51.1 tokenizer

**URL:** <https://discuss.huggingface.co/t/t5-tokenizer-vs-t51-1-tokenizer/75421>\
**Category:** 🤗Tokenizers\
**Created:** [March 1, 2024, 2:35am UTC](https://discuss.huggingface.co/t/t5-tokenizer-vs-t51-1-tokenizer/75421 "2024-03-01T02:35:55Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nevermetyou](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nevermetyou/32/21618_2.png) [@Nevermetyou](https://discuss.huggingface.co/u/Nevermetyou)\
**Post date:** [March 1, 2024, 2:35am UTC](https://discuss.huggingface.co/t/t5-tokenizer-vs-t51-1-tokenizer/75421/1 "2024-03-01T02:35:55Z")

</div>

Hello, there

I wonder if the tokenizer of t51.1 is the same as that of normal t5.  
 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/0/c/0c9790f4600c4434b458f1dbb6cdd87ea049f070.png)

Or is the pre-tokenization step not the same?
