# Writing custom tokenizer and wrapping it in tokenizer object

**URL:** <https://discuss.huggingface.co/t/writing-custom-tokenizer-and-wrapping-it-in-tokenizer-object/39679>\
**Category:** 🤗Tokenizers\
**Created:** [May 12, 2023, 5:53pm UTC](https://discuss.huggingface.co/t/writing-custom-tokenizer-and-wrapping-it-in-tokenizer-object/39679 "2023-05-12T17:53:55Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![arnab9learns](https://avatars.discourse-cdn.com/v4/letter/a/57b2e6/32.png) [@arnab9learns](https://discuss.huggingface.co/u/arnab9learns)\
**Post date:** [May 12, 2023, 5:53pm UTC](https://discuss.huggingface.co/t/writing-custom-tokenizer-and-wrapping-it-in-tokenizer-object/39679/1 "2023-05-12T17:53:55Z")

</div>

how to write my own tokenizer and wrap it in huggingface tokenizer object?  
we will follow this procedure instead of using any pretokenizer and get a tokenizer object like if we use one pretrained tokenizer?

> **[Byte-Pair Encoding tokenization - Hugging Face NLP Course](https://huggingface.co/learn/nlp-course/chapter6/5?fw=pt)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![maveriq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/maveriq/32/27075_2.png) [@maveriq](https://discuss.huggingface.co/u/maveriq)\
**Post date:** [June 25, 2023, 8:25am UTC](https://discuss.huggingface.co/t/writing-custom-tokenizer-and-wrapping-it-in-tokenizer-object/39679/2 "2023-06-25T08:25:15Z")

</div>

Did you find a solution?

---

<div class="post-metadata">

**Author:** ![arnab9learns](https://avatars.discourse-cdn.com/v4/letter/a/57b2e6/32.png) [@arnab9learns](https://discuss.huggingface.co/u/arnab9learns)\
**Post date:** [June 26, 2023, 12:16am UTC](https://discuss.huggingface.co/t/writing-custom-tokenizer-and-wrapping-it-in-tokenizer-object/39679/3 "2023-06-26T00:16:21Z")

</div>

No, Nobody seems to be interested in it. and I am kind of disappointed. let me know if you find something. I am still stuck on this issue.
