# Train a large transformer with Custom Tokenizer/Data

**URL:** <https://discuss.huggingface.co/t/train-a-large-transformer-with-custom-tokenizer-data/28350>\
**Category:** 🤗Transformers\
**Created:** [December 23, 2022, 9:44pm UTC](https://discuss.huggingface.co/t/train-a-large-transformer-with-custom-tokenizer-data/28350 "2022-12-23T21:44:51Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![htlerdo](https://avatars.discourse-cdn.com/v4/letter/h/e9c0ed/32.png) [@htlerdo](https://discuss.huggingface.co/u/htlerdo)\
**Post date:** [December 23, 2022, 9:44pm UTC](https://discuss.huggingface.co/t/train-a-large-transformer-with-custom-tokenizer-data/28350/1 "2022-12-23T21:44:51Z")

</div>

I’d like to train large transformer model with my own data and tokenizer. I have billions of examples, so I’d like to avoid using anything that’s pre-trained, I just want to train from scratch. Ideally I’d like to use tf/keras (hoping to upload to Google Cloud and train with TPUs there since I already have TPUs).

What’s the easiest way to do this?

I can think of several options:

1. I dump a dataset where each document is just sequence of integers and the model doesn’t care about tokenization etc… It just learns to predict the next integer given a list of integers. Is there any TF code where I can do this easily? (I expect I will have a few thousand documents where each document have ~million integers/tokens).

2. I implement my tokenizer and the model allows me to plug-in a custom tokenizer. Is there any TF code where I can do this easily?