# How is the encoding done for transformers? What encoder is used?

**URL:** <https://discuss.huggingface.co/t/how-is-the-encoding-done-for-transformers-what-encoder-is-used/21928>\
**Category:** 🤗Transformers\
**Created:** [August 22, 2022, 4:57pm UTC](https://discuss.huggingface.co/t/how-is-the-encoding-done-for-transformers-what-encoder-is-used/21928 "2022-08-22T16:57:23Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![cyrilw](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cyrilw/32/4311_2.png) [@cyrilw](https://discuss.huggingface.co/u/cyrilw)\
**Post date:** [August 22, 2022, 4:57pm UTC](https://discuss.huggingface.co/t/how-is-the-encoding-done-for-transformers-what-encoder-is-used/21928/1 "2022-08-22T16:57:23Z")

</div>

I’m running into the issue of the text is of a larger size than the vector embedding space supported.

- I’ve trimmed down my data using `len(text)` operator to identify the size and then trimmed it using `text[:510]`. But debugging this error, I noticed this is showing up not due to English text in the dataset, but for other languages.
- I noticed the text items in Mandarin seems to cause this and somewhere during the encoding in Transformers, it is being expanded to be more than 512.

Question;

1. How do I address this issue at a data level?  
OR
2. What is the encoder used in Transformers so I could use it at the data pre-processing level to take care of this issue and to remove items more than 512 in length?

Thank you.

This is the full error log;

```auto
  File "/mnt/DataDrive/Experiments/python_environments/lib/python3.6/site-packages/transformers/pipelines/pt_utils.py", line 111, in __next__
    item = next(self.iterator)
  File "/mnt/DataDrive/Experiments/python_environments/lib/python3.6/site-packages/transformers/pipelines/pt_utils.py", line 113, in __next__
    processed = self.infer(item, **self.params)
  File "/mnt/DataDrive/Experiments/python_environments/lib/python3.6/site-packages/transformers/pipelines/base.py", line 943, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/mnt/DataDrive/Experiments/python_environments/lib/python3.6/site-packages/transformers/pipelines/text_classification.py", line 137, in _forward
    return self.model(**model_inputs)
  File "/mnt/DataDrive/Experiments/python_environments/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/DataDrive/Experiments/python_environments/lib/python3.6/site-packages/transformers/models/bert/modeling_bert.py", line 1554, in forward
    return_dict=return_dict,
  File "/mnt/DataDrive/Experiments/python_environments/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/DataDrive/Experiments/python_environments/lib/python3.6/site-packages/transformers/models/bert/modeling_bert.py", line 994, in forward
    past_key_values_length=past_key_values_length,
  File "/mnt/DataDrive/Experiments/python_environments/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/DataDrive/Experiments/python_environments/lib/python3.6/site-packages/transformers/models/bert/modeling_bert.py", line 220, in forward
    embeddings += position_embeddings
RuntimeError: The size of tensor a (629) must match the size of tensor b (512) at non-singleton dimension 1
```

---

<div class="post-metadata">

**Author:** ![rwheel](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rwheel/32/5603_2.png) [@rwheel](https://discuss.huggingface.co/u/rwheel)\
**Post date:** [August 23, 2022, 5:50am UTC](https://discuss.huggingface.co/t/how-is-the-encoding-done-for-transformers-what-encoder-is-used/21928/2 "2022-08-23T05:50:52Z")

</div>

Hi @cyrilw,

What model do you use to tokenize your data? Can you copy the code to replicate the error?

According to the documentation ([Handling multiple sequences - Hugging Face Course](https://huggingface.co/course/chapter2/5?fw=pt#longer-sequences)), when the length of sequence is greater than the limit of the transformer model, a solution is to truncate your sentences, as you said.

I don’t know what is your task but, here is an example to train a causal language model ([Training a causal language model from scratch - Hugging Face Course](https://huggingface.co/course/chapter7/6?fw=pt#preparing-the-dataset)) and in the section _Preparing the dataset_ addresses the problem of working with large contexts.

---

<div class="post-metadata">

**Author:** ![cyrilw](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cyrilw/32/4311_2.png) [@cyrilw](https://discuss.huggingface.co/u/cyrilw)\
**Post date:** [August 23, 2022, 2:13pm UTC](https://discuss.huggingface.co/t/how-is-the-encoding-done-for-transformers-what-encoder-is-used/21928/3 "2022-08-23T14:13:08Z")

</div>

Hi Ramón @rwheel,

So I am trying to run a classification task using [unitary/toxic-bert · Hugging Face](https://huggingface.co/unitary/toxic-bert). This is a script of how it is currently preprocessed and runs through the prediction. [toxic\_bert classification · GitHub](https://gist.github.com/cyriltw/4e99ba8f8ce13348416e4125ffb601c7)

I didn’t do any tokenization since the examples in the toxic bert had just sending the text in to the models. I will look at those resources as well, thanks

---

<div class="post-metadata">

**Author:** ![cyrilw](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cyrilw/32/4311_2.png) [@cyrilw](https://discuss.huggingface.co/u/cyrilw)\
**Post date:** [August 24, 2022, 2:54am UTC](https://discuss.huggingface.co/t/how-is-the-encoding-done-for-transformers-what-encoder-is-used/21928/4 "2022-08-24T02:54:53Z")

</div>

@rwheel so I looked at the documentation and seems like truncate does have it’s issues because the prediction/classification still breaks even when it goes through checks for truncation. Do you have any idea on the encoding scheme or how an input is converted for the vector form?

---

<div class="post-metadata">

**Author:** ![rwheel](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rwheel/32/5603_2.png) [@rwheel](https://discuss.huggingface.co/u/rwheel)\
**Post date:** [August 25, 2022, 5:26am UTC](https://discuss.huggingface.co/t/how-is-the-encoding-done-for-transformers-what-encoder-is-used/21928/5 "2022-08-25T05:26:19Z")

</div>

I’ve just found a thread that can be of your interest [https://discuss.huggingface.co/t/transformer-architecture-and-theory/14558/2](https://discuss.huggingface.co/t/transformer-architecture-and-theory/14558/2).

On the other hand, it comes to my mind two additional resources: the paper of _Attention is all you need_ and the book _Natural Language Processing with Transformers_, in which you can find good diagrams that explain the encoder. There is a repository on github of that book. Although all chapters are not released, you can see the images and some code ([https://github.com/nlp-with-transformers/notebooks/blob/main/03\_transformer-anatomy.ipynb](https://github.com/nlp-with-transformers/notebooks/blob/main/03_transformer-anatomy.ipynb)).