# How is CLS special token embedding initialized?

**URL:** <https://discuss.huggingface.co/t/how-is-cls-special-token-embedding-initialized/15768>\
**Category:** Intermediate\
**Created:** [March 16, 2022, 11:07am UTC](https://discuss.huggingface.co/t/how-is-cls-special-token-embedding-initialized/15768 "2022-03-16T11:07:06Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![sgavela](https://avatars.discourse-cdn.com/v4/letter/s/e9a140/32.png) [@sgavela](https://discuss.huggingface.co/u/sgavela)\
**Post date:** [March 16, 2022, 11:07am UTC](https://discuss.huggingface.co/t/how-is-cls-special-token-embedding-initialized/15768/1 "2022-03-16T11:07:06Z")

</div>

Hi! Does anyone know how CLS token is initialized in BERT? I mean, let’s say I would like to train a BERT model from scratch (which of course I’m not doing), how should I initialize CLS embedding? Just at random under some distribution such as uniform? How is this done in BERT?

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [March 16, 2022, 3:46pm UTC](https://discuss.huggingface.co/t/how-is-cls-special-token-embedding-initialized/15768/2 "2022-03-16T15:46:26Z")

</div>

Just like other tokens, the CLS token is randomly initialized from a normal distribution. The only exception is the padding token, which is set to zero.

> <https://github.com/huggingface/transformers/blob/204c54d411c2b4c7f31405203533a51632f46ab1/src/transformers/models/bert/modeling_bert.py#L731-L734>