# Image captioning decoder

**URL:** <https://discuss.huggingface.co/t/image-captioning-decoder/13366>\
**Category:** Languages at Hugging Face\
**Created:** [January 4, 2022, 1:12pm UTC](https://discuss.huggingface.co/t/image-captioning-decoder/13366 "2022-01-04T13:12:29Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![toyl](https://avatars.discourse-cdn.com/v4/letter/t/73ab20/32.png) [@toyl](https://discuss.huggingface.co/u/toyl)\
**Post date:** [January 4, 2022, 1:12pm UTC](https://discuss.huggingface.co/t/image-captioning-decoder/13366/1 "2022-01-04T13:12:29Z")

</div>

excuse me does the decoder of the language model deal with words or sentences to do the captioning . I mean, if i use BERT, will i use it for word embedding or sentence embedding … Thanks in advance

---

<div class="post-metadata">

**Author:** ![merve](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/merve/32/49809_2.png) [@merve](https://discuss.huggingface.co/u/merve)\
**Post date:** [January 4, 2022, 6:20pm UTC](https://discuss.huggingface.co/t/image-captioning-decoder/13366/2 "2022-01-04T18:20:58Z")

</div>

Hello,  
AFAIK you can use a GPT-2 for the decoder. Here’s an [example](https://huggingface.co/flax-community/vit-gpt2) with ViT + GPT-2.

---

<div class="post-metadata">

**Author:** ![toyl](https://avatars.discourse-cdn.com/v4/letter/t/73ab20/32.png) [@toyl](https://discuss.huggingface.co/u/toyl)\
**Post date:** [January 4, 2022, 6:30pm UTC](https://discuss.huggingface.co/t/image-captioning-decoder/13366/3 "2022-01-04T18:30:41Z")

</div>

thanks for the link but I need to use BERT and I couldn’t find it in the link

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [January 5, 2022, 9:04am UTC](https://discuss.huggingface.co/t/image-captioning-decoder/13366/4 "2022-01-05T09:04:37Z")

</div>

You can initialize the weights of the decoder with the weights of any encoder-only model too, like BERT. This is because a decoder is also just a stack of blocks (self-attention + feedforward neural networks), similar to an encoder. The only difference is that a decoder also adds cross-attention layers. Hence, if you initialize the weights of a decoder with the weights of an encoder-only model, the weights of the cross-attention layers will be randomly initialized, and need to be fine-tuned on a downstream task (like summarization, machine translation, image captioning).

Also, to answer your question: models like BERT/RoBERTa/GPT-x all operate on subword tokens, rather than words. This means that a word like “cookie” might be tokenized into “coo” and “kie”. These models learn an embedding vector for each individual subword token.

---

<div class="post-metadata">

**Author:** ![toyl](https://avatars.discourse-cdn.com/v4/letter/t/73ab20/32.png) [@toyl](https://discuss.huggingface.co/u/toyl)\
**Post date:** [January 6, 2022, 4:53pm UTC](https://discuss.huggingface.co/t/image-captioning-decoder/13366/5 "2022-01-06T16:53:30Z")

</div>

thanks a lot for replying . so excuse me if extracted the features from images using CNN , can I extract weights of captions using BERT and apply them to the weights of the embedding layer and then use LSTM for example ?  
and will I need to extract features for words using BERT from all layers or I need to ignore the classification layer?
