# Accessing uncontextualized BERT word embeddings

**URL:** <https://discuss.huggingface.co/t/accessing-uncontextualized-bert-word-embeddings/1812>\
**Category:** Beginners\
**Created:** [October 29, 2020, 10:15pm UTC](https://discuss.huggingface.co/t/accessing-uncontextualized-bert-word-embeddings/1812 "2020-10-29T22:15:53Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jdyearsley](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jdyearsley/32/535_2.png) [@jdyearsley](https://discuss.huggingface.co/u/jdyearsley)\
**Post date:** [October 29, 2020, 10:15pm UTC](https://discuss.huggingface.co/t/accessing-uncontextualized-bert-word-embeddings/1812/1 "2020-10-29T22:15:53Z")

</div>

Hi there! Once I’ve imported a BERT model from HuggingFace, is there a way to convert a sequence of encoded tokens into BERT’s raw embeddings without contextualizing them using self-attention, or otherwise extract the raw embedding for a given token?

---

<div class="post-metadata">

**Author:** ![talolard](https://avatars.discourse-cdn.com/v4/letter/t/258eb7/32.png) [@talolard](https://discuss.huggingface.co/u/talolard)\
**Post date:** [October 30, 2020, 12:03pm UTC](https://discuss.huggingface.co/t/accessing-uncontextualized-bert-word-embeddings/1812/2 "2020-10-30T12:03:46Z")

</div>

Try this.  
I think the apis change a bit between models so take a look before you copy paste 🙂

```auto
            model = DistilBertForTokenClassification.from_pretrained(
                "distilbert-base-cased", num_labels=self.num_labels
            )
            word_embeddings = model.distilbert.embeddings.word_embeddings(["my token ids here"])
            word_embeddings_with_positions = model.distilbert.embeddings(["my token ids here"])

```

---

<div class="post-metadata">

**Author:** ![jdyearsley](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jdyearsley/32/535_2.png) [@jdyearsley](https://discuss.huggingface.co/u/jdyearsley)\
**Post date:** [October 30, 2020, 7:32pm UTC](https://discuss.huggingface.co/t/accessing-uncontextualized-bert-word-embeddings/1812/3 "2020-10-30T19:32:14Z")

</div>

This got me there! Thank you so much.
