# Retrain Electra model with different embedings from scratch

**URL:** <https://discuss.huggingface.co/t/retrain-electra-model-with-different-embedings-from-scratch/16372>\
**Category:** Beginners\
**Created:** [April 1, 2022, 2:30pm UTC](https://discuss.huggingface.co/t/retrain-electra-model-with-different-embedings-from-scratch/16372 "2022-04-01T14:30:49Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![perry123](https://avatars.discourse-cdn.com/v4/letter/p/f9ae1b/32.png) [@perry123](https://discuss.huggingface.co/u/perry123)\
**Post date:** [April 1, 2022, 2:30pm UTC](https://discuss.huggingface.co/t/retrain-electra-model-with-different-embedings-from-scratch/16372/1 "2022-04-01T14:30:49Z")

</div>

Hello all,

I have implemented Electra model with a different (ELMO-based) embedings. I have based my code on BERT version from [GitHub - helboukkouri/character-bert: Main repository for "CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters"](https://github.com/helboukkouri/character-bert). However, I have an issue with training. If I train pure Electra from scratch, all is working correctly and loss is decreasing.  
However, when I try to train the new version, loss is stucked and oscilate (the same happend also with original BERT). I have tried to train ELMO separatelly with a different network and it seems to be working as it should.  
Any idea how to debug this, or how to find what is wrong?

Thank you all
