# MLM vs CLM, can be exchanged?

**URL:** <https://discuss.huggingface.co/t/mlm-vs-clm-can-be-exchanged/21886>\
**Category:** Models\
**Created:** [August 21, 2022, 6:02pm UTC](https://discuss.huggingface.co/t/mlm-vs-clm-can-be-exchanged/21886 "2022-08-21T18:02:18Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![mdelas](https://avatars.discourse-cdn.com/v4/letter/m/b5ac83/32.png) [@mdelas](https://discuss.huggingface.co/u/mdelas)\
**Post date:** [August 21, 2022, 6:02pm UTC](https://discuss.huggingface.co/t/mlm-vs-clm-can-be-exchanged/21886/1 "2022-08-21T18:02:18Z")

</div>

Hello everyone, I have been building a BERT model with a Masked Language Model approach as my pretraining goal. All of a sudden I need to, instead of using a Masked Language Model, need the outputs of a Causal Language Model (predict the n+1 token of the sequence).

I was thinking if it would be correct to use this model and just the last token in the preprocessing and re-train my model. Do you think that’s a good approach? Or I should build again a model specific to an autoregressive behaviour…

Any references?

Thank you very much