# How to evaluate all model ckpts when using \`run\_language\_modeling\` with trainer?

**URL:** <https://discuss.huggingface.co/t/how-to-evaluate-all-model-ckpts-when-using-run-language-modeling-with-trainer/793>\
**Category:** 🤗Transformers\
**Created:** [August 20, 2020, 5:38pm UTC](https://discuss.huggingface.co/t/how-to-evaluate-all-model-ckpts-when-using-run-language-modeling-with-trainer/793 "2020-08-20T17:38:28Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![mralexis](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@mralexis](https://discuss.huggingface.co/u/mralexis)\
**Post date:** [August 20, 2020, 5:38pm UTC](https://discuss.huggingface.co/t/how-to-evaluate-all-model-ckpts-when-using-run-language-modeling-with-trainer/793/1 "2020-08-20T17:38:28Z")

</div>

> <https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py#L272>

Here it seems the code only evaluates the last ckpt, while the common practice is to evaluate all ckpts and select the best one. How could we achieve that with `trainer`? Thanks!

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [August 20, 2020, 9:06pm UTC](https://discuss.huggingface.co/t/how-to-evaluate-all-model-ckpts-when-using-run-language-modeling-with-trainer/793/2 "2020-08-20T21:06:25Z")

</div>

This code evaluates the model you pass to it. I think you mean that the train method does not save the best checkpoint at the root at the end of training, only the last checkpoint.

---

<div class="post-metadata">

**Author:** ![mralexis](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@mralexis](https://discuss.huggingface.co/u/mralexis)\
**Post date:** [August 20, 2020, 9:15pm UTC](https://discuss.huggingface.co/t/how-to-evaluate-all-model-ckpts-when-using-run-language-modeling-with-trainer/793/3 "2020-08-20T21:15:54Z")

</div>

@sgugger yes, sort of. Ideally it should pick the best model from all ckpts and save it at the root. To do so I have to modify the code to keep track of the best model, right?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [August 21, 2020, 1:06pm UTC](https://discuss.huggingface.co/t/how-to-evaluate-all-model-ckpts-when-using-run-language-modeling-with-trainer/793/4 "2020-08-21T13:06:06Z")

</div>

Yes, there is nothing for that in `Trainer` yet, so you should change the code to check at each eval if the model is better or not, This is something I have on my TODO to add at some point.

---

<div class="post-metadata">

**Author:** ![mralexis](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@mralexis](https://discuss.huggingface.co/u/mralexis)\
**Post date:** [August 21, 2020, 4:47pm UTC](https://discuss.huggingface.co/t/how-to-evaluate-all-model-ckpts-when-using-run-language-modeling-with-trainer/793/5 "2020-08-21T16:47:34Z")

</div>

But there seems to be no callback function at each `save_step`. For example, the correct pipeline should be sth like

```auto
best_perf = 0 # assume larger is better
when current_step % save_step == 0:
    current_perf = check_model_perf()
    if current_perf > best_perf:
         best_perf = current_perf
         save_model() # will also delete any previous best model

```

But now I have to save all dumps first, and the check performance in eval stage. Is this the only thing I could do? Thanks, @sgugger

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [August 21, 2020, 4:55pm UTC](https://discuss.huggingface.co/t/how-to-evaluate-all-model-ckpts-when-using-run-language-modeling-with-trainer/793/6 "2020-08-21T16:55:49Z")

</div>

If you’re not changing the code inside Trainer, yes it’s the only thing you can do for now. It’s not ideal, but like I said, making this better is on my TODO
