# Running model.generate() in deep speed training

**URL:** <https://discuss.huggingface.co/t/running-model-generate-in-deep-speed-training/62725>\
**Category:** DeepSpeed\
**Created:** [November 19, 2023, 3:46am UTC](https://discuss.huggingface.co/t/running-model-generate-in-deep-speed-training/62725 "2023-11-19T03:46:34Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![lingy](https://avatars.discourse-cdn.com/v4/letter/l/6f9a4e/32.png) [@lingy](https://discuss.huggingface.co/u/lingy)\
**Post date:** [November 19, 2023, 3:46am UTC](https://discuss.huggingface.co/t/running-model-generate-in-deep-speed-training/62725/1 "2023-11-19T03:46:34Z")

</div>

Hi all,

I am trying to build a customized trainer evaluation loop using trainer library, however, I encounter some problem because my evaluation metric depends on string comparasion instead of logit loss. However, it seems that that model(input\_ids) followed by finding max probability for each position gives a very different output from model.generate(). But model.generate() cannot be triggered since it’s wrapped with deepspeed during evaluation.

May I ask is it possible to replicate model.generate() behavior using a wrapped model? Or is there any solution that I missed out.

---

<div class="post-metadata">

**Author:** ![terjenf](https://avatars.discourse-cdn.com/v4/letter/t/a8b319/32.png) [@terjenf](https://discuss.huggingface.co/u/terjenf)\
**Post date:** [June 6, 2024, 2:12am UTC](https://discuss.huggingface.co/t/running-model-generate-in-deep-speed-training/62725/2 "2024-06-06T02:12:20Z")

</div>

Did you find a solution for this?

---

<div class="post-metadata">

**Author:** ![Xkev](https://avatars.discourse-cdn.com/v4/letter/x/bc79bd/32.png) [@Xkev](https://discuss.huggingface.co/u/Xkev)\
**Post date:** [July 25, 2024, 3:44am UTC](https://discuss.huggingface.co/t/running-model-generate-in-deep-speed-training/62725/3 "2024-07-25T03:44:03Z")

</div>

I also encounter this issue. Is there any solution for this?
