# Can I compute \`eval\_loss\` and \`bleu\` score simultaneously for decoder only transformers

**URL:** <https://discuss.huggingface.co/t/can-i-compute-eval-loss-and-bleu-score-simultaneously-for-decoder-only-transformers/44297>\
**Category:** 🤗Transformers\
**Created:** [June 23, 2023, 10:05am UTC](https://discuss.huggingface.co/t/can-i-compute-eval-loss-and-bleu-score-simultaneously-for-decoder-only-transformers/44297 "2023-06-23T10:05:33Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![namespace-Pt](https://avatars.discourse-cdn.com/v4/letter/n/13edae/32.png) [@namespace-Pt](https://discuss.huggingface.co/u/namespace-Pt)\
**Post date:** [June 23, 2023, 10:05am UTC](https://discuss.huggingface.co/t/can-i-compute-eval-loss-and-bleu-score-simultaneously-for-decoder-only-transformers/44297/1 "2023-06-23T10:05:33Z")

</div>

I’m trying to fine-tune LLaMA, and I want to evalute both the `eval_loss` and the `bleu` score during training, where the former needs teacher-forcing while the latter does not.

I find the `Seq2SeqTrainer` simultaneously executes `model(**inputs).loss` to compute evaluation loss and `model.generate(**inputs)` to compute generated tokens.

However, the `inputs` are different across the two function calls, as `model(**inputs).loss`**requires the inputs to include `labels`**while `model.generate(**inputs)` **requires the inputs not to include `labels`**.

Take the sentence `I love you, do you` as an example. I train LLaMA with `I love you,` as context and `do you` as target. When evaluating, I want to inspect:

1. `log[p(do|I love you,)] + log[p(you|I love you, do)]`, i.e. `eval_loss`, which requires `I love you, do you` as `input_ids`
2. the generation results from `p(*|I love you,)`, and computing `bleu` score based on the generated results. This only requires `I love you,` as `input_ids`.

How to resolve such a conflict so as to evaluate both `eval_loss` and `bleu` score in `Seq2SeqTrainer.prediction_step()`?