# How to run validation on multiple evaluation datasets simultaneously during Qwen2.5-VL-7B-Instruct fine-tuning?

**URL:** <https://discuss.huggingface.co/t/how-to-run-validation-on-multiple-evaluation-datasets-simultaneously-during-qwen2-5-vl-7b-instruct-fine-tuning/166313>\
**Category:** 🤗Transformers\
**Created:** [August 10, 2025, 1:42pm UTC](https://discuss.huggingface.co/t/how-to-run-validation-on-multiple-evaluation-datasets-simultaneously-during-qwen2-5-vl-7b-instruct-fine-tuning/166313 "2025-08-10T13:42:57Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![cvlyt](https://avatars.discourse-cdn.com/v4/letter/c/ebca7d/32.png) [@cvlyt](https://discuss.huggingface.co/u/cvlyt)\
**Post date:** [August 10, 2025, 1:42pm UTC](https://discuss.huggingface.co/t/how-to-run-validation-on-multiple-evaluation-datasets-simultaneously-during-qwen2-5-vl-7b-instruct-fine-tuning/166313/1 "2025-08-10T13:42:57Z")

</div>

I’m trying to fine-tune **Qwen2.5-VL-7B-Instruct** and want to use **two evaluation datasets** (eval\_dataset\_A, eval\_dataset\_B) to compute validation loss **during training**.  
I referred to the official fine-tuning parameters here:  
[https://github.com/QwenLM/Qwen2.5-VL/tree/main/qwen-vl-finetune](https://github.com/QwenLM/Qwen2.5-VL/tree/main/qwen-vl-finetune)

I added the following parameters:

```auto
--eval_dataset_use eval_dataset_A,eval_dataset_B
--eval_strategy steps
--eval_steps 2

```

My goal: use **both eval\_dataset\_A and eval\_dataset\_B** for validation loss calculation at each `eval_steps`.

In `qwen-vl-finetune/qwenvl/train/trainer.py`, there is the following code:

```python
if data_args.data_packing:
    data_module = make_supervised_data_module_packed(tokenizer=tokenizer, data_args=data_args)
else:
    data_module = make_supervised_data_module(tokenizer=tokenizer, data_args=data_args)

trainer = Trainer(
    model=model, processing_class=tokenizer, args=training_args, **data_module
)

```

How should I modify this so that `Trainer` evaluates on **two different evaluation datasets**?

If anyone has done something similar, could you share example code?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [August 10, 2025, 1:54pm UTC](https://discuss.huggingface.co/t/how-to-run-validation-on-multiple-evaluation-datasets-simultaneously-during-qwen2-5-vl-7b-instruct-fine-tuning/166313/2 "2025-08-10T13:54:08Z")

</div>

Seems [relatively new `Trainer` natively evaluates each dataset when `eval_dataset` is a dict and prefixes metrics with the dict key](https://github.com/huggingface/transformers/issues/24962) like this:

```py
eval_datasets = {"A": eval_dataset_A, "B": eval_dataset_B}
...
trainer = Trainer(
    model=model, processing_class=tokenizer, args=training_args, eval_dataset=eval_datasets, **data_module
)

```

### [Train with multiple eval datasets raises an Exception](https://github.com/huggingface/transformers/issues/20565)

---

<div class="post-metadata">

**Author:** ![cvlyt](https://avatars.discourse-cdn.com/v4/letter/c/ebca7d/32.png) [@cvlyt](https://discuss.huggingface.co/u/cvlyt)\
**Post date:** [August 10, 2025, 3:01pm UTC](https://discuss.huggingface.co/t/how-to-run-validation-on-multiple-evaluation-datasets-simultaneously-during-qwen2-5-vl-7b-instruct-fine-tuning/166313/3 "2025-08-10T15:01:52Z")

</div>

Thanks to your advice, I was able to create the following code. I really appreciate your help.

```python
if data_args.data_packing:
    data_module = make_supervised_data_module_packed(tokenizer=tokenizer, data_args=data_args)
else:
    data_module = make_supervised_data_module(tokenizer=tokenizer, data_args=data_args)

if training_args.do_eval:
    if data_args.eval_dataset_use is None:
        raise ValueError("do_eval is True but eval_dataset_use is not set")
    
    eval_datasets = {}
    eval_dataset_names = data_args.eval_dataset_use.split(',')
    for name in eval_dataset_names:
        eval_data_args = copy.deepcopy(data_args)
        eval_data_args.dataset_use = name
        eval_datasets[name] = LazySupervisedDataset(tokenizer=tokenizer, data_args=eval_data_args)
else:
    eval_datasets = None

if "eval_dataset" in data_module:
    del data_module["eval_dataset"]

trainer = Trainer(
    model=model,
    tokenizer=tokenizer,
    args=training_args,
    eval_dataset=eval_datasets,
    **data_module,
)

```