# Using Tensorboard SummaryWriter with HuggingFace TrainerAPI

**URL:** <https://discuss.huggingface.co/t/using-tensorboard-summarywriter-with-huggingface-trainerapi/23015>\
**Category:** Intermediate\
**Created:** [September 12, 2022, 11:27am UTC](https://discuss.huggingface.co/t/using-tensorboard-summarywriter-with-huggingface-trainerapi/23015 "2022-09-12T11:27:00Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Anna-Kay](https://avatars.discourse-cdn.com/v4/letter/a/97f17d/32.png) [@Anna-Kay](https://discuss.huggingface.co/u/Anna-Kay)\
**Post date:** [September 12, 2022, 11:27am UTC](https://discuss.huggingface.co/t/using-tensorboard-summarywriter-with-huggingface-trainerapi/23015/1 "2022-09-12T11:27:00Z")

</div>

I am fine-tuning a HuggingFace transformer model (PyTorch version), using the HF Seq2SeqTrainingArguments & Seq2SeqTrainer, and I want to display in Tensorboard the train and validation losses (in the same chart).

As far as I understand in order to plot the two losses together I need to use the SummaryWriter. The HF Callbacks documenation describes a TensorBoardCallback function that can receive a tb\_writer argument:

> **[Callbacks](https://huggingface.co/docs/transformers/v4.21.1/en/main_classes/callback#transformers.integrations.TensorBoardCallback)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

However, I cannot figure out what is the right way to use it, if it is even supposed to be used with the Trainer API.

My code looks something like this:

```auto
args = Seq2SeqTrainingArguments(
    output_dir=output_dir,
    evaluation_strategy='epoch',
    learning_rate= 1e-5,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    weight_decay=0.01,
    save_total_limit=3,
    num_train_epochs=num_train_epochs,
    predict_with_generate=True,
    logging_steps=logging_steps,
    report_to='tensorboard',
    push_to_hub=False,  
)

trainer = Seq2SeqTrainer(
    model,
    args,
    train_dataset=tokenized_train_data,
    eval_dataset=tokenized_val_data,
    data_collator=data_collator,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
)

```

I am fine-tuning a HuggingFace transformer model (PyTorch version), using the HF Seq2SeqTrainingArguments & Seq2SeqTrainer, and I want to display in Tensorboard the train and validation losses (in the same chart).

As far as I understand in order to plot the two losses together I need to use the SummaryWriter. The HF Callbacks documenation describes a TensorBoardCallback function that can receive a tb\_writer argument:

> **[Callbacks](https://huggingface.co/docs/transformers/v4.21.1/en/main_classes/callback#transformers.integrations.TensorBoardCallback)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

However, I cannot figure out what is the right way to use it, if it is even supposed to be used with the Trainer API.

My code looks something like this:

```auto
args = Seq2SeqTrainingArguments(
    output_dir=output_dir,
    evaluation_strategy='epoch',
    learning_rate= 1e-5,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    weight_decay=0.01,
    save_total_limit=3,
    num_train_epochs=num_train_epochs,
    predict_with_generate=True,
    logging_steps=logging_steps,
    report_to='tensorboard',
    push_to_hub=False,  
)

trainer = Seq2SeqTrainer(
    model,
    args,
    train_dataset=tokenized_train_data,
    eval_dataset=tokenized_val_data,
    data_collator=data_collator,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
)

```

I would assume I should include the callback to TensorBoard in the trainer, e.g.,

```auto
callbacks = [TensorBoardCallback(tb_writer=tb_writer)]

```

but I cannot find a comprehensive example of how to use/what to import to use it.

I also found this feature request on GitHub,

> <https://github.com/huggingface/transformers/pull/4020>
>
> Implements feature request for issue #4019.
> 
> You should be able to pass in you…'re own \`SummaryWriter\`s to \`Trainer\` via the \`tb\_writer\` parameter to the \`\_\_init\_\_\` function:
> 
> \`\`\`
> tb\_writer = SummaryWriter(log\_dir="my\_log\_dir")
> tb\_writer.add\_hparams(my\_hparams\_dict, my\_metrics\_dict)
> 
> trainer = Trainer(
> model = model,
> args = training\_args,
> train\_dataset = train\_dataset,
> tb\_writer = tb\_writer
> )
> 
> trainer.train()
> \`\`\`

but no example of use, so I am confused…

Any insight will be appreciated

---

<div class="post-metadata">

**Author:** ![SUNM](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SUNM](https://discuss.huggingface.co/u/SUNM)\
**Post date:** [May 18, 2023, 11:37pm UTC](https://discuss.huggingface.co/t/using-tensorboard-summarywriter-with-huggingface-trainerapi/23015/2 "2023-05-18T23:37:00Z")

</div>

Hi @Anna-Kay , I hope you re well. sorry, did you find any solution for your problem? I have the same question in my mind.

---

<div class="post-metadata">

**Author:** ![Anna-Kay](https://avatars.discourse-cdn.com/v4/letter/a/97f17d/32.png) [@Anna-Kay](https://discuss.huggingface.co/u/Anna-Kay)\
**Post date:** [May 19, 2023, 2:55pm UTC](https://discuss.huggingface.co/t/using-tensorboard-summarywriter-with-huggingface-trainerapi/23015/3 "2023-05-19T14:55:38Z")

</div>

Hello @SUNM !

The following solution was helpful for me:

> <https://stackoverflow.com/questions/73281901/is-there-a-way-to-plot-training-and-validation-losses-on-the-same-graph-with-hug>

---

<div class="post-metadata">

**Author:** ![SUNM](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SUNM](https://discuss.huggingface.co/u/SUNM)\
**Post date:** [May 20, 2023, 1:02am UTC](https://discuss.huggingface.co/t/using-tensorboard-summarywriter-with-huggingface-trainerapi/23015/4 "2023-05-20T01:02:11Z")

</div>

@Anna-Kay , many many thanks for your attention, Sorry during training I can see the saved checkpoints, but when the training is finished no checkpoints is saved for testing. all checkpoints disappear in the folder. would you please tell m e how I can sav ethe best model , my code is as follow

```auto

training_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=10, evaluation_strategy="epoch", logging_strategy="epoch",save_strategy="epoch",seed=42,load_best_model_at_end=True,
        report_to="tensorboard",per_device_train_batch_size=2, save_total_limit=1,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path)

Trainer(model=model, args=training_args, tokenizer=tokenizer,train_dataset=train_dataset,
        eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]),
                                                              'attention_mask': torch.stack([f[1] for f in data]),
                                                              'labels': torch.stack([f[0] for f in data])}).train()

```

---

<div class="post-metadata">

**Author:** ![ReatKay](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/reatkay/32/18831_2.png) [@ReatKay](https://discuss.huggingface.co/u/ReatKay)\
**Post date:** [August 24, 2023, 7:33am UTC](https://discuss.huggingface.co/t/using-tensorboard-summarywriter-with-huggingface-trainerapi/23015/5 "2023-08-24T07:33:42Z")

</div>

Adjusted code that shows how to save. What I changed:

- Removed save\_total\_limit=1 (this basically limits your potential saves to 1, making “selection of the best model” rather useless, as there is always just 1 checkpoint if even.  
-Assigned Trainer to a local variable, so we can use the object reference after training is done.  
-Changed save strategy to steps, to enable faster first save for demonstration  
-Added save\_steps = 200, resulting in a checkpoint being saved every 200 steps.  
-Changed evaluation\_strategy=“steps” and added eval\_steps = 200 (so we can sync it with the save\_steps. In order to select the best checkpoint after training end, it makes sense, that we have evaluation (and not just training) metrics as up to date as possible when we save the model.
- Changed logging\_strategy also to “steps” , keeping everything in the same range.
- Added calls to save model etc. state after training.

```auto
training_args = TrainingArguments(output_dir=Results_Path, learning_rate=5e-5,num_train_epochs=10, evaluation_strategy="steps", logging_strategy="steps", eval_steps=200,save_strategy="steps",seed=42,load_best_model_at_end=True, save_steps = 200,
        report_to="tensorboard",per_device_train_batch_size=2,per_device_eval_batch_size=2,warmup_steps=100, weight_decay=0.01, logging_dir=Results_Path)

trainer = Trainer(model=model, args=training_args, tokenizer=tokenizer,train_dataset=train_dataset,
        eval_dataset=val_dataset,data_collator=lambda data: {'input_ids': torch.stack([f[0] for f in data]),
                                                              'attention_mask': torch.stack([f[1] for f in data]),
                                                              'labels': torch.stack([f[0] for f in data])})

trainer.train()
# With auto best model set, what happens after training, is that the system going through all the checkpoint records available for this context at the time (meaning: Being in the training output directory set in this script, being valid (in terms of meta data compared to this training, being not corrupted or errorous). After training the best checkpoint available is set and thus will be used by the following 2 statements.

model.save_pretrained(training_args.output_dir)
processor.save_pretrained(training_args.output_dir)

#If no error occurred, your output_dir now has all the files required to be a valid target for loading the model by using the path as source.

```
