# How to continue training with HuggingFace Trainer?

**URL:** <https://discuss.huggingface.co/t/how-to-continue-training-with-huggingface-trainer/39397>\
**Category:** 🤗Transformers\
**Created:** [May 10, 2023, 11:54am UTC](https://discuss.huggingface.co/t/how-to-continue-training-with-huggingface-trainer/39397 "2023-05-10T11:54:32Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![alvations](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alvations/32/97_2.png) [@alvations](https://discuss.huggingface.co/u/alvations)\
**Post date:** [May 10, 2023, 11:54am UTC](https://discuss.huggingface.co/t/how-to-continue-training-with-huggingface-trainer/39397/1 "2023-05-10T11:54:33Z")

</div>

When training a model with Huggingface Trainer object, e.g. from [Neural Plasticity - Bert2Bert on WMT14 | Kaggle](https://www.kaggle.com/code/alvations/neural-plasticity-bert2bert-on-wmt14)

```auto
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments

import os
os.environ["WANDB_DISABLED"] = "true"

batch_size = 2

# set training arguments - these params are not really tuned, feel free to change
training_args = Seq2SeqTrainingArguments(
    output_dir="./",
    evaluation_strategy="steps",
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    predict_with_generate=True,
    logging_steps=2, # set to 1000 for full training
    save_steps=16, # set to 500 for full training
    eval_steps=4, # set to 8000 for full training
    warmup_steps=1, # set to 2000 for full training
    max_steps=16, # delete for full training
    # overwrite_output_dir=True,
    save_total_limit=1,
    #fp16=True, 
)

# instantiate trainer
trainer = Seq2SeqTrainer(
    model=multibert,
    tokenizer=tokenizer,
    args=training_args,
    train_dataset=train_data,
    eval_dataset=val_data,
)

trainer.train()

```

When it finished training, it outputs:

```auto
TrainOutput(global_step=16, training_loss=10.065429925918579, metrics={'train_runtime': 541.4209, 'train_samples_per_second': 0.059, 'train_steps_per_second': 0.03, 'total_flos': 19637939109888.0, 'train_loss': 10.065429925918579, 'epoch': 0.03})

```

**If we want to continue training with more steps, e.g. `max_steps=16` (from previous `trainer.train()` run) and another `max_steps=160`, do we do something like this?**

```auto
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments

import os
os.environ["WANDB_DISABLED"] = "true"

batch_size = 2

# set training arguments - these params are not really tuned, feel free to change
training_args = Seq2SeqTrainingArguments(
    output_dir="./",
    evaluation_strategy="steps",
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    predict_with_generate=True,
    logging_steps=2, # set to 1000 for full training
    save_steps=16, # set to 500 for full training
    eval_steps=4, # set to 8000 for full training
    warmup_steps=1, # set to 2000 for full training
    max_steps=16, # delete for full training
    # overwrite_output_dir=True,
    save_total_limit=1,
    #fp16=True, 
)

# instantiate trainer
trainer = Seq2SeqTrainer(
    model=multibert,
    tokenizer=tokenizer,
    args=training_args,
    train_dataset=train_data,
    eval_dataset=val_data,
)

# First 16 steps.
trainer.train()

# set training arguments - these params are not really tuned, feel free to change
training_args_2 = Seq2SeqTrainingArguments(
    output_dir="./",
    evaluation_strategy="steps",
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    predict_with_generate=True,
    logging_steps=2, # set to 1000 for full training
    save_steps=16, # set to 500 for full training
    eval_steps=4, # set to 8000 for full training
    warmup_steps=1, # set to 2000 for full training
    max_steps=160, # delete for full training
    # overwrite_output_dir=True,
    save_total_limit=1,
    #fp16=True, 
)

# instantiate trainer
trainer = Seq2SeqTrainer(
    model=multibert,
    tokenizer=tokenizer,
    args=training_args_2,
    train_dataset=train_data,
    eval_dataset=val_data,
)

# Continue training for 160 steps
trainer.train()

```

**If the above is not the canonical way to continue training a model, how to continue training with HuggingFace Trainer?**

---

<div class="post-metadata">

**Author:** ![alvations](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alvations/32/97_2.png) [@alvations](https://discuss.huggingface.co/u/alvations)\
**Post date:** [May 10, 2023, 11:54am UTC](https://discuss.huggingface.co/t/how-to-continue-training-with-huggingface-trainer/39397/2 "2023-05-10T11:54:55Z")

</div>

Also asked on [python - How to continue training with HuggingFace Trainer? - Stack Overflow](https://stackoverflow.com/questions/76217781/how-to-continue-training-with-huggingface-trainer)

---

<div class="post-metadata">

**Author:** ![alvations](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alvations/32/97_2.png) [@alvations](https://discuss.huggingface.co/u/alvations)\
**Post date:** [May 15, 2023, 5:49pm UTC](https://discuss.huggingface.co/t/how-to-continue-training-with-huggingface-trainer/39397/3 "2023-05-15T17:49:54Z")

</div>

There’s a suggestion to use `resume_from_checkpoint` argument in Trainer but it seems like it’s not available for Seq2SeqTrainer:

```auto
trainer = Seq2SeqTrainer(
    model=multibert,
    tokenizer=tokenizer,
    args=training_args,
    train_dataset=train_data,
    eval_dataset=val_data,
    resume_from_checkpoint=True
)

```

[out]:

```auto
TypeError: Seq2SeqTrainer. __init__ () got an unexpected keyword argument 'resume_from_checkpoint'

```

---

<div class="post-metadata">

**Author:** ![maciejskorski](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/maciejskorski/32/57000_2.png) [@maciejskorski](https://discuss.huggingface.co/u/maciejskorski)\
**Post date:** [May 16, 2023, 8:58am UTC](https://discuss.huggingface.co/t/how-to-continue-training-with-huggingface-trainer/39397/4 "2023-05-16T08:58:02Z")

</div>

The [original answer on SO](https://stackoverflow.com/questions/76217781/how-to-continue-training-with-huggingface-trainer/76246674#76246674) was referencing the correct example in  
[huggingface docs](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer.train.resume_from_checkpoint), but had a bug in describing it. Namely, the parameter `resume_from_checkpoint` should be used in the `train` call rather than in the `init`.

---

<div class="post-metadata">

**Author:** ![vergilus](https://avatars.discourse-cdn.com/v4/letter/v/0ea827/32.png) [@vergilus](https://discuss.huggingface.co/u/vergilus)\
**Post date:** [September 5, 2024, 10:04am UTC](https://discuss.huggingface.co/t/how-to-continue-training-with-huggingface-trainer/39397/5 "2024-09-05T10:04:49Z")

</div>

trainer.train(resume\_from\_checkpoint=True)