# Model trains with Seq2SeqTrainer but gets stuck using Trainer

**URL:** <https://discuss.huggingface.co/t/model-trains-with-seq2seqtrainer-but-gets-stuck-using-trainer/9333>\
**Category:** 🤗Transformers\
**Created:** [August 18, 2021, 6:39pm UTC](https://discuss.huggingface.co/t/model-trains-with-seq2seqtrainer-but-gets-stuck-using-trainer/9333 "2021-08-18T18:39:32Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ssam9](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@ssam9](https://discuss.huggingface.co/u/ssam9)\
**Post date:** [August 18, 2021, 6:39pm UTC](https://discuss.huggingface.co/t/model-trains-with-seq2seqtrainer-but-gets-stuck-using-trainer/9333/1 "2021-08-18T18:39:33Z")

</div>

Hi,

I’ve been trying to finetune the BART large pre-trained on MNLI with the Financial Phrasebank dataset to build a model for news sentiment analysis. I’m just a beginner and so, I mostly use the code from [GEM Getting Started](https://gem-benchmark.com/get_started).

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, Seq2SeqTrainer, Seq2SeqTrainingArguments

tokenizer = AutoTokenizer.from\_pretrained(‘facebook/bart-large-mnli’)  
model = AutoModelForSeq2SeqLM.from\_pretrained(‘facebook/bart-large-mnli’)

The model only trains when I use the AutoModelForSeq2SeqLM, Seq2SeqTrainer andSeq2SeqTrainingArguments. When I use model = AutoModelForSequenceClassification.from\_pretrained(‘facebook/bart-large-mnli’) with the Trainer and TrainingArguments, the model does not train.

Is it appropriate to use seq2seq for sentiment classification tasks?

Any suggestions would be immensely helpful.

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![aliosk](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@aliosk](https://discuss.huggingface.co/u/aliosk)\
**Post date:** [August 19, 2021, 1:06pm UTC](https://discuss.huggingface.co/t/model-trains-with-seq2seqtrainer-but-gets-stuck-using-trainer/9333/2 "2021-08-19T13:06:31Z")

</div>

Are you framing your classification problem as a sequence generation task? what types of labels do you have for your training data? Are the labels text/sequence or a finite number of categories? If your task is classification I believe you’re using the wrong model class. You could probably use BertForSequenceClassification for a sentiment analysis task as has been done in the link below:

> **[nlptown/bert-base-multilingual-uncased-sentiment at main](https://huggingface.co/nlptown/bert-base-multilingual-uncased-sentiment/tree/main)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

And instead of using Seq2SeqTrainer, just use Trainer and TrainingArguments.

---

<div class="post-metadata">

**Author:** ![ssam9](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@ssam9](https://discuss.huggingface.co/u/ssam9)\
**Post date:** [August 20, 2021, 8:54am UTC](https://discuss.huggingface.co/t/model-trains-with-seq2seqtrainer-but-gets-stuck-using-trainer/9333/3 "2021-08-20T08:54:58Z")

</div>

Thanks a lot for replying.

> Are you framing your classification problem as a sequence generation task?

I dont know what this means. What I’ve been trying is: given a news headline, predict the sentiment of it as being positive or negative, just like how the financial phrasebank dataset looks like.

> what types of labels do you have for your training data?

I’m trying to use the bart-large-mnli model and finetune it on the financial phrasebank dataset. The financial phrasebank data has 3 labels: positive, neutral and negative.

> Are the labels text/sequence or a finite number of categories?

Yes. Positive, negative and neutral.

> If your task is classification I believe you’re using the wrong model class. You could probably use BertForSequenceClassification for a sentiment analysis task as has been done in the link below:

Thanks for the link. I dont know why but if I use TrainingArguments and Trainer, I either get an error as “CUDA out of memory” or “Expect input batch size to meet targeted batch size”.

Thanks again for your inputs.

---

<div class="post-metadata">

**Author:** ![aliosk](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@aliosk](https://discuss.huggingface.co/u/aliosk)\
**Post date:** [August 20, 2021, 9:55am UTC](https://discuss.huggingface.co/t/model-trains-with-seq2seqtrainer-but-gets-stuck-using-trainer/9333/4 "2021-08-20T09:55:02Z")

</div>

CUDA out of memory happens when your model is using more memory than the GPU can offer. You could try reducing the batch size or turning on the gradient\_checkpointing. Or alternatively use a GPU with higher memory.

---

<div class="post-metadata">

**Author:** ![ssam9](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@ssam9](https://discuss.huggingface.co/u/ssam9)\
**Post date:** [August 23, 2021, 9:40am UTC](https://discuss.huggingface.co/t/model-trains-with-seq2seqtrainer-but-gets-stuck-using-trainer/9333/5 "2021-08-23T09:40:46Z")

</div>

Thank you so much for your suggestions!!