# Issues with fine tuning an Encoder Decoder Model

**URL:** <https://discuss.huggingface.co/t/issues-with-fine-tuning-an-encoder-decoder-model/48880>\
**Category:** 🤗Transformers\
**Created:** [July 31, 2023, 8:40pm UTC](https://discuss.huggingface.co/t/issues-with-fine-tuning-an-encoder-decoder-model/48880 "2023-07-31T20:40:21Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sid0212](https://avatars.discourse-cdn.com/v4/letter/s/0ea827/32.png) [@Sid0212](https://discuss.huggingface.co/u/Sid0212)\
**Post date:** [July 31, 2023, 8:40pm UTC](https://discuss.huggingface.co/t/issues-with-fine-tuning-an-encoder-decoder-model/48880/1 "2023-07-31T20:40:21Z")

</div>

I am training an encode decoder for summarization of news articles into title with the help of news. I am trying to fine tune encoder decoder model as mentioned in the code below. the problem I am facing is that the model is not learning and just generating the same thing. I am not able to Understanding what I am doing wrong.

batch\_size=64  
encoder\_max\_length=128  
decoder\_max\_length=16

roberta2roberta = EncoderDecoderModel.from\_encoder\_decoder\_pretrained(  
“roberta-base”,  
“roberta-base”  
)#.to(“cuda”)  
tokenizer = RobertaTokenizer.from\_pretrained(“roberta-base”)

# set special tokens

roberta2roberta.config.decoder\_start\_token\_id = tokenizer.bos\_token\_id  
roberta2roberta.config.eos\_token\_id = tokenizer.eos\_token\_id  
roberta2roberta.config.pad\_token\_id = tokenizer.pad\_token\_id  
roberta2roberta.config.bos\_token\_id = tokenizer.bos\_token\_id

# sensible parameters for beam search

roberta2roberta.config.vocab\_size = tokenizer.vocab\_size  
roberta2roberta.config.max\_length = 16  
roberta2roberta.config.min\_length = 4  
roberta2roberta.config.no\_repeat\_ngram\_size = 1  
roberta2roberta.config.early\_stopping = True  
roberta2roberta.config.length\_penalty = 2.0  
roberta2roberta.config.num\_beams = 4

def process\_data\_to\_model\_inputs(batch):

# tokenize the inputs and labels

```
inputs = tokenizer(batch["title_s_article_s"], padding="max_length", truncation=True, max_length=encoder_max_length)
outputs = tokenizer(batch["highlight"], padding="max_length", truncation=True, max_length=decoder_max_length)

batch["input_ids"] = inputs.input_ids
batch["attention_mask"] = inputs.attention_mask
batch["decoder_input_ids"] = outputs.input_ids
batch["decoder_attention_mask"] = outputs.attention_mask
batch["labels"] = outputs.input_ids.copy()

# because BERT automatically shifts the labels, the labels correspond exactly to `decoder_input_ids`.
# We have to make sure that the PAD token is ignored
batch["labels"] = [[-100 if token == tokenizer.pad_token_id else token for token in labels] for labels in batch["labels"]]

return batch

```

# only use 32 training examples for notebook - DELETE LINE FOR FULL TRAINING

train\_data = dataset[‘train’]#.select(range(32))

train\_data = train\_data.map(  
process\_data\_to\_model\_inputs,  
batched=True,  
batch\_size=batch\_size,  
remove\_columns=[“title\_s\_article\_s”, “highlight”]  
)  
train\_data.set\_format(  
type=“torch”, columns=[“input\_ids”, “attention\_mask”, “decoder\_input\_ids”, “decoder\_attention\_mask”, “labels”],  
)

# only use 16 training examples for notebook - DELETE LINE FOR FULL TRAINING

val\_data = dataset[‘eval’]#.select(range(16))

val\_data =val\_data.map(  
process\_data\_to\_model\_inputs,  
batched=True,  
batch\_size=batch\_size,  
remove\_columns=[“title\_s\_article\_s”, “highlight”]  
)  
val\_data.set\_format(  
type=“torch”, columns=[“input\_ids”, “attention\_mask”, “decoder\_input\_ids”, “decoder\_attention\_mask”, “labels”],  
)

# set training arguments - these params are not really tuned, feel free to change

training\_args = Seq2SeqTrainingArguments(  
output\_dir=“./checkpoint”,  
evaluation\_strategy=“steps”,  
per\_device\_train\_batch\_size=batch\_size,  
per\_device\_eval\_batch\_size=batch\_size,  
predict\_with\_generate=True,  
logging\_steps=100,  
save\_steps=1000,  
eval\_steps=100,  
warmup\_steps=100,  
overwrite\_output\_dir=True,  
save\_total\_limit=1,  
num\_train\_epochs = 12,  
learning\_rate=1e-2,  
fp16=True,  
lr\_scheduler\_type = ‘cosine’,  
# resume\_from\_checkpoint = ‘./checkpoint/checkpoint-2000’,  
# no\_cuda = True,  
)

# load rouge for validation

rouge = load(“rouge”)

def compute\_metrics(pred):  
labels\_ids = pred.label\_ids  
pred\_ids = pred.predictions

```
# all unnecessary tokens are removed
pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
labels_ids[labels_ids == -100] = tokenizer.pad_token_id
label_str = tokenizer.batch_decode(labels_ids, skip_special_tokens=True)
print('START')
for x,y in zip(pred_str[:10],label_str[:10]):
    print('PRED: ',x,"LABEL: ",y)
print('END')
rouge_output = rouge.compute(
    predictions=pred_str,
    references=label_str
)
print_gpu_utilization()
return rouge_output

```

# instantiate trainer

trainer = Seq2SeqTrainer(  
model=roberta2roberta,  
tokenizer=tokenizer,  
args=training\_args,  
compute\_metrics=compute\_metrics,  
train\_dataset=train\_data,  
eval\_dataset=val\_data,  
)  
trainer.train()

OUTPUT (For training loop )

[1201/4776 07:18 \< 21:48, 2.73 it/s, Epoch 3.02/12]  
Step Training Loss Validation Loss Rouge1 Rouge2 Rougel Rougelsum  
100 7.940000 7.168086 0.065070 0.000000 0.064863 0.064918  
200 7.074200 7.004178 0.079902 0.000000 0.079906 0.079832  
300 7.017600 6.979647 0.079902 0.000000 0.079906 0.079832  
400 6.997000 6.959394 0.079902 0.000000 0.079906 0.079832  
500 6.914100 6.962692 0.079902 0.000000 0.079906 0.079832  
600 6.934400 6.955380 0.079902 0.000000 0.079906 0.079832  
700 6.936800 6.958053 0.079902 0.000000 0.079906 0.079832  
800 6.941300 6.962934 0.079902 0.000000 0.079906 0.079832  
900 6.910400 6.973436 0.079902 0.000000 0.079906 0.079832  
1000 6.921500 6.962850 0.079902 0.000000 0.079906 0.079832  
1100 6.921500 6.969145 0.079902 0.000000 0.079906 0.079832  
[4/11 00:02 \< 00:04, 1.50 it/s]

The prediction are repetitive. I am not able to understand what I am doing wrong.  
My dataset is 25K datapoint