# Using Pegasus Model for Transfer Learning is generating garbage summaries

**URL:** <https://discuss.huggingface.co/t/using-pegasus-model-for-transfer-learning-is-generating-garbage-summaries/1181>\
**Category:** 🤗Transformers\
**Created:** [September 18, 2020, 11:47am UTC](https://discuss.huggingface.co/t/using-pegasus-model-for-transfer-learning-is-generating-garbage-summaries/1181 "2020-09-18T11:47:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![anant0308](https://avatars.discourse-cdn.com/v4/letter/a/3d9bf3/32.png) [@anant0308](https://discuss.huggingface.co/u/anant0308)\
**Post date:** [September 18, 2020, 11:47am UTC](https://discuss.huggingface.co/t/using-pegasus-model-for-transfer-learning-is-generating-garbage-summaries/1181/1 "2020-09-18T11:47:01Z")

</div>

Hi ,  
I am using the `PegasusForConditionalGeneration` from `transformers` library for transfer learning and generate summaries of chats based on samsum dataset. But, the output generated is repetitive and not related at all to the context.

Also, getting `loss` as `nan`. Can someone help me regarding what went wrong?

Please note that the same code works perfectly fine for `T5ForConditionalGeneration`.  
Is there any difference in the implementation method?  
Should I try with other Loss Metrics or Optimizer?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 18, 2020, 12:03pm UTC](https://discuss.huggingface.co/t/using-pegasus-model-for-transfer-learning-is-generating-garbage-summaries/1181/2 "2020-09-18T12:03:03Z")

</div>

Could you post the command you are using for fine-tuning ?

---

<div class="post-metadata">

**Author:** ![anant0308](https://avatars.discourse-cdn.com/v4/letter/a/3d9bf3/32.png) [@anant0308](https://discuss.huggingface.co/u/anant0308)\
**Post date:** [September 18, 2020, 12:24pm UTC](https://discuss.huggingface.co/t/using-pegasus-model-for-transfer-learning-is-generating-garbage-summaries/1181/3 "2020-09-18T12:24:33Z")

</div>

I’m using a Python Code for the same. A snippet of which I’m sharing below -

```auto
       for _ in range(epochs):
            self.model.train()
            train_loss = 0
            for idx, data in tqdm(enumerate(self.train_loader)):
                self.optimizer.zero_grad()
                output = self.model(input_ids = data["input_ids"], attention_mask = data["attention_mask"], lm_labels = data["lm_labels"])
                loss, prediction_scores = output[:2]  
                train_loss += loss.item()
                loss.backward()
                self.optimizer.step()
                if((idx % 1000) == 0):
                    print("loss: ", loss.item(), " train_loss: ", train_loss/(idx+1))

```

As the name of each variable suggests, I’ve `PegasusForConditionalGeneration` in the variable `self.model`, `Adam` Optimizer in `self.optimizer` and `self.train_loader` is of type `DataLoader`.
