# Training issue of a Transformer based Encoder-Decoder model based on pre-trained BanglaBERT

**URL:** <https://discuss.huggingface.co/t/training-issue-of-a-transformer-based-encoder-decoder-model-based-on-pre-trained-banglabert/17632>\
**Category:** Models\
**Created:** [May 7, 2022, 6:55am UTC](https://discuss.huggingface.co/t/training-issue-of-a-transformer-based-encoder-decoder-model-based-on-pre-trained-banglabert/17632 "2022-05-07T06:55:33Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![AbuUbaida](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@AbuUbaida](https://discuss.huggingface.co/u/AbuUbaida)\
**Post date:** [May 7, 2022, 6:55am UTC](https://discuss.huggingface.co/t/training-issue-of-a-transformer-based-encoder-decoder-model-based-on-pre-trained-banglabert/17632/1 "2022-05-07T06:55:33Z")

</div>

I just tried to train an EncoderDeoder model for **summarization** task based on pre-trained _[BanglaBERT](https://huggingface.co/csebuetnlp/banglabert)_, which is an **ELECTRA** discriminator model pre-trained with the Replaced Token Detection (RTD) objective. Surprisingly, after spending 4500 steps on 10k training data, the model wasn’t trained a bit since the **ROUGE-2** scores were just **0.0000**. To make sure I used that 4500-checkpoint to generate summaries for testing purposes; it generated a fixed-length (50) output (even if I change the test input of different lengths) containing the **[CLS]** token **49** times and a **[SEP]** token **lastly**. Basically, I followed the [Warm-starting encoder-decoder models with 🤗Transformers](https://colab.research.google.com/drive/1Ekd5pUeCX7VOrMx94_czTkwNtLN32Uyu?usp=sharing) notebook. Can anybody give any clue what could be the issue here? Thanks in advance.

In my case,

Tokenization **BanglaBERT** model:

```auto
tokenizer = AutoTokenizer.from_pretrained("csebuetnlp/banglabert")
tokenizer.bos_token = tokenizer.cls_token
tokenizer.eos_token = tokenizer.sep_token

```

Input pre-processing function:

```auto
def process_data_to_model_inputs(batch):
    inputs = tokenizer(batch['text'], padding="max_length", truncation=True, max_length=encoder_max_length)
    outputs = tokenizer(batch['summary'], padding="max_length", truncation=True, max_length=decoder_max_length)

    batch["input_ids"] = inputs.input_ids
    batch["attention_mask"] = inputs.attention_mask
    batch["decoder_input_ids"] = outputs.input_ids
    batch["decoder_attention_mask"] = outputs.attention_mask
    batch["labels"] = outputs.input_ids.copy()

    batch["labels"] = [[-100 if token == tokenizer.pad_token_id else token for token in labels] for labels in batch["labels"]]
    return batch

```

Mapping the pre-processing function to the batches of examples:

```auto
train_data = train_data.map(
    process_data_to_model_inputs, 
    batched=True,
    batch_size=batch_size,
    remove_columns=["text", "summary"]
)
train_data.set_format(
    type="torch", columns=["input_ids", "attention_mask", "decoder_input_ids", "decoder_attention_mask", "labels"],
)

valid_data = valid_data.map(
    process_data_to_model_inputs, 
    batched=True, 
    batch_size=batch_size,
    remove_columns=["text", "summary"]
)
valid_data.set_format(
    type="torch", columns=["input_ids", "attention_mask", "decoder_input_ids", "decoder_attention_mask", "labels"],
)

```

**BanglaBERT** model and its config settings:

```auto
bert2bert = EncoderDecoderModel.from_encoder_decoder_pretrained("csebuetnlp/banglabert", "csebuetnlp/banglabert")
bert2bert.config.decoder_start_token_id = tokenizer.bos_token_id
bert2bert.config.eos_token_id = tokenizer.eos_token_id
bert2bert.config.pad_token_id = tokenizer.pad_token_id

bert2bert.config.vocab_size = bert2bert.config.decoder.vocab_size
bert2bert.config.max_length = 128
bert2bert.config.min_length = 42
bert2bert.config.early_stopping = True
bert2bert.config.length_penalty = 2.0
bert2bert.config.num_beams = 8
bert2bert.config.remove_invalid_values = True
bert2bert.config.repetition_penalty = 2.0
bert2bert.config.length_penalty = 2.0

```

I used the **Seq2SeqTrainer** for training. The **Seq2SeqTrainingArguments** were as follows:

```auto
    evaluation_strategy = "steps",
    per_device_train_batch_size = batch_size,
    per_device_eval_batch_size = batch_size,
    predict_with_generate = True,
    logging_steps = 1000, 
    save_steps = 500, 
    eval_steps = 5000, 
    warmup_steps = 500,
    overwrite_output_dir = True,
    save_total_limit = 2,
    num_train_epochs = 20,
    fp16 = True

```

---

<div class="post-metadata">

**Author:** ![AbuUbaida](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@AbuUbaida](https://discuss.huggingface.co/u/AbuUbaida)\
**Post date:** [May 12, 2022, 5:23pm UTC](https://discuss.huggingface.co/t/training-issue-of-a-transformer-based-encoder-decoder-model-based-on-pre-trained-banglabert/17632/2 "2022-05-12T17:23:40Z")

</div>

see here [encoder-decoder (bert2bert) model for summarization task doesn't work in v4.18 · Issue #292 · huggingface/blog · GitHub](https://github.com/huggingface/blog/issues/292#issue-1228916923)
