# Finetuning a specific task when pretrained model isn't trained on that specific task? Using the task model vs using the base model

**URL:** <https://discuss.huggingface.co/t/finetuning-a-specific-task-when-pretrained-model-isnt-trained-on-that-specific-task-using-the-task-model-vs-using-the-base-model/1113>\
**Category:** Beginners\
**Created:** [September 14, 2020, 6:45am UTC](https://discuss.huggingface.co/t/finetuning-a-specific-task-when-pretrained-model-isnt-trained-on-that-specific-task-using-the-task-model-vs-using-the-base-model/1113 "2020-09-14T06:45:37Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mayanksatnalika](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mayanksatnalika/32/344_2.png) [@mayanksatnalika](https://discuss.huggingface.co/u/mayanksatnalika)\
**Post date:** [September 14, 2020, 6:45am UTC](https://discuss.huggingface.co/t/finetuning-a-specific-task-when-pretrained-model-isnt-trained-on-that-specific-task-using-the-task-model-vs-using-the-base-model/1113/1 "2020-09-14T06:45:37Z")

</div>

I want to fine tune a `RobertaForSequenceClassification` task on `microsoft/codebert-base` model. This `microsoft/codebert-base` model hasn’t been trained for Sequence-Classification task.  
Can I load this pre-trained model inside a SequenceClassification function and fine tune it on my dataset?

```auto
model = RobertaForSequenceClassification.from_pretrained( "microsoft/codebert-base" )

Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at microsoft/codebert-base and are newly initialized: ['classifier.dense.weight', 'classifier.out_proj.weight', 'classifier.out_proj.bias', 'classifier.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

```

While loading I get this message which is expected as the model isn’t trained on the task and thus would not have the weights.

Can I proceed with fine-tuning this `RobertaForSequenceClassification` model or would I need to define my own classifier layer on top of `RobertaModel` and train that?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 14, 2020, 6:51am UTC](https://discuss.huggingface.co/t/finetuning-a-specific-task-when-pretrained-model-isnt-trained-on-that-specific-task-using-the-task-model-vs-using-the-base-model/1113/2 "2020-09-14T06:51:35Z")

</div>

Hi @mayanksatnalika  
Yes, you can load a pre-trained base model for SequenceClassification.  
`RobertaForSequenceClassification` adds the classification head itself, you won’t need to do that manually. So you can fine-tune it for classification.

---

<div class="post-metadata">

**Author:** ![mayanksatnalika](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mayanksatnalika/32/344_2.png) [@mayanksatnalika](https://discuss.huggingface.co/u/mayanksatnalika)\
**Post date:** [September 14, 2020, 6:57am UTC](https://discuss.huggingface.co/t/finetuning-a-specific-task-when-pretrained-model-isnt-trained-on-that-specific-task-using-the-task-model-vs-using-the-base-model/1113/3 "2020-09-14T06:57:21Z")

</div>

Thank you @valhalla for the quick reply.

Just trying to understand it, I came across this [https://github.com/curiousily/Getting-Things-Done-with-Pytorch/blob/master/08.sentiment-analysis-with-bert.ipynb](https://github.com/curiousily/Getting-Things-Done-with-Pytorch/blob/master/08.sentiment-analysis-with-bert.ipynb) here a classifier is added on top of base model and trained rather than directly using a …forSequenceClassification task, would it be possible to tell me how the 2 approaches differ or any relevant links on it 😅

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 14, 2020, 7:01am UTC](https://discuss.huggingface.co/t/finetuning-a-specific-task-when-pretrained-model-isnt-trained-on-that-specific-task-using-the-task-model-vs-using-the-base-model/1113/4 "2020-09-14T07:01:27Z")

</div>

`ForSequenceClassification` models does pretty much the same. It takes a base model and adds pooler and classfier head on top of it, so you won’t have to do it manually. You can see how it’s done [here](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_roberta.py#L428), its’ pretty easy to follow

---

<div class="post-metadata">

**Author:** ![mayanksatnalika](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mayanksatnalika/32/344_2.png) [@mayanksatnalika](https://discuss.huggingface.co/u/mayanksatnalika)\
**Post date:** [September 14, 2020, 7:11am UTC](https://discuss.huggingface.co/t/finetuning-a-specific-task-when-pretrained-model-isnt-trained-on-that-specific-task-using-the-task-model-vs-using-the-base-model/1113/5 "2020-09-14T07:11:59Z")

</div>

Thanks a lot 🙂
