# Peft Model For SequenceClassification failing \_is\_peft\_model

**URL:** <https://discuss.huggingface.co/t/peft-model-for-sequenceclassification-failing-is-peft-model/72750>\
**Category:** Beginners\
**Created:** [February 11, 2024, 6:39pm UTC](https://discuss.huggingface.co/t/peft-model-for-sequenceclassification-failing-is-peft-model/72750 "2024-02-11T18:39:59Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![JacksonMcDaniels](https://avatars.discourse-cdn.com/v4/letter/j/e274bd/32.png) [@JacksonMcDaniels](https://discuss.huggingface.co/u/JacksonMcDaniels)\
**Post date:** [February 11, 2024, 6:39pm UTC](https://discuss.huggingface.co/t/peft-model-for-sequenceclassification-failing-is-peft-model/72750/1 "2024-02-11T18:39:59Z")

</div>

I am attempting to finetune MixtralForSequenceClassification, and believe I have a peft model. However, when I call trainer.train() I run into the error “ValueError: You cannot perform fine-tuning on purely quantized models. Please attach trainable adapters on top of the quantized model to correctly perform fine-tuning. Please see: [Load adapters with 🤗 PEFT](https://huggingface.co/docs/transformers/peft) for more details”

Here’s what I did to get here, following the steps in a few popular colab example notebooks:

1. Loaded mixtral using 4bit quantization with a standard config

```auto
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

```

1. Prepared for kbit training

```auto
from peft import prepare_model_for_kbit_training

model.gradient_checkpointing_enable()

model = prepare_model_for_kbit_training(model)

```

1. Used a standard LoraConfig with task\_type SEQ\_CLS

```auto
from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=32,
    lora_alpha=64,
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
        "lm_head",
    ],
    bias="none",
    lora_dropout=0.05, # Conventional
    task_type="SEQ_CLS",
)

model = get_peft_model(model, config)
print_trainable_parameters(model)

```

When I check if the model is an instance of a PeftModel, it returns true:

`isinstance(model, peft.PeftModel)`

And printing the model yields this:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/6/6/661dd4f50cc5decd2d097ff802d64149731b52b5.png)

However \_is\_peft\_model is returning false

I followed similar steps for finetuning Llama ~4 months ago and did not run into this error. Doing the exact same thing as some of these colab notebooks I have seen, so not sure what the error is. Perhaps something specific to sequence classification?

Any help would be much appreciated!
