# Resolving "Cannot Perform Fine-Tuning on Purely Quantized Models" Error in Falcon Model Training? **URL:** \ **Category:** šŸ¤—Transformers\ **Created:** [December 20, 2023, 7:00am UTC](https://discuss.huggingface.co/t/resolving-cannot-perform-fine-tuning-on-purely-quantized-models-error-in-falcon-model-training/66428 "2023-12-20T07:00:06Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![rajveer43](https://avatars.discourse-cdn.com/v4/letter/r/4af34b/32.png) [@rajveer43](https://discuss.huggingface.co/u/rajveer43)\ **Post date:** [December 20, 2023, 7:00am UTC](https://discuss.huggingface.co/t/resolving-cannot-perform-fine-tuning-on-purely-quantized-models-error-in-falcon-model-training/66428/1 "2023-12-20T07:00:06Z")
I am training Falcon model for QnA task and I want to use it for medical QnA but while traning I got the following error I also tried installing `bitsandbytes` version mentioned below and I am srtill facing the same issue can anyone suggest a better idea? How to resolve this error ```python training_args = transformers.TrainingArguments( per_device_train_batch_size=4, gradient_accumulation_steps=4, #4 num_train_epochs=6, learning_rate=2e-4, fp16=True, save_total_limit=3, logging_steps=500, output_dir="experiments", optim="paged_adamw_8bit", lr_scheduler_type="cosine", warmup_ratio=0.05, push_to_hub=True, ) trainer = transformers.Trainer( model=model, train_dataset=train_data_transformed, args=training_args, data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False) ) model.config.use_cache = False ``` error is: ```auto ValueError Traceback (most recent call last) Cell In[76], line 1 ----> 1 trainer = transformers.Trainer( 2 model=model, 3 train_dataset=train_data_transformed, 4 args=training_args, 5 data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False) 6 ) 7 model.config.use_cache = False File /opt/conda/lib/python3.10/site-packages/transformers/trainer.py:408, in Trainer. __init__ (self, model, args, data_collator, train_dataset, eval_dataset, tokenizer, model_init, compute_metrics, callbacks, optimizers, preprocess_logits_for_metrics) 406 # At this stage the model is already loaded 407 if _is_quantized_and_base_model and not _is_peft_model: --> 408 raise ValueError( 409 "You cannot perform fine-tuning on purely quantized models. Please attach trainable adapters on top of" 410 " the quantized model to correctly perform fine-tuning. Please see: https://huggingface.co/docs/transformers/peft style="color:rgb(175,0,0)">" 411 " for more details" 412 ) 413 elif _is_quantized_and_base_model and not getattr(model, "_is_quantized_training_enabled", False): 414 raise ValueError( 415 "The model you want to train is loaded in 8-bit precision. if you want to fine-tune an 8-bit" 416 " model, please make sure that you have installed `bitsandbytes>=0.37.0`. " 417 ) ValueError: You cannot perform fine-tuning on purely quantized models. Please attach trainable adapters on top of the quantized model to correctly perform fine-tuning. Please see: https://huggingface.co/docs/transformers/peft for more details ``` ---
**Author:** ![abhihash01](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@abhihash01](https://discuss.huggingface.co/u/abhihash01)\ **Post date:** [January 4, 2024, 1:35pm UTC](https://discuss.huggingface.co/t/resolving-cannot-perform-fine-tuning-on-purely-quantized-models-error-in-falcon-model-training/66428/2 "2024-01-04T13:35:39Z")
Like the error says, finetuning on purely quantized models is not possible because of certain constraints. You will have to add adapters on top of the trained model and use that. Something like peft worked for me. [Issue with Fine-tuning LLM for Classification Ā· Issue #27702 Ā· huggingface/transformers (github.com)](https://github.com/huggingface/transformers/issues/27702) [Load adapters with šŸ¤— PEFT (huggingface.co)](https://huggingface.co/docs/transformers/peft) For example you can setup a LoraConfig with peft and give it as peft\_config parameter to your trainer. from peft import LoraConfig Lora\_config = LoraConfig( r=16, lora\_alpha=32, lora\_dropout=0.05, bias=ā€œnoneā€ ) trainer = transformers.Trainer( model=model, train\_dataset=train\_data\_transformed, args=training\_args, data\_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False), peft\_config=Lora\_config ) \*\* not the expert here, but encountered the same problem ---
**Author:** ![nisarg-ai](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nisarg-ai/32/41521_2.png) [@nisarg-ai](https://discuss.huggingface.co/u/nisarg-ai)\ **Post date:** [February 19, 2025, 12:12pm UTC](https://discuss.huggingface.co/t/resolving-cannot-perform-fine-tuning-on-purely-quantized-models-error-in-falcon-model-training/66428/3 "2025-02-19T12:12:05Z")
TypeError: Trainer. **init** () got an unexpected keyword argument ā€˜peft\_configā€™ ---
**Author:** ![nisarg-ai](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nisarg-ai/32/41521_2.png) [@nisarg-ai](https://discuss.huggingface.co/u/nisarg-ai)\ **Post date:** [February 19, 2025, 12:25pm UTC](https://discuss.huggingface.co/t/resolving-cannot-perform-fine-tuning-on-purely-quantized-models-error-in-falcon-model-training/66428/4 "2025-02-19T12:25:41Z")
Restarting the kernel solves the issue. ---
**Author:** ![shubhankarwcte](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@shubhankarwcte](https://discuss.huggingface.co/u/shubhankarwcte)\ **Post date:** [May 9, 2025, 9:21am UTC](https://discuss.huggingface.co/t/resolving-cannot-perform-fine-tuning-on-purely-quantized-models-error-in-falcon-model-training/66428/5 "2025-05-09T09:21:15Z")
yess, its working now