# Fine Tune with/without LORA

**URL:** <https://discuss.huggingface.co/t/fine-tune-with-without-lora/100477>\
**Category:** 🤗Transformers\
**Created:** [August 1, 2024, 9:32pm UTC](https://discuss.huggingface.co/t/fine-tune-with-without-lora/100477 "2024-08-01T21:32:57Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![bochen0909](https://avatars.discourse-cdn.com/v4/letter/b/df788c/32.png) [@bochen0909](https://discuss.huggingface.co/u/bochen0909)\
**Post date:** [August 1, 2024, 9:32pm UTC](https://discuss.huggingface.co/t/fine-tune-with-without-lora/100477/1 "2024-08-01T21:32:57Z")

</div>

Fine tune a 8G model on chat data with/without LORA.

With LORA:

```auto
peft_config = LoraConfig(
        lora_alpha=128,
        lora_dropout=0.05,
        r=256,
        bias="none",
        target_modules="all-linear",
        task_type="CAUSAL_LM",
    )
trainer = SFTTrainer(
        model=model,
        args=training_args,
        train_dataset=dataset,
        peft_config=peft_config,
        max_seq_length=max_seq_length,
        tokenizer=tokenizer,
        packing=True,
        dataset_kwargs={
            "add_special_tokens": False,
            "append_concat_token": False,
        },
    )

```

Without LORA:

```auto
    trainer = SFTTrainer(
        model=model,
        args=training_args,
        train_dataset=dataset,
        max_seq_length=max_seq_length,
        tokenizer=tokenizer,
        packing=True,
        dataset_kwargs={
            "add_special_tokens": False,
            "append_concat_token": False,
        },
    )

```

All other things and parameters are same. LORA gave really good results, without-LORA produced non related response.

Anyone has experience and clues on the reasons?

---

<div class="post-metadata">

**Author:** ![huggingzob](https://avatars.discourse-cdn.com/v4/letter/h/b3f665/32.png) [@huggingzob](https://discuss.huggingface.co/u/huggingzob)\
**Post date:** [October 7, 2024, 7:39pm UTC](https://discuss.huggingface.co/t/fine-tune-with-without-lora/100477/2 "2024-10-07T19:39:30Z")

</div>

Isn’t it the expected result? LORA would be easier to train so if you are using same number of epochs LORA should learn faster I believe.
