# LoRA Adapter Loading Issue with Llama 3.1 8B - Missing Keys Warning

**URL:** <https://discuss.huggingface.co/t/lora-adapter-loading-issue-with-llama-3-1-8b-missing-keys-warning/148164>\
**Category:** Beginners\
**Created:** [March 31, 2025, 8:10am UTC](https://discuss.huggingface.co/t/lora-adapter-loading-issue-with-llama-3-1-8b-missing-keys-warning/148164 "2025-03-31T08:10:44Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mohanadhafez](https://avatars.discourse-cdn.com/v4/letter/m/0ea827/32.png) [@mohanadhafez](https://discuss.huggingface.co/u/mohanadhafez)\
**Post date:** [March 31, 2025, 8:10am UTC](https://discuss.huggingface.co/t/lora-adapter-loading-issue-with-llama-3-1-8b-missing-keys-warning/148164/1 "2025-03-31T08:10:44Z")

</div>

Hi everyone,

I’m having trouble loading my LoRA adapters for inference after fine-tuning Llama 3.1 8B. When I try to load the adapter files in a new session, I get a warning about missing adapter keys:

`/usr/local/lib/python3.11/dist-packages/peft/peft_model.py:599: UserWarning: Found missing adapter keys while loading the checkpoint: ['base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight', 'base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight', 'base_model.model.model.layers.0.self_attn.k_proj.lora_A.default.weight',...`

The warning continues with many similar missing keys. The model behaves as if it wasn’t fine-tuned at all.

My setup:

- transformers 4.43.1
- peft 0.14.0
- bitsandbytes 0.45.4
- torch 2.6.0+cu124
- accelerate 0.31.0

After training, I saved the model and got these files:

1. adapter\_model.safetensors
2. adapter\_config.json
3. training\_args.bin

Here’s how I loaded the model and adapters:

```auto
# Load base model
base_model_id = "meta-llama/Llama-3.1-8B"
lora_weights = "path to weights"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

base_model = AutoModelForCausalLM.from_pretrained(
    pretrained_model_name_or_path=base_model_id,
    quantization_config=bnb_config,
    trust_remote_code=True,
    token=True
)

# Set up tokenizer
eval_tokenizer = AutoTokenizer.from_pretrained(
    base_model_id, 
    add_bos_token=True, 
    trust_remote_code=True, 
    use_fast=True
)
eval_tokenizer.pad_token = eval_tokenizer.eos_token

# Load LoRA adapters
ft_model = PeftModel.from_pretrained(base_model, lora_weights, output_loading_info=True)

```

Any ideas on what might be causing this issue or how to fix it?

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 31, 2025, 11:12am UTC](https://discuss.huggingface.co/t/lora-adapter-loading-issue-with-llama-3-1-8b-missing-keys-warning/148164/2 "2025-03-31T11:12:44Z")

</div>

I think this is probably a warning related to this specification.

> <https://github.com/huggingface/peft/issues/2252>
>
> \### System Info
> 
> \- peft=0.13.3.dev0
> \- transformers=4.46.3
> \- Python=3.12.6
> 
> …
> \### Who can help?
> 
> @BenjaminBossan 
> @stevhliu 
> 
> \### Information
> 
> \- \[\] The official example scripts
> \- \[X\] My own modified scripts
> 
> \### Tasks
> 
> \- \[\] An officially supported task in the \`examples\` folder
> \- \[X\] My own task or dataset (give details below)
> 
> \### Reproduction
> 
> \`\`\`python
> import peft 
> import torch
> from pathlib import Path
> from transformers import AutoModelForCausalLM, AutoTokenizer
> from peft import LoraConfig, TaskType
> from tempfile import TemporaryDirectory
> 
> 
> def get\_adapter\_state\_dict(model, adapter\_name):
> adapter\_state\_dict = model.state\_dict()
> adapter\_weights = {key: value for key, value in adapter\_state\_dict.items() if adapter\_name in key}
> return adapter\_weights
> 
> def test\_adapter\_name():
> MODEL\_ID = "openai-community/gpt2"
> ADAPTER\_NAME = "lora"
>     
> # Load the base model
> base\_model = AutoModelForCausalLM.from\_pretrained(MODEL\_ID)
>     
> # Define LoraConfig without base\_model\_name\_or\_path
> peft\_config = LoraConfig(
> task\_type=TaskType.CAUSAL\_LM, 
> inference\_mode=False, 
> r=8
> )
>     
> # Get the PEFT model with the adapter name
> peft\_model = peft.get\_peft\_model(base\_model, peft\_config, adapter\_name=ADAPTER\_NAME)
>     
> # Saving and loading the model
> with TemporaryDirectory() as temp\_dir:
> temp\_dir = Path(temp\_dir)
> 
> adapter\_weights = get\_adapter\_state\_dict(peft\_model, ADAPTER\_NAME)
> peft\_model.save\_pretrained(temp\_dir)
>         
> loaded\_peft\_model = peft.PeftModel.from\_pretrained(model=base\_model,
> model\_id= temp\_dir / ADAPTER\_NAME,
> adapter\_name=ADAPTER\_NAME)
> loaded\_adapter\_weights = get\_adapter\_state\_dict(loaded\_peft\_model, ADAPTER\_NAME)
>        
> # Assertion fails due to adapter weights been newly intialized
> for key in adapter\_weights:
> assert torch.isclose(adapter\_weights\[key\], loaded\_adapter\_weights\[key\]).all()
> 
> \`\`\`
> 
> \### Expected behavior
> 
> A clear error or warning message indicating that the adapter\_name and tuner\_prefix should not be the same.

> [@The state\_dict export by peft.get\_peft\_model\_state\_dict doesn't contain adapter name](https://discuss.huggingface.co/t/the-state-dict-export-by-peft-get-peft-model-state-dict-doesnt-contain-adapter-name/136788):
>
> I want to load state\_dict export bypeft.get\_peft\_model\_state\_dict into a new model with same lora config, but I find that the keys of state\_dict export by get\_peft\_model\_state\_dict doesn’t contain lora name. For example: model\_name\_or\_path = "bert-base-cased" bert: BertModel = AutoModel.from\_pretrained(model\_name\_or\_path) lora\_bert = get\_peft\_model(bert, lora\_config) state\_dict = get\_peft\_model\_state\_dict(lora\_bert, save\_embedding\_layers=False) for key in state\_dict.keys(): print(key) """ f…

> <https://github.com/tloen/alpaca-lora/issues/334>
>
> \*\*I trained Alpaca-LoRA model with params:\*\*
> base\_model: /usr/local/dbbd/model/…llama-7b-hf
> data\_path: alpaca\_data.json
> output\_dir: ./lora-alpaca
> batch\_size: 128
> micro\_batch\_size: 4
> num\_epochs: 2
> learning\_rate: 0.0001
> cutoff\_len: 512
> val\_set\_size: 2000
> lora\_r: 8
> lora\_alpha: 16
> lora\_dropout: 0.05
> lora\_target\_modules: \['q\_proj', 'v\_proj'\]
> train\_on\_inputs: True
> group\_by\_length: True
> wandb\_project:
> wandb\_run\_name:
> wandb\_watch:
> wandb\_log\_model:
> resume\_from\_checkpoint: False
> prompt template: alpaca
> 
> \*\*But I get a wrong adapter\_model.bin which is only 443 bytes, and some output like:\*\*
> \*\*There were missing keys in the checkpoint model loaded\*\*: \['base\_model.model.model.embed\_tokens.weight', 'base\_model.model.model.layers.0.self\_attn.q\_proj.weight', 'base\_model.model.model.layers.0.self\_attn.q\_proj.lora\_A.default.weight', '.model.layers.0.self\_attn.q\_proj.lora\_B.default.weight', 'base\_model.model.model.layers.0.self\_attn.k\_proj.weight', 'base\_model.model.model.layers.0.self\_attn.v\_proj.weight', 'base\_model.model.model.layers.0.self\_attn.v\_proj.lora\_A.defaue\_model.model.model.layers.0.self\_attn.v\_proj.lora\_B.default.weight', 'base\_model.model.model.layers.0.self\_attn.o\_proj.weight', 'base\_model.model.model.layers.0.self\_attn.rotary\_emb.inv\_freq', 'base\_model.model.model.layers.0.mlp.gate\_pse\_model.model.model.layers.0.mlp.down\_proj.weight', 'base\_model.model.model.layers.0.mlp.up\_proj.weight', 'base\_model.model.model.layers.0.input\_layernorm.weight', 'base\_model.model.model.layers.0.post\_attention\_layernorm.weight', 'baseel.layers.1.self\_attn.q\_proj.weight', 'base\_model.model.model.layers.1.self\_attn.q\_proj.lora\_A.default.weight', 'base\_model.model.model.layers.1.self\_attn.q\_proj.lora\_B.default.weight', 'base\_model.model.model.layers.1.self\_attn.k\_proj.wdel.model.model.layers.1.self\_attn.v\_proj.weight', 'base\_model.model.model.layers.1.self\_attn.v\_proj.lora\_A.default.weight', 'base\_model.model.model.layers.1.self\_attn.v\_proj.lora\_B.default.weight', 'base\_model.model.model.layers.1.self\_ht', 'base\_model.model.model.layers.1.self\_attn.rotary\_emb.inv\_freq', 'base\_model.model.model.layers.1.mlp.gate\_proj.weight', 'base\_model.model.model.layers.1.mlp.down\_proj.weight', 'base\_model.model.model.layers.1.mlp.up\_proj.weight', 
> 
> Does anyone know why this is

---

<div class="post-metadata">

**Author:** ![BenjaminB](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/benjaminb/32/30898_2.png) [@BenjaminB](https://discuss.huggingface.co/u/BenjaminB)\
**Post date:** [April 7, 2025, 1:27pm UTC](https://discuss.huggingface.co/t/lora-adapter-loading-issue-with-llama-3-1-8b-missing-keys-warning/148164/3 "2025-04-07T13:27:12Z")

</div>

How did you save the checkpoint? In some circumstances, it can happen that values are lost if not careful, e.g. in distributed training.

Regarding the checkpoint, could you try loading it directly, e.g.

```python
from safetensors.torch import load_file
sd = load_file(...)
print(sd.keys())

```

It would be interesting to know which keys are present.
