# Error Debugging

**URL:** <https://discuss.huggingface.co/t/error-debugging/152800>\
**Category:** Beginners\
**Created:** [April 29, 2025, 8:02am UTC](https://discuss.huggingface.co/t/error-debugging/152800 "2025-04-29T08:02:01Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ibsaa](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ibsaa/32/46564_2.png) [@Ibsaa](https://discuss.huggingface.co/u/Ibsaa)\
**Post date:** [April 29, 2025, 8:02am UTC](https://discuss.huggingface.co/t/error-debugging/152800/1 "2025-04-29T08:02:01Z")

</div>

## i have been encountering the following error message for a while:

## `.to` is not supported for `4-bit` or `8-bit` bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct `dtype`.

and the code that throws the error is as follows:

# loading an LLM locally

import os  
import torch  
from dotenv import load\_dotenv  
from huggingface\_hub import login  
from transformers import (  
AutoTokenizer,  
AutoModelForCausalLM,  
BitsAndBytesConfig,  
)  
from transformers.utils import is\_flash\_attn\_2\_available

# 00. Loading the environment variable

load\_dotenv()

# 01. Grab token

hf\_token = os.getenv(“HF\_TOKEN”)  
if not hf\_token:  
raise RuntimeError(“Hugging Face token not found! Please set HF\_TOKEN in your .env file.”)

# 02. (Optional) Cache it locally so you can also use CLI-backed commands

login(token=hf\_token)

# 1.Create a quantization config

from transformers import BitsAndBytesConfig  
quantization\_config = BitsAndBytesConfig(load\_in\_4bit=True,  
bnb\_4bit\_compute\_dtype=torch.float16)

# if use\_quantization\_config:

# quantization\_config = BitsAndBytesConfig(

# load\_in\_4bit=True,

# bnb\_4bit\_compute\_dtype=torch.float16,

# )

# else:

# quantization\_config = None

# flash attention 2 - faster attention mechanism

# Flash attention requires a GPU capability 8.0+

if (is\_flash\_attn\_2\_available()) and (torch\_cuda.get\_device\_capability(0)[0] \>=8):  
attn\_implementation = “flash\_attention\_2”  
else:  
attn\_implementation = “sdpa” # scaled dot product attention

# 2. Pick a model

model\_id = “google/gemma-2-2b-it”  
model\_id = model\_id

# 3. Instantiate tokenizer (tokenizer turns text into tokens)

tokenizer = AutoTokenizer.from\_pretrained(pretrained\_model\_name\_or\_path=model\_id,  
token=hf\_token,)

# 4 Instantiate the model

llm\_model = AutoModelForCausalLM.from\_pretrained(  
pretrained\_model\_name\_or\_path=model\_id,  
device\_map=“auto”,  
token=hf\_token,  
torch\_dtype=torch.float16,  
quantization\_config=quantization\_config if use\_quantization\_config else None,  
# quantization\_config=quantization\_config,  
low\_cpu\_mem\_usage=True,  
attn\_implementation=attn\_implementation,  
# device\_map=“auto”,  
)

if not use\_quantization\_config:  
llm\_model.to(“cuda”)

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [April 29, 2025, 8:55am UTC](https://discuss.huggingface.co/t/error-debugging/152800/2 "2025-04-29T08:55:43Z")

</div>

Hmm… How about this?

```py
pretrained_model_name_or_path=model_id,
#device_map=“auto”,
token=hf_token,

```

> <https://stackoverflow.com/questions/79551724/fastapi-transformers-4-bit-mistral-to-is-not-supported-for-bitsandbytes>