# Error when quantization codellama 70b

**URL:** <https://discuss.huggingface.co/t/error-when-quantization-codellama-70b/92911>\
**Category:** Models\
**Created:** [June 19, 2024, 6:59am UTC](https://discuss.huggingface.co/t/error-when-quantization-codellama-70b/92911 "2024-06-19T06:59:54Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![UnauthorizedShelley](https://avatars.discourse-cdn.com/v4/letter/u/8dc957/32.png) [@UnauthorizedShelley](https://discuss.huggingface.co/u/UnauthorizedShelley)\
**Post date:** [June 19, 2024, 6:59am UTC](https://discuss.huggingface.co/t/error-when-quantization-codellama-70b/92911/1 "2024-06-19T06:59:54Z")

</div>

When I use bitsandbytes to quantize codellama 70b I occurred error:  
my code is:

```auto
MODEL_NAME = 'codellama/CodeLlama-70b-hf'
bnb_config = BitsAndBytesConfig(
       load_in_4bit=True,
       bnb_4bit_quant_type="nf4",
       bnb_4bit_compute_dtype=torch.float16,
   )

model = AutoModelForCausalLM.from_pretrained(
   MODEL_NAME,
   use_safetensors=True,
   quantization_config=bnb_config,
   trust_remote_code=True,
   device_map="auto",
)

```

---

<div class="post-metadata">

**Author:** ![UnauthorizedShelley](https://avatars.discourse-cdn.com/v4/letter/u/8dc957/32.png) [@UnauthorizedShelley](https://discuss.huggingface.co/u/UnauthorizedShelley)\
**Post date:** [June 19, 2024, 7:00am UTC](https://discuss.huggingface.co/t/error-when-quantization-codellama-70b/92911/2 "2024-06-19T07:00:31Z")

</div>

![1](https://us1.discourse-cdn.com/hellohellohello/original/3X/9/2/92de30ab0a528e02a39efd4415bd30920ab46534.png)  
And this is the error

---

<div class="post-metadata">

**Author:** ![UnauthorizedShelley](https://avatars.discourse-cdn.com/v4/letter/u/8dc957/32.png) [@UnauthorizedShelley](https://discuss.huggingface.co/u/UnauthorizedShelley)\
**Post date:** [June 19, 2024, 7:23am UTC](https://discuss.huggingface.co/t/error-when-quantization-codellama-70b/92911/3 "2024-06-19T07:23:33Z")

</div>

![2](https://us1.discourse-cdn.com/hellohellohello/original/3X/4/0/408cce8b374f6540e1814a5bb65880d3487a6a88.png)  
the other part.

---

<div class="post-metadata">

**Author:** ![RaushanTurganbay](https://avatars.discourse-cdn.com/v4/letter/r/848f3c/32.png) [@RaushanTurganbay](https://discuss.huggingface.co/u/RaushanTurganbay)\
**Post date:** [June 20, 2024, 6:54am UTC](https://discuss.huggingface.co/t/error-when-quantization-codellama-70b/92911/4 "2024-06-20T06:54:45Z")

</div>

Hi! The error msg says that you don’t have enough GPU memory to load the 70B model. To see the workaround with cpu-offloading, you can follow the link from error message 🙂