# Inference 8 bit or 4 bit bit models on cpu?

**URL:** <https://discuss.huggingface.co/t/inference-8-bit-or-4-bit-bit-models-on-cpu/47237>\
**Category:** Beginners\
**Created:** [July 18, 2023, 5:57pm UTC](https://discuss.huggingface.co/t/inference-8-bit-or-4-bit-bit-models-on-cpu/47237 "2023-07-18T17:57:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![sd3ntato](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@sd3ntato](https://discuss.huggingface.co/u/sd3ntato)\
**Post date:** [July 18, 2023, 5:57pm UTC](https://discuss.huggingface.co/t/inference-8-bit-or-4-bit-bit-models-on-cpu/47237/1 "2023-07-18T17:57:10Z")

</div>

hello, is it possible to run inference of quantized 8 bit or 4 bit models on cpu?

---

<div class="post-metadata">

**Author:** ![YaTharThShaRma999](https://avatars.discourse-cdn.com/v4/letter/y/41988e/32.png) [@YaTharThShaRma999](https://discuss.huggingface.co/u/YaTharThShaRma999)\
**Post date:** [August 2, 2023, 9:06pm UTC](https://discuss.huggingface.co/t/inference-8-bit-or-4-bit-bit-models-on-cpu/47237/2 "2023-08-02T21:06:41Z")

</div>

I don’t believe so since the bitsandbytes library is just a wrapper around cuda functions which are for gpu.

---

<div class="post-metadata">

**Author:** ![sd3ntato](https://avatars.discourse-cdn.com/v4/letter/s/838e76/32.png) [@sd3ntato](https://discuss.huggingface.co/u/sd3ntato)\
**Post date:** [August 3, 2023, 8:14am UTC](https://discuss.huggingface.co/t/inference-8-bit-or-4-bit-bit-models-on-cpu/47237/3 "2023-08-03T08:14:03Z")

</div>

for those still searching, I found some sources about

- optimum intel for quantization on intel cpus  
[🤗 Optimum Intel](https://huggingface.co/docs/optimum/intel/index)

- core ml has quantization tools for apple cpus  
[Compressing Neural Network Weights](https://coremltools.readme.io/docs/quantization)