# Load\_in\_8bit requires device\_map but also does not support it

**URL:** <https://discuss.huggingface.co/t/load-in-8bit-requires-device-map-but-also-does-not-support-it/28111>\
**Category:** 🤗Transformers\
**Created:** [December 19, 2022, 12:38pm UTC](https://discuss.huggingface.co/t/load-in-8bit-requires-device-map-but-also-does-not-support-it/28111 "2022-12-19T12:38:30Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![owinton](https://avatars.discourse-cdn.com/v4/letter/o/b5e925/32.png) [@owinton](https://discuss.huggingface.co/u/owinton)\
**Post date:** [December 19, 2022, 12:38pm UTC](https://discuss.huggingface.co/t/load-in-8bit-requires-device-map-but-also-does-not-support-it/28111/1 "2022-12-19T12:38:30Z")

</div>

Hi! I am trying to optimize inference on a MarianMTModel, following the huggingface [guide](https://huggingface.co/docs/transformers/perf_infer_gpu_one#running-mixedint8-models-multi-gpu-setup) for optimizing inference on single GPU. Upon trying to run as a mixed-int8 model

```auto
model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint, load_in_8bit=True).to(device)

```

I am prompted to provide a device map

```auto
ValueError: A device map needs to be passed to run convert models into mixed-int8 format. Please run`.from_pretrained` with `device_map='auto'`

```

Adding the device\_map

```auto
model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint, device_map="auto", load_in_8bit=True).to(device)

```

results in the following

```auto
ValueError: MarianMTModel does not support `device_map='auto'` yet.

```

I have tried recreating my Conda environment from, and still get the same problems. Has anyone had similar issues?

I am using PyTorch 1.12.0, Accelerate 0.15.0, bitsandbytes 0.35.4, transformers 4.24.0
