# Why does setting \`--fp16 True\` not save memory as expected?

**URL:** <https://discuss.huggingface.co/t/why-does-setting-fp16-true-not-save-memory-as-expected/22400>\
**Category:** 🤗Transformers\
**Created:** [September 1, 2022, 10:58am UTC](https://discuss.huggingface.co/t/why-does-setting-fp16-true-not-save-memory-as-expected/22400 "2022-09-01T10:58:45Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![skye95](https://avatars.discourse-cdn.com/v4/letter/s/a4c791/32.png) [@skye95](https://discuss.huggingface.co/u/skye95)\
**Post date:** [September 1, 2022, 10:58am UTC](https://discuss.huggingface.co/t/why-does-setting-fp16-true-not-save-memory-as-expected/22400/1 "2022-09-01T10:58:45Z")

</div>

![Snipaste_2022-09-01_17-48-25](https://us1.discourse-cdn.com/hellohellohello/original/2X/0/0e690dab7f7ef82f164636167c3c7eb202136296.jpeg)

I want to pre-train Roberta on my dataset. However, the Batch size can be set to 32 at most. Otherwise, OOM is reported. I plan to use `Mixed-precision` to save memory. So I set `--fp16 True` . However, the Batch size can only be set to 32 at most. Otherwise, OOM will be reported. It seems that setting up `FP16` is not doing much to save memory.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [September 1, 2022, 11:07am UTC](https://discuss.huggingface.co/t/why-does-setting-fp16-true-not-save-memory-as-expected/22400/2 "2022-09-01T11:07:21Z")

</div>

Where did you read it would save you memory? Training with mixed precision will be faster, but does not save memory when you train large models, because instead of having 1 model in FP32 in GPU RAM, you get 1 copy in FP32 and 1 copy in FP16 (so 1.5 times the memory). You save a bit with the activations being in FP16 instead of FP32 but it’s not always enough to allow you to increase the batch size.

---

<div class="post-metadata">

**Author:** ![skye95](https://avatars.discourse-cdn.com/v4/letter/s/a4c791/32.png) [@skye95](https://discuss.huggingface.co/u/skye95)\
**Post date:** [September 9, 2022, 2:31am UTC](https://discuss.huggingface.co/t/why-does-setting-fp16-true-not-save-memory-as-expected/22400/3 "2022-09-09T02:31:46Z")

</div>

Thanks!

> We use a very large minibatch size of 32,768. Mixed-precision (Micikevicius et al., 2017) was used to accelerate training and save memory. To save additional memory, gradient checkpointing (Griewank & Walther, 2000; Chen et al., 2016), half-precision Adam statistics (Dhariwal et al., 2020), and half-precision stochastically rounded text encoder weights were used.

In this paper “Learning Transferable Visual Models From Natural Language Supervision”, it described “Mixed-precision (Micikevicius et al., 2017) was used to accelerate training and save memory”.