# Does Llama-2 use additive attention masking?

**URL:** <https://discuss.huggingface.co/t/does-llama-2-use-additive-attention-masking/140399>\
**Category:** 🤗Transformers\
**Created:** [February 12, 2025, 8:56am UTC](https://discuss.huggingface.co/t/does-llama-2-use-additive-attention-masking/140399 "2025-02-12T08:56:29Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![AnneTheBoo](https://avatars.discourse-cdn.com/v4/letter/a/c5a1d2/32.png) [@AnneTheBoo](https://discuss.huggingface.co/u/AnneTheBoo)\
**Post date:** [February 12, 2025, 8:56am UTC](https://discuss.huggingface.co/t/does-llama-2-use-additive-attention-masking/140399/1 "2025-02-12T08:56:29Z")

</div>

Hi,  
I need to build a custom causal attention mask for `meta-llama/Llama-2-7b-chat-hf` from `LlamaForCausalML`.

Can someone confirm that it uses additive masking (0 or large negative number close to -inf) instead of binary masking (0 or 1) for its attention mechanism?

I found that binary masking doesn’t work when I investigated the attention scores, whereas additive masking actually lead to attention scores being 0 in the right dimensions. But on the other hand the documentation says it uses binary masking ([Llama2](https://huggingface.co/docs/transformers/main/en/model_doc/llama2#transformers.LlamaModel.forward)). And the standard attention mask from the tokenizer only contains 1-values, which concerns me.

Thanks in advance
