# How do LLMs identify generation start point during fine-tuning?

**URL:** <https://discuss.huggingface.co/t/how-do-llms-identify-generation-start-point-during-fine-tuning/105760>\
**Category:** 🤗Transformers\
**Created:** [September 7, 2024, 7:27am UTC](https://discuss.huggingface.co/t/how-do-llms-identify-generation-start-point-during-fine-tuning/105760 "2024-09-07T07:27:43Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![IAymi](https://avatars.discourse-cdn.com/v4/letter/i/5f9b8f/32.png) [@IAymi](https://discuss.huggingface.co/u/IAymi)\
**Post date:** [September 7, 2024, 7:27am UTC](https://discuss.huggingface.co/t/how-do-llms-identify-generation-start-point-during-fine-tuning/105760/1 "2024-09-07T07:27:43Z")

</div>

When fine-tuning an LLM, we pass a long string containing `context` , `user input` , and `the expected answer` . How does the model know which word to start generating from.  
Is the loss during fine-tuning not only from generating the `answer`?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [September 8, 2024, 10:34am UTC](https://discuss.huggingface.co/t/how-do-llms-identify-generation-start-point-during-fine-tuning/105760/2 "2024-09-08T10:34:59Z")

</div>

Hi,

During supervised fine-tuning (SFT), LLMs are trained to predict the next token, just like during the pre-training stage. The data consists of (instruction, completion) pairs, like [this dataset](https://huggingface.co/datasets/cognitivecomputations/dolphin) for instance. By default, one trains on all tokens (and just makes the LLM predict the next token on each of the tokens).

## Data preparation

Let’s take an (input, output) or (instruction, completion) pair as example. Let’s say we have as instruction: “give me a recipe to bake pancakes” with the completion “to bake pancakes, one takes a pan”, then you would prepare this for the model like so:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")

instruction = "give me a recipe to bake pancakes"
completion = "to bake pancakes, one takes a pan"

messages = [
    {"role": "user", "content": instruction},
    {"role": "assistant", "content": completion},
]

inputs = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
print(inputs)

```

As you will see, the instruction and completion are concatenated and some special tokens are added which indicate where the completion starts:

```bash
<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 26 Jul 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\ngive me a recipe to bake pancakes<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nto bake pancakes, one takes a pan<|eot_id|>'

```

As can be seen, the tokenizer has added a default system prompt, then the instruction (marked with a special `user` token, then the completion (marked with a special `assistant` token), all concatenated into a single string.

## Training the model

Next, we will train the LLM to predict the next token for each of the tokens. By default, one simply trains on all tokens (yes, that includes the system and user prompts). This means that the model is teached to predict `<|start_header_id|>` after `<|begin_of_text|>`, then `system` after `<|start_header_id|>`, and so on (the model is trained to predict the next token for each tokens all in parallel in one go). By default, the labels are in other words just a copy of the inputs (the model will shift the labels internally one time step to predict the next token). Visually:

 ![Screenshot 2024-09-08 at 12.28.03](https://us1.discourse-cdn.com/hellohellohello/original/3X/4/6/46e75d1112c01b3e999dbcc2a43d77ed0c7d0eb7.jpeg)

One could avoid this by using the `DataCollatorForCompletionOnlyLM` in the TRL library, which ensures that the labels are a copy of the inputs, with the system prompt and user prompt tokens replaced by -100 (the ignore index of PyTorch’s CrossEntropyLoss). This ensures that the model is only trained the predict the completion (“assistant”) tokens. Visually:

 ![Screenshot 2024-09-08 at 12.32.07](https://us1.discourse-cdn.com/hellohellohello/original/3X/2/5/25bc4d5d216f0b8b4508b94c757b138e9a05b419.jpeg)

---

<div class="post-metadata">

**Author:** ![IAymi](https://avatars.discourse-cdn.com/v4/letter/i/5f9b8f/32.png) [@IAymi](https://discuss.huggingface.co/u/IAymi)\
**Post date:** [September 8, 2024, 7:04pm UTC](https://discuss.huggingface.co/t/how-do-llms-identify-generation-start-point-during-fine-tuning/105760/3 "2024-09-08T19:04:24Z")

</div>

Does this not only ignore the calculation of PyTorch’s CrossEntropyLoss for the `prompt tokens`, but also ignore  
`<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 26 Jul 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\ngive me a recipe to bake pancakes<|eot_id|><|start_header_id|>assistant<|end_header_id|>`  
as well ?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [September 8, 2024, 7:40pm UTC](https://discuss.huggingface.co/t/how-do-llms-identify-generation-start-point-during-fine-tuning/105760/4 "2024-09-08T19:40:45Z")

</div>

That is a good question. I think for that we would need to look into the actual code of DataCollatorForCompletionOnlyLM, which is defined [here](https://github.com/huggingface/trl/blob/8a518ee619cac3e1644b7acf52cb52fb9c94e5fa/trl/trainer/utils.py#L81). As can be seen, the class takes a response\_template as input, where you can indicate where the response (completion) starts.

Of course, you could also create the labels yourself from the `input_ids`, and making sure they are set to -100 for any input id you don’t want to get predicted.

---

<div class="post-metadata">

**Author:** ![IAymi](https://avatars.discourse-cdn.com/v4/letter/i/5f9b8f/32.png) [@IAymi](https://discuss.huggingface.co/u/IAymi)\
**Post date:** [September 9, 2024, 3:50am UTC](https://discuss.huggingface.co/t/how-do-llms-identify-generation-start-point-during-fine-tuning/105760/5 "2024-09-09T03:50:23Z")

</div>

Thank you, I now understand how the `DataCollatorForCompletionOnlyLM` works. It marks the start of the response and simply ignores tokens before that point.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [September 9, 2024, 3:50pm UTC](https://discuss.huggingface.co/t/how-do-llms-identify-generation-start-point-during-fine-tuning/105760/6 "2024-09-09T15:50:50Z")

</div>

Closed on September 9, 2024, 3:50pm UTC

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.
