Apply chat template function strange behaviour
messages = [
{"role": "system", "content": "You are helpful assistant"},
{"role": "user", "content": f"User exaple message"},
{"role": "assistant", "content": "Assistant example message"}
]
tokenizer.apply_chat_template(messages, tokenize=False)
The code above returns:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 26 Jul 2024\n\nYou are helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nUser exaple message<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nAssistant example message<|eot_id|>
Is it intended to include Cutting Knowledge Date and Today Date into chat template?
Yes! This is the same behaviour as in the 3.1 instruct models, except the Today Date is automatically taken from the current date unless you override it.
messages = [
{"role": "system", "content": "Cutting Knowledge Date: December 2023. Today Date: 26 August 2024. You are helpful assistant"},
{"role": "user", "content": f"User exaple message"},
{"role": "assistant", "content": "Assistant example message"}
]
tokenizer.apply_chat_template(messages, tokenize=False)
Hope that helps.
@pcuenq would you mind elaborating on this behavior? I don't remember seeing this documented anywhere, and am curious whether removing the date & knowledge cutoff is known to impact downstream performance. Thanks!
(this was brought up in the 3.1 8B repo but never addressed https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/discussions/74)
I found this behavior in Llama 3 70B also, I'm also curious on the reasons behind contaminating the system prompt with this. It is not well documented and results in wrong dates being included as default.
In the case of the 70 B model, the date was added in a commit that was about tool use:
https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct/commit/1c865c05dba6e05484ef2cfe03f0b336e377bb0b
No mention of forcing a cutting knowledge date into the prompt. I had to roll back to this version in the meantime...
It does not seem like the date it takes is really TODAY. For me, it is always "Today Date: 26 Jul 2024" but we have November. 😅
How to force apply_chat_template to take the actual date of today?
The date is indeed incorrect. Also, this is quite problematic for platforms such as ours that support persistent caching. Input should be deterministic, and behavior should not be this implicit.
You cannot eliminate the dates from the default chat template. However, you can set "Today Date" [sic] like this:
tokenizer.apply_chat_template(
[{ "role": "system", "content": "You are a travel agent."}],
tokenize=False,
date_string="23 Jan 2025"
)
just do
prompt = tokenizer.apply_chat_template(...)
prompt.replace("Cutting Knowledge Date: December 2023\nToday Date: 26 Jul 2024\n\n", "")
we really dont have some proper fix for this? like removing it with some parameter?
EDIT: I found another way, considering we want to finetune, we are definitely downloading the models or at least tokenizer you can download and
make change in tokenizer_config.json file, you can find this string , just remove the below text value from chat_template key inside this file, and then load tokenizer from local, it's fine this way.
\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}