Thinking Inertia: LLMs Keep Thinking When Told Not To
Abstract
Researchers propose multi-level metrics to evaluate LLM no-thinking behavior, finding that explicit reasoning persists despite controls and varies with task openness, establishing no-thinking as a distinct capability requiring systematic assessment.
Large Language Models (LLMs) increasingly ship with explicit "thinking modes", yet their counterpart, "no-thinking", has received far less attention. We study LLMs' no-thinking behavior along two axes. a. How to measure no-thinking? Prior work typically defines no-thinking through proxies such as a disabled thinking mode or the absence of long traces. These proxies are unreliable: disabled thinking modes may still emit reasoning, while long traces may contain filler rather than genuine inference. We instead normalize each response into a pre-answer trace and final answer, and evaluate it at three levels: (i) Empty-Thinking Rate for strict answer-only compliance; (ii) instruction-aware Question-Pre-answer Relevance for similarity between the question and pre-answer trace; and (iii) LLM-as-judge Explicit Inference Rate for visible explicit inference. Together, these metrics distinguish answer-only output, relevant but non-inferential text, and explicit inference. b. How does no-thinking vary across tasks and models? We evaluate six prompting interventions on six LLMs across Boolean, multiple-choice, and open-ended questions. We find that explicit no-think controls cannot reliably eliminate visible inference. Models instead exhibit "Thinking Inertia": explicit inference persists even under strict controls and becomes more prevalent as the answer space opens. Accuracy remains stable on Boolean and multiple-choice tasks, whereas open-ended tasks reveal a trade-off between answer-only compliance and task accuracy. Rewriting the same questions across answer spaces shows that supplying candidate answers makes answer-only responses easier to produce. These findings establish no-thinking as a non-trivial capability: stopping explicit reasoning cannot be assumed from model settings or instructions alone and deserves systematic evaluation alongside reasoning ability.
Community
Today’s LLMs are used to thinking longer. But can they stop when explicitly told NOT to?
Excited to share our #NeurIPS2026 work:
LLMs Keep Thinking When Told Not To
We observe persistent “Thinking Inertia”:
LLMs keep reasoning even when explicitly told not to. They may get the answer right, yet fail to follow the no-thinking instruction.
💡 Knowing when NOT to think may be just as important as knowing how to think.
Project Page: 🌐 https://thinking-inertia.github.io
I don't understand what this paper is getting at. Like, yeah, if you train a model to think, it'll think.
At what point in the dataset is it trained to recognize “DON'T USE REASONING TOKENS”? None... and why would it be?
Help me understand.
Guess that's the whole point of the paper -- models should also be trained to avoid overthinking and to quickly arrive at conclusions when given simple enough problems. That of course poses its own set of questions e.g. how do we even quantify how hard a task is, or how many tokens of reasoning is acceptable or expected for a task
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Detecting Hidden Chain-of-Thought in Large Language Models with Linguistic, Behavioral, and Mechanistic Indicators (2026)
- REALHOP: Rethinking Multi-Hop Reasoning Evaluation via Behavioral Auditing (2026)
- Reasoning Instructions Can Break Answer Decoding in Vision--Language Models (2026)
- Nudgeability: Reasoning Models Follow Confidence Signals Without Tracking Their Own Competence (2026)
- Evaluation Awareness in Language Models: Representation, Verbalization, and Control (2026)
- StateSwap: Probing Support-Elimination Hidden States in Multiple-Choice Questions (2026)
- Whether LLMs Can Navigate Beliefs and Facts Depends on How You Phrase It (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Thanks for sharing this work! It was interesting to see a systematic study of something we also encountered in our earlier work, Zero-CoT Probe: https://huggingface.co/papers/2605.21856. We wanted to suppress explicit CoT and elicit direct answers to probe memorization and detect data contamination.
One implementation detail that might be useful: in the released code’s chat path, the M6 answer prefix appears to be placed at the end of the user message. In our open-weight experiments, we instead placed it at the beginning of the assistant response, ending with an opening LaTeX boxed command, and let generation continue directly from there. This worked well for eliciting direct answers on both MATH and GSM8K in our setup.
It could be interesting to try this assistant-prefill variant with your models too! We’d be curious to see how well it carries over across model families.
For closed-source models such as GPT and Claude, we relied on strict answer-only prompts and found that they worked for some models but not others, so your findings on model-dependent compliance resonated with our experience.
Thanks again for exploring this direction!
The prefill results surprised me, so I took a look at the code. Would love to see more implementation details and analysis of prefilling in the paper!
Get this paper in your agent:
hf papers read 2610.11765 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper