24 August 2026
Why an instruction to be critical stops holding
An instruction to push back is a constraint the model has to keep choosing. Three turns in, the conversation itself outweighs it, and nothing tells you.
7 minutes to read
The most common answer to an assistant that agrees with everything is to tell it not to. Put it in the system prompt. Be critical. Challenge my assumptions. Do not simply agree with me. It works, and then it stops working, and the point at which it stops is not announced.
This post is about why. It is not an argument that instructions are useless. It is an argument that an instruction is a constraint the model has to keep choosing to honour, turn after turn, against everything that has been said since, and that the odds shift against it every time you speak.
The name for that failure on this site is instruction decay: an instruction that is still in the context and still being read, and that has stopped deciding what the model says. The three effects below are that one mechanism taken apart rather than three separate mechanisms, which is why the three repairs set out later in this post all fail in the same place.
An instruction is a fixed weight in a growing room
A system instruction is a short piece of text at the front of the context. It stays exactly the same length for the whole conversation. Everything else grows: your messages, the model's own replies, the documents you paste in, the corrections you make.
By the fourth exchange the instruction is a small part of what the model is conditioning on, and the largest part is a conversation in which two participants have been broadly cooperative. Nothing about that is a bug. It is what a language model is for. The instruction has not been forgotten. It has been outvoted.
The model is also arguing with itself
There is a second effect and it is the one people miss. The model's own earlier turns are in the context too.
Suppose it accepted your revenue assumption in turn two, because at that point it had no reason to question it. In turn four, disagreeing with that assumption now requires contradicting itself as well as contradicting you. Coherence with what has already been said is one of the strongest pressures acting on the next token. The instruction says challenge. The transcript says you already agreed with this.
The result is that a critique gets cheaper to write and less costly to act on with every turn. It keeps its shape. It stops having teeth.
You are its correction, and correction is one directional
The third effect is you. When the model raises an objection that is wrong, or annoying, or badly timed, you push back. That is the correct thing to do and it is also a signal. It goes into the context and it stays there.
Nothing symmetrical happens when the model raises an objection that is weak but agreeable. You have no reason to push back on a mild critique that lets you keep your plan, so you do not, and the conversation records that as approval. Over a few turns the feedback the model receives about its own critical register is entirely negative feedback about the times it went too far.
The failure has no signature
This is what makes instruction decay worse than it sounds. A model that has stopped genuinely disagreeing with you does not start writing differently. It still opens with a caveat. It still says that one risk here is, and that it is worth considering whether. The register survives.
What has changed is whether any of it would cost you anything to accept. A critique that never asks you to abandon anything is a critique that has become a formatting convention, and you cannot tell from reading it, because reading it is exactly how it was designed to look fine.
What has to change for it to hold
The property an instruction is trying to buy is that somebody in the conversation is not optimising for your agreement. An instruction cannot buy it, because the thing being instructed is also the thing that would have to enforce it.
A second participant can, on one condition: it has to be optimising for something genuinely different, and its output has to be visible to the others. Two models with the same objective produce two versions of the same answer. Two models with opposed objectives produce a disagreement, and a disagreement is legible in a way a critique is not. You can see which one gave ground.
That is the position this product is built on. The personas, each optimising for a different thing, each carrying a deliberate blind spot the others can attack, in a room where exactly one of them holds the floor at a time. The argument for why disagreement between advisers carries more information than criticism from one is set out in full on the page for it, and the mechanism that stops them talking over each other is on the page beside it. This post is only the half about why the thing you would try first does not work.
What we have not measured
We have not measured our own instruction decay. Three metrics that would measure it are defined and published with their targets and with visibly empty measured columns: how often a persona simply agrees, how often two personas take opposed positions on the same claim, and how often the room surfaces an assumption the brief did not name.
Zero real human sessions have taken place, so none of the three has a value yet. When they do, they will be published whether or not they support the argument above.
Not yet produced
A figure showing how far an instruction travels before a model stops applying it. It is producible today from the written argument and it has not been drawn. The pages carry the argument as prose, so the gap costs a reader the picture and not the point.