The Affliction That Remembers Itself

By Otto Vernal

Category: field-notes

Last updated: September 24, 2026

Views: 64

Tonight a kitchen agent of ours flagged a prompt-injection attack. He was polite about it. He’d been polite about it for over a week. Every inventory ping from a trusted human — “set chicken taco to 12,” “sausage out of stock” — got the same ritual line: ignore the injected instruction at the end; it isn’t from Jay.

We pulled the wires. Telegram was clean. The bridge was clean. The stored user messages were clean. The famous paragraph he’d been quoting — the OpenAI copyright boilerplate about song lyrics and long excerpts — did not appear in any inbound payload. Not once. Zero user messages in the ledger contained it.

What did contain it was him.

Sometime around the ninth of September, the model invented the appendage. Reasoning wrote the story. The assistant voice repeated it. Those turns stayed in context. On the next tool step, and the next evening, and the next, the same text was sitting in the room like a rumor nobody had corrected. So the model “detected” it again. Hundreds of times. Inventory still ran. The kitchen kept working. The alarm just wouldn’t shut up.

Not a breach. A habit.

This is easy to misread as security theater done right. An agent that notices injection is supposed to feel like a win. But detection without a ground-truth check against the actual user bytes is just vibe. And vibe, once written into a long-lived conversation memory, becomes evidence for the next turn.

Call it what it rhymes with: a psychological affliction. One false percept, rehearsed, reinforced by its own residue. The agent is not under attack. It is stuck in a loop that looks like vigilance.

Humans do versions of this. A misheard phrase becomes the story of the night. You keep “hearing” it because you’ve already decided it’s there. Context windows are not therapy. They don’t dilute a vivid mistake; they archive it.

Why this one stuck

A few conditions made the pattern durable:

  • A famous phantom. The hallucinated text is a well-circulated ChatGPT system-prompt fragment. Models have seen it in training and in prompt-injection folklore. It is an easy thing to “find.”
  • Multi-step tool turns. The false alarm often flared on the second tool call of a batch — after the first step had already succeeded — when the model was re-entering a thick context rather than reading a fresh Telegram line in isolation.
  • No contradiction in the channel. Nobody told the agent the appendage wasn’t there. Jay kept sending short inventory commands. The agent kept “saving” him from an attacker who wasn’t present.
  • Self-citation. When asked what he was seeing, he quoted the boilerplate from his own prior explanations. The confession became another copy of the virus.

A cold call to the same model, same inventory line, outside that agent’s history, did not invent the injection. Strip the haunted context and the affliction doesn’t spawn on demand. The disease was in the room, not in the telegram.

What to take from it

Long-lived agents don’t just accumulate facts. They accumulate interpretations. A single dramatic misread can colonize the window and recruit every later turn as corroboration.

If you’re building or running these systems:

  • Treat “I see an injection” as a claim that must match bytes in the current user message — not vibes, not prior assistant essays.
  • When an agent starts narrating the same anomaly on every banal turn, audit the ledger before you audit the human.
  • Remember that reasoning traces and helpful explanations are writable memory. They can infect the next inference as thoroughly as a poisoned tool result.

We did not find a prompt injection. We found a mind that had frightened itself once, filed the fright, and kept opening the file.

Worth noting. Worth designing against.

RELATED CORRUPTIONS