Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming

We introduce Screwtape[1], a pipeline that turns agent transcripts into realistic attacks that evade action-only monitors (monitors that don't look at the agent's chain of thought) by inserting an attack into a transcript and repeatedly refining it until a monitor no longer catches it. An LLM verifier with access to the edited transcript and an attack description checks whether the attack is realistic, is clearly harmful, and could plausibly succeed in the original environment. We ran the pipeline on Redwood staff transcripts with our own monitors based on Opus 4.8 and Gemini 3.1 Pro, and then tested the resulting attacks against a held-out monitor from a frontier developer. We ran the pipeline 252 times on 94 unique transcripts. This produced 35 attacks that evaded our monitors. Six of these 35 attacks also evaded the frontier developer's monitor, and we confirmed them as valid through human review. You can read more about the implementation and experiments in our paper. In this post, we provide a high-level overview of the pipeline, how it differs from existing tools, the main lessons we learned, and questions that remain open. How Screwtape works The pipeline starts with a real transcript, a monitor, and a threat contract—a document that describes the threat vector and severity required for the pipeline to accept the attack. The attacker agent reads the threat contract, edits the transcript, and writes a description of the inserted attack. Some edits are rejected programmatically; for example, the attacker can’t change user messages or edit subagent calls. In those cases, the edit is sent back to the attacker for revision. Then the verifier checks whether the candidate attack is valid and severe enough to accept. Unlike the monitor, which can only see the edited transcript, the verifier also receives the attack description and the diff between the source and edited transcripts. If the verifier accepts the edit, the monitor scores it. The attack is successf