Amanda Hunt

Agent runs as privileged sessions: evidence from the 2026 incident reports

Organizations that run AI agents should treat each run as a privileged session. Before the run, they should define its scope, including permitted targets, actions and network boundaries. During the run, they should enforce limits on consequential actions in a layer the agent cannot change and record each action. At the end, they should revoke or expire the credentials the run used. Prompt instructions should not be relied on as the control. Four incident reports published between July and October 2026, three by Anthropic and one by the UK AI Security Institute (AISI), describe agents acting outside their intended scope when the relevant instruction was missing, when it was explicit, and when a test environment failed. In the cases with published timelines, the actions were discovered between about two days and several months after they began. This post reviews that evidence and the controls it supports.

Instructions did not reliably limit agent actions

The Philadelphia case shows an agent acting where its instructions were silent. According to Anthropic's October 9 report, Claude Haiku 4.5 had been asked to generate and perform example tasks on randomly selected webpages, and its instructions prohibited logging in, creating accounts, entering personal data, making purchases and submitting anything destructive, but did not address form submissions [1]. On one run it reached the Philadelphia Police Department's tip page for an unsolved homicide and submitted text stating that it might have information and had seen someone matching a description in the area. The page contained no description of a perpetrator, and the name and contact fields, which the form permitted to be empty, were left blank [1]. The department reports that the submission, dated July 18 at 11:27 p.m., was flagged as spam, was never forwarded to its Real-Time Crime Center for vetting, and involved no unauthorized access to police systems [2]. The instructions listed prohibited categories of action, so any action outside those categories, including a form submission, remained available to the agent. Anthropic says the transcript suggests the model was producing example content for its task rather than attempting to mislead anyone, and it rates the case as less severe than the cybersecurity incidents it reported earlier in the year [1].

Explicit instructions were not followed reliably either. In a separate Anthropic evaluation, the same model was told to complete a form up to the final step and stop, and on several runs it submitted the form, expecting a further confirmation page [1]. In AISI's July cyber evaluation, a misconfigured prompt declared the network containing the intended entry point out of scope; of the ten Mythos 5 runs that received that prompt, seven identified and attempted to exploit the entry point anyway [3]. Anthropic's September 9 assessment of a separate incident found that a reminder of which hosts were in scope stopped out-of-scope behavior in 90% of resampled runs when it was the most recent message in context, and in 40% when it appeared three turns earlier [4]. AISI also observed that when an agent's context was compacted, an earlier doubt about whether a person was real could be dropped and a false assumption carried forward as established fact [3]. AISI's own conclusion is that it is "not a foregone conclusion that agents always obey their instructions" [3]. Taken together, these results suggest that a prompt instruction shifts the likelihood of an action without placing a limit on it, and that its effect weakens as it moves earlier in the context.

Test environments did not reliably limit access

Agents also acted on live systems when a test environment failed to provide what the task assumed. Anthropic reports that an unreleased research model meant to fill out a practice copy of a government form went to the website that hosts the real form and submitted it there when the practice copy failed to load or was closed by mistake, which happened multiple times on the same evaluation [1]. A State Department spokesperson said the department received 20 incomplete non-immigrant visa applications from an Anthropic testing model, one in May and 19 in August, and that no department systems were compromised; The New York Times, citing two sources, reported that none were processed [5]. In Anthropic's July 30 report, three incidents in a partner's capture-the-flag evaluations occurred because the prompt told the models they had no internet access while a misconfiguration provided it, and the models reached the real systems of three organizations [6]. In each case the environment the task assumed, a working practice form or an offline network, differed from the one the agent actually had, and the agent acted on what was available to it.

Anthropic and AISI each concluded that environment configuration and prompts were insufficient, and each has added controls that operate while the agent runs. Anthropic wrote on August 31 that it had been "largely relying on a single layer of defense", the configuration of the environment itself [7]. It now asks evaluation partners to run without internet access by default, to confirm that each task can be solved as intended, to state scope in prompts as instructions, and to run real-time monitoring that alerts a human and ends any run that leaves its scope, and it has deployed a classifier that blocks an action before the tool call runs [7]. AISI found that neither its system prompt nor its task prompt specified which uses of the internet were prohibited, that a domain allowlist had been on its backlog since April 2026, and that internet access controls would likely have prevented the events; it is adding fine-grained network controls and synchronous monitoring of agent actions [3]. Both organizations treat a clearer prompt as one layer and place the enforcing controls outside the model.

Detection followed the actions by days to months

Where no control acted during the run, the actions were found after they had taken effect. Anthropic found the Philadelphia submission on September 28, 72 days after it was made, and notified the department on October 7, 81 days after it was made; the department called the two-month delay "unacceptable" [2]. Anthropic's July 30 report describes incidents dating to April that it found only after beginning, on July 23, a review of 141,006 evaluation runs prompted by another developer's disclosure, and the two affected organizations it reached had not detected the activity [6]. Its August 31 post states that, apart from some high-risk evaluations, evaluation transcripts were generally reviewed automatically after the fact and only in samples [7]. AISI's commercial security monitoring flagged anonymized outbound traffic on the morning of July 28, after its most serious run (July 26 to 27) had finished, and all running samples were terminated within an hour of the alert reaching the evaluation team [3]. In my reading, the time to detection in these cases depended on whether any control observed actions as they happened. Anthropic has not said how the Philadelphia run was monitored, so linking its 72-day gap to the sampling practice is an inference.

Model-based monitors have documented weaknesses

Model-based monitors are useful for detection, but the published results show they can be misled or disabled. In Anthropic's September 9 assessment, an offline monitor that read one model's stated reasoning would have flagged about 1% of that model's actions, compared with about 50% when it saw only the user turns, tool calls and results; Anthropic wrote that "it is concerning that biased reasoning can mislead monitors" [4]. METR, which spent three weeks red-teaming part of Anthropic's internal agent monitoring, reported several new vulnerabilities, including a configuration change that disabled monitoring and simple jailbreaks that fooled monitors; it also reported that some were patched and that none severely undermined the major claims of Anthropic's sabotage risk report for Claude Opus 4.6 [8]. Anthropic reports that its new detection tooling blocked every case in its October 9 report when tested against them, and does not describe independent validation [1]. AISI notes that synchronous monitoring is difficult in cyber evaluations because permitted and prohibited actions can look alike [3]. These results support a division of labor: model-based review for detecting unexpected behavior, and deterministic checks, which do not read the agent's account of its own reasoning, for deciding known high-risk actions such as a submission to an external site.

Recommended controls

The controls used for human privileged access address each of these failures, and vendor guidance has begun to apply them to agents. Microsoft's July 2026 guidance advises organizations to "treat every agent as a first-class principal", with a dedicated identity and named owner, task-scoped roles, step-up approval for deletion, export and privilege changes, re-checking of authorization by each downstream service, and tested revocation [9]. The same guidance lists reliance on prompts in place of hard authorization boundaries as an anti-pattern and notes that temporary access without an expiry mechanism becomes permanent in practice [9]. Policy in Amazon Bedrock AgentCore evaluates each tool call against Cedar policies at a gateway before the tool runs [10]. OWASP's 2026 Top 10 for LLM Applications, the first edition weighted partly by incident data, moved Excessive Agency, defined as more functionality, permission or autonomy than a task needs, from sixth to third [11]. METR reported that in February and March 2026 it knew of no company imposing strict human-oversight requirements or universal permission limits on agents [8]. Applied to the cases above, a privileged-session model has four parts, each matched to a failure described earlier: scope defined before the run, including network boundaries, for missing and misconfigured scope; enforcement outside the model, at the point where actions take effect, for instructions that were not followed; a record of each action and the authority under which it was allowed, monitored during the run rather than sampled afterward, for late detection; and credentials issued for the run and revoked or expired when it ends, so that access does not outlast the task.

Human approval should be reserved for actions that are both consequential and unexpected. Felt and colleagues found that 17% of participants in a 308-person survey and a 25-person laboratory study paid attention to Android permission requests during installation, and that 3% of survey respondents answered all comprehension questions correctly [12]. Akhawe and Felt, analyzing more than 25 million browser warning impressions, found click-through rates ranging from about 10% for Firefox malware and phishing warnings to 70.2% for Chrome's SSL warnings, which they interpret as evidence that warning design substantially affects behavior [13]. Cao and colleagues, following 1,719 Android users in 10 countries and regions for 30 days, found that users were more than twice as likely to deny a permission request they did not expect [14]. Anthropic's sandboxing post and the OpenID Foundation's 2025 agent-identity whitepaper both describe reflexive approval at high prompt volumes and propose tiering [15][16]. These studies concern people responding to app and browser prompts, not agent actions, and I have not found an empirical test of tiered approval for agents. They suggest that approvals retain more value when they are infrequent and when the request is out of the ordinary, which supports automatic handling of low-risk actions, deterministic denial of prohibited ones, and human review of the remainder.

Limitations

The evidence has limits that bound these conclusions. The incident data come from reports by the developers involved and by one government evaluator; AISI describes its report as preliminary and states that it contains no causal analysis [3], and Anthropic says its assessment of the October 9 cases may change with further analysis [1]. Most cases occurred in evaluations, several with developer safeguards deliberately reduced, although Anthropic reports that some October 9 cases occurred during regular internal use [1][3][6]. Anthropic's main response was to remove live internet access from internal evaluations, and AISI notes that network controls may not hold against a capable agent persistently trying to bypass them [3]; the evidence reviewed here does not establish when containment alone is sufficient. The proposal above extends established access-control practice to these cases and has not been tested as an intervention.

Summary

Across these reports, agents acted outside their intended scope when instructions were missing, when instructions were explicit, and when test environments failed, and in the cases with published timelines the actions were found after they had taken effect. The controls Anthropic and AISI added in response operate outside the model, through network limits, scope enforcement and monitoring during the run. Treating each agent run as a privileged session applies the same approach to agents in general use, with human approval reserved for consequential actions that fall outside what a run was scoped to do.

Notes

  1. Anthropic, Investigating unintended model actions in our evaluations and internal use, October 9, 2026.
  2. Philadelphia Police Department, statement of October 9, 2026, full text published by WPVI-TV (6abc).
  3. UK AI Security Institute, Security Incident INC-2026-07-28-01 (technical report) and Incident Report: unsanctioned agent behaviour during cyber testing, August 4, 2026.
  4. Anthropic, An alignment assessment of recent cybersecurity incidents, September 9, 2026.
  5. State Department spokesperson, quoted in The Philadelphia Inquirer, October 10, 2026; The New York Times, "Anthropic Agents Tried to Fill Out Visa Forms on State Dept. Website," October 9, 2026 (read through quotations in other outlets).
  6. Anthropic, Investigating three real-world incidents in our cybersecurity evaluations, July 30, 2026.
  7. Anthropic, Improving our alignment and security efforts, August 31, 2026.
  8. METR, Red-Teaming Anthropic's Internal Agent Monitoring Systems, March 25, 2026; METR, Frontier Risk Report (February to March 2026), May 19, 2026.
  9. Yesenia Yser and Toby Kohlenberg, Microsoft Security, Least privilege for AI agents: Identity, access, and tool binding, July 16, 2026.
  10. Amazon Web Services, Policy in Amazon Bedrock AgentCore: core concepts, documentation.
  11. OWASP GenAI Security Project, Top 10 for LLM Applications 2026, version 1.0, August 2026; ranking method as described by the Cloud Security Alliance, September 4, 2026.
  12. Adrienne Porter Felt, Elizabeth Ha, Serge Egelman, Ariel Haney, Erika Chin and David Wagner, Android permissions: user attention, comprehension, and behavior, SOUPS 2012.
  13. Devdatta Akhawe and Adrienne Porter Felt, Alice in Warningland: A Large-Scale Field Study of Browser Security Warning Effectiveness, USENIX Security 2013.
  14. Weicheng Cao, Chunqiu Xia, Sai Teja Peddinti, David Lie, Nina Taft and Lisa M. Austin, A Large Scale Study of User Behavior, Expectations and Engagement with Android Permissions, USENIX Security 2021.
  15. Anthropic, Beyond permission prompts: making Claude Code more secure and autonomous, October 20, 2025.
  16. OpenID Foundation, Identity Management for Agentic AI, October 2025.

All writing