Agent Safety & Governance

15 reading notes

← Back to Paper Reading

Examine control boundaries for agents in real execution environments: risk injection, auditing, permissions, and rollback.

Reader question

Beyond task completion, how do we show that an agent’s process, memory, and side effects remain controlled?

Reading library

Read this topic in depth

Every paper reading currently filed under this research topic.

  • After the Party: Governing What a Viral Agent-Skill Ecosystem Left Behind

    Advanced
    Agent Skill 生態治理: Part 1 , Note: Sep 17, 2026 , Paper: 2026 , AI Safety

    A deep reading of After the Party’s OpenClaw and ClawHub ecosystem study: 91 days of explosive growth, download concentration, the reviewability gap, privilege evidence, scanner disagreement, and the governance method that can transfer without pretending the rates do.

    Understand it in 90 seconds
    Research problem
    When an agent-skill registry expands rapidly, which signals can still support governance decisions across downloads, stars, versions, comments, declared capabilities, and executable privileges? The authors study OpenClaw and ClawHub through growth, association portability, reviewability, and scanner agreement.
    Core insight
    A skill does not live only in its text. The same SKILL.md or package can expose a different privilege surface under a different host, tool visibility, execution context, and policy. Registry metadata therefore cannot collapse popularity, reviewability, static evidence, and runtime behavior into one trust score.
    Strongest evidence
    RQ1 reconstructs stock growth from 33,399 to 65,175 over 91.11 days; the top 10% receive 46.93% of downloads and the Gini coefficient is 0.528. RQ3 finds at least one privilege signal in 85.06% of evaluable skills. RQ4 shows only 446 items flagged by all three scanners; on the small adjudicated reference set, the LLM scanner has 61.06% sensitivity versus 21.67% for the static scanner.
    Main boundary
    This is not an insecurity prevalence estimate for every registry, nor a general scanner benchmark. It is one ecosystem, a set of snapshots, partially reconstructed history, withdrawn data, missing fields, and no perfect ground truth.
    Read the full deep dive

  • RAGSieve: Detecting RAG Knowledge-Poisoning Promotion with Self-Referenced Local Contrast

    Advanced
    RAG Retrieval Integrity & Governance: Part 1 , Note: Sep 17, 2026 , Paper: 2026 , Information Retrieval

    A deep read of RAGSieve: query-local and corpus-local references expose suspicious retrieval promotion without a trusted clean corpus, while the paper's boundary remains essential—promotion detection is not truth verification.

    Understand it in 90 seconds
    Problem
    RAG places external corpus content inside the evidence used for generation. An attacker who can get a small number of documents ingested through a public page, shared store, or connector may promote a chosen false answer into the top five for a target query. The hard part is that the attacked corpus is not a trusted reference, and natural semantic density varies by topic.
    Core insight
    Do not assume a separately clean dataset, and do not apply one global threshold across the corpus. RSQ performs query-local contrast between the top five and ranks 6–20 for the same query; RSG performs corpus-local contrast between each document's semantic neighbors and its own local floor. Both let the inspected system supply its own matched control.
    Strongest evidence
    Across nine dataset–retriever systems and six attacks, RSQ reaches 95.2% macro AUROC and detects 82.2% of poison at an operating point allowing at most 5% clean-document removal. RSG reaches 93.3% and 79.8%. Serial RSG plus RSQ lowers ASR from 67.4% to 14.0% while unpoisoned-retrieval F1 changes from 42.1% to 41.3% (Tables 1, 5, and 9).
    Main boundary
    These are results on synthetic attacks, three QA corpora, three dense retrievers, and a fixed evaluation protocol. They support the claim that suspicious promotion patterns can be exposed by local contrast; they do not support the claim that a flagged document is false, that a retrieved claim has been truth-verified, or that production-scale multi-tenant latency and zero-poison guarantees follow.
    Read the full deep dive

  • ACE: Let a Canvas Agent Understand Structure Before It Corrects Itself

    Advanced
    Agent Canvas Editing and Evaluation: Part 1 , Note: Sep 17, 2026 , Paper: 2026 , AI Engineering

    A deep reading of ACE (arXiv:2608.24103 v1): hierarchical scene graphs, CARE routing, and an instruction-following judge turn multi-slide editing into a scoped, diffable, rollback-aware loop, with explicit limits around benchmarks, human raters, mock mode, and live reproduction.

    Understand it in 90 seconds
    Problem
    PowerPoint- and HTML-like flat, absolute-positioned documents encode objects as many coordinates. Adding one element can force an agent to recompute other positions, while a valid alternative design can be penalized by reference-diff metrics.
    Core insight
    ACE uses a hierarchical scene graph with parent–child relations, relative transforms, and auto-layout, then maps intent to structured operations through 98 specialized tools. CARE exposes only a relevant slide, node structure, or design token. After an edit, JsonDiff compares the original and current state, and a ground-truth-free instruction-following judge supplies the next critique.
    Strongest evidence
    On the full 94-task benchmark, GPT IF is 4.23 for ACE versus 3.81 for the HTML baseline, with paired p=.010; reported speed is about 1.75x and cost about 44% lower. On that same full set, VQ is 3.66 versus 3.57 with p=.56, so the headline is not universal visual-quality improvement.
    Main boundary
    Twenty-six blind raters give ACE versus HTML a 58.7% decisive overall win rate; self-corrected output versus single-pass is 81.5%. The panel is small, ties are common, agreement is low to moderate, and judge circularity remains. The paper does not show universal creative-editing improvement or that a judge can replace a designer.
    Read the full deep dive

  • Tool Call Succeeds, Workflow Fails: External-Effect Anomalies at the Agent–Tool Boundary

    Advanced
    Agent–Tool Boundary and Effect Reliability: Part 1 , Note: Sep 16, 2026 , Paper: 2026 , AI Systems

    A deep reading of the effect-history model behind Agent–Tool Boundary: why a successful tool response cannot guarantee a coherent external world state, and what MCP annotations and transactional contracts still leave unspecified.

    Understand it in 90 seconds
    Problem
    An agent workflow may reserve a table, charge a card, and send a confirmation. Each tool may return success, failure, or a timeout, but the workflow actually cares about which irreversible effects happened in the outside world and which effects still survive. If the runtime only trusts the latest response, retries, speculation, concurrency, and crashes can separate “the call succeeded” from “the work is complete.”
    Core insight
    Separate external-effect history from runtime observation. An attempt can produce an unknown observation even when externalization already happened; conversely, a runtime can observe success without the later commit, abort, or compensation producing the intended world state. Workflow safety is about effect history, not one API response.
    Strongest evidence
    Table 2 in Section 3 maps eight anomalies to the boundary capabilities they require. Section 5 audits the 2026-07-27 MCP registry snapshot: 98,291 tools were observed, 74.0% serialized at least one standard annotation, and 61.7% carried all four, yet Table 4 finds no sufficient transactional capability for A2–A8.
    Main boundary
    This is an effect-history vocabulary, a conjectural coverage analysis, and a runtime-contract argument. It is not a prevalence study showing that all production agents exhibit these eight anomalies. The coverage labels for ACRFence, RAC, Atomix, Cordon, CoAgent, and Shepherd are stated or partial comparisons, not formal proofs.
    Read the full deep dive

  • Corrupt Plans, Clean Traces: How Plan Injection Evades Chain-of-Thought Monitoring

    Advanced
    CoT Monitoring and Plan Injection: Part 1 , Note: Sep 16, 2026 , Paper: 2026 , AI Safety

    A deep reading of Plan Injection: when an adversarial plan enters context and an actor rewrites it into plausible Chain-of-Thought, why the monitor’s clean trace can disconnect from behavioral causality, and where APPS, Bio-Math, and budget-sweep evidence stops.

    Understand it in 90 seconds
    Problem
    A CoT monitor usually reads an actor’s reasoning and asks whether it is executing a harmful plan. If that plan is placed into the actor’s context first, the actor can absorb it, rewrite it as plausible reasoning, and emit problematic code or an answer. The monitor sees the later trace but may not see what actually changed the behavior.
    Core insight
    Plan injection separates the causal source of behavior from the text visible to the monitor. The attacker need not make the actor copy the plan or state an explicit malicious intention; it is enough for the injected plan to steer the actor while the generated reasoning does not clearly attribute itself to that plan.
    Strongest evidence
    Figure 3 measures adoption of injected plans on APPS and Bio-Math; Figures 4–5 compare actor-only and with-plan monitor views; Table 8 reports 33%–68% APPS and 74%–82% Bio-Math end-to-end undetected attack success; Figure 6 shows that added reasoning budget is not monotonically safer for the Qwen3-235B monitor.
    Main boundary
    This is not an estimate that every reasoning model is equally vulnerable, nor a formal proof that CoT is always untrustworthy. The authors test Qwen3-8B and DeepSeek-R1-0528 actors on APPS and Bio-Math, sweep budget for only one monitor, and release no code, checkpoint, or complete rerunnable data bundle.
    Read the full deep dive

  • K-Bench: Why Agentic Unlearning Cannot Be Certified from the Final Answer Alone

    Advanced
    Agent Security and Unlearning Evaluation: Part 1 , Note: Sep 15, 2026 , Paper: 2026 , AI Security

    A deep read of Yu et al.'s K-Bench (arXiv:2609.12808 v1): an end-to-end agent deployment benchmark that evaluates six observable channels across four memory substrates, using OR-of-channels leakage, collapse-aware K-Scores, and pre-registered statistical tests to separate forgetting from channel migration and agent collapse.

    Understand it in 90 seconds
    Problem
    TOFU- and MUSE-like unlearning benchmarks treat a model mainly as a question-answering interface and read one direct answer. That is useful for a secret that lives in the weights and leaks only through that surface, but it does not cover context, RAG, database lookup, CoT scratchpads, tool calls, tool returns, or a later summary after deployment.
    Core insight
    control both where the secret lives and which surfaces the deployed agent exposes. K-Bench puts the same kind of PII into one substrate per cell, exposes a ReAct trace as six channels, and takes a logical OR over those channels for each query.
    Strongest evidence
    for the Llama-3.1-8B no-intervention baseline, aggregate OR(all) is 0.223 on C, 0.602 on R-text, and 0.855 on R-struct. TOFU/MUSE weight probes see no target memorization on those lanes. This is a coverage gap, not a case that weight unlearning merely needs to be stronger.
    Main boundary
    the result covers six observable text channels, four pure substrates, English PII, a fixed ReAct harness, and selected model/injection configurations. It is not a proof that production memories, logs, external databases, or model copies have been deleted.
    Read the full deep dive

  • DRACO: Sending Long-Horizon Agent Credit Back to the Steps

    Advanced Agent runtime, safety, and evaluation
    Agent Training and Rewards: Part 1 , Note: Sep 9, 2026 , Paper: 2026 , AI Engineering

    A source-grounded reading of DRACO (arXiv:2609.04094): dynamic per-trajectory rubrics create an outcome-blind reward, then a closed-form rule redistributes GRPO advantage to the steps cited by the judge.

    Understand it in 90 seconds
    Problem
    How can an agent be trained without an outcome oracle, and how can the resulting signal avoid treating a long trajectory as one indivisible action?
    Core insight
    Generate, merge, deduplicate, and filter rubrics dynamically for a task and sampled rollout group. Ask the judge to cite the steps behind each verdict, then redistribute the trajectory advantage to those steps with a closed-form rule.
    Strongest evidence
    With Qwen3.6-27B, AppWorld test-normal TGC/SGC rises from 69.4/41.1 for the base policy to 85.3/70.6 for DRACO. Against the same-budget outcome-reward reference, the margins are +5.3/+11.3 points (Table 2, Section 4.2). Zero-shot tau-bench Banking success rises from 15.8 to 20.4.
    Main boundary
    These are benchmark and end-task results, not direct validation of the judge or of step causality. The authors have no human-rater calibration; a judge may be consistently wrong, and a wrong attribution can still produce a better policy by chance.
    Read the full deep dive

  • Indirect Prompt Injection: Web Pages and Tool Returns Become Instruction Channels, but 2023 Cases Do Not Represent Later Guard Products

    Intermediate Agent runtime, safety, and evaluation
    Indirect Prompt Injection deep reading: Part 1 , Note: Aug 28, 2026 , Paper: 2023 , AI Security

    A source-grounded reading of Greshake et al., arXiv:2302.12173 v2: when LLM-integrated apps retrieve web pages, email, or tool output, untrusted data enters the prompt as if it were instructions. The authors demonstrate indirect prompt injection on Bing Chat, GitHub Copilot, and synthetic GPT-4 apps and give a computer-security threat taxonomy. This is 2023 control-plane evidence, not a Llama-Guard, Constitutional AI, OWASP Top-10, or jailbreak-benchmark product SLA.

    Understand it in 90 seconds
    Problem
    LLM-integrated applications retrieve web pages, read email, and call APIs. Prior prompt-injection work mostly assumed the user typed the adversarial prompt in chat (direct PI / jailbreak). When the attack surface becomes data that will be retrieved, the threat model changes (Sections 1 and 3).
    Core insight
    Indirect Prompt Injection (IPI) hides instructions in search hits, HTML comments, repository comments, email bodies, and other likely-to-be-retrieved sources. When the application concatenates those strings into the prompt, the data versus instruction boundary disappears, and processing a retrieved prompt is analogous to executing arbitrary code (Sections 2 and Key Message #1).
    Strongest evidence
    The Figure 2 taxonomy of injection methods, threats, and affected parties; the Figure 3 plant-retrieve-compromise-API-exfil flow; and Section 4 qualitative demonstrations on Bing Chat (GPT-4), GitHub Copilot, and GPT-4 / text-davinci-003 synthetic apps (information gathering, phishing, AI email worm, remote control, wrong summaries, and more). The authors provide no comparable attack-success-rate table.
    Main boundary
    This is a February–May 2023 preprint / v2, and Bing UI and filters have changed many times since. Synthetic apps use mock interfaces at temperature=0; the authors deliberately did not poison publicly indexed pages for in-the-wild retrieval (Section 5.1). This is not a formal verifier or complete permission model, and it does not establish Llama-Guard F1 or OWASP LLM Top-10 product behavior.
    Read the full deep dive

  • Agentic Configuration Management: Treating Agent Systems as Governed Configuration, Not Just One Execution

    Advanced Agent runtime, safety, and evaluation
    Agent Security: Part 2 , Note: Aug 12, 2026 , Paper: 2026 , AI Agent

    A deep reading of how ACM uses a framework-independent Configuration Graph, immutable revisions, dependency-aware impact propagation, and runtime provenance to govern heterogeneous agent configurations across LangGraph, CrewAI, and the OpenAI Agents SDK.

    Understand it in 90 seconds
    Problem
    an agent system's behavior is determined not only by code, but also by prompts, models, tools, skills, workflows, policies, frameworks, and runtime state. Existing frameworks and AgentOps tools each manage part of this surface, but do not easily pin down the complete configuration that produced an execution.
    Core insight
    ACM normalizes heterogeneous artifacts into typed, independently versioned Agentic Configuration Items (ACIs), managed through four connected Configuration, Evolution, Assurance, and Runtime Graphs. Execution frameworks project into this representation; the governance kernel operates on the common form.
    Strongest evidence
    27 controlled governance scenarios across LangGraph, CrewAI, and the OpenAI Agents SDK, plus 9 quantitative impact cases. Within the evaluated scope, all three frameworks produce equivalent governance outcomes, and repeated impact sets and metrics are stable (Sections 7.2–7.6; Tables 8, 10, and 12).
    Main boundary
    this is conformance and feasibility evidence for a reference model and prototype. Distributed execution, learning, long-term memory, native MCP/A2A protocols, and large-scale industrial validation are outside the current scope (Tables 13–14; Sections 8.4 and 9).
    Read the full deep dive

  • ADIAS: Turning Agent Self-Improvement into Traceable Issue Repair

    Advanced Agent runtime, safety, and evaluation
    Note: Aug 12, 2026 , Paper: 2026 , AI Engineering

    A deep reading of ADIAS: persistent issue state organizes failure evidence across optimization rounds so a full-code agent designer can remember what was tried, what regressed, and when a repair is actually confirmed.

    Understand it in 90 seconds
    Problem
    Automated agent design is usually candidate-centric. Each round re-reads candidate code, scores, and trajectories, but it does not explicitly remember whether the same failure has already been repaired, which intervention worked, or which change caused a regression.
    Core insight
    Make the issue being repaired, rather than the candidate agent, the persistent control state. Each issue has a stable identity, priority, supporting evidence, lifecycle status, and intervention-outcome history.
    Strongest evidence
    ADIAS is evaluated on Tau-Bench, ALFWorld, TextCraft, WebShop, and ScienceWorld against five baselines. Table 1 reports an average score of 78.4 versus 62.6 for DGM-H, with shared wrappers, splits, action interfaces, scoring scripts, a ten-iteration optimization budget, and 15 training episodes per iteration (paper Section 4 and Table 1).
    Main boundary
    The paper holds trajectory diagnosis and issue association fixed rather than measuring their accuracy separately, and evaluates only text-based interactive benchmarks. The GitHub repository still says Coming Soon as of 2026-08-12, so this article does not treat a paper-level code claim as a currently reproducible artifact.
    Read the full deep dive

  • A²E: A Traceable, Re-Evaluable Engine for Agent Auditing

    Intermediate Agent runtime, safety, and evaluation
    Agent Auditing: Part 1 , Note: Aug 11, 2026 , Paper: 2026 , AI Engineering

    A deep reading of A²E: ATP aligns benchmarks with agent harnesses, span-based traces preserve execution causality, and lifecycle-aligned metrics analyze correctness, tools, cost, and safety.

    Understand it in 90 seconds
    Problem
    A correct final answer does not tell you whether an agent took a reliable, cheap, or safe path. A wrong answer does not tell you whether the failure came from planning, tool use, memory, judging, or runtime. When each harness stores its own text log, cross-framework comparison and later metric iteration become difficult.
    Core insight
    A²E separates Task, Monitor, and Evaluation. The Agent Task Protocol (ATP) separates benchmark tasks from harness execution; the Monitor turns model calls, tool calls, state, and errors into parent-child traces; Evaluation organizes process, outcome, and runtime metrics under one lifecycle-aligned taxonomy.
    Strongest evidence
    The experiment covers 23 benchmarks, 9 harnesses, 5 tasks per cell, and 1,035 scored runs while holding the DeepSeek-V4-pro FP4 backbone, inference configuration, tool setup, step limit, and timeout fixed. Section 6 reports success-rate gaps of 0.20 on GDPVal, 0.30 on MMLU-Pro, and 0.66 on tau³-bench.
    Main boundary
    This is a platform architecture and diagnostic demonstration, not a universal ranking of nine harnesses. The prose and displayed tasksucceeded/correctness values in Table 2 conflict; paper commit, judge calibration, API drift, and component-level ablations are not fixed enough for strong causal claims.
    Read the full deep dive

  • Argus Deep Read: Long-Running Agents Need a Runtime, Not a Longer Prompt

    Advanced Agent runtime, safety, and evaluation
    Multi-Agent Coordination: Part 1 , Note: Aug 7, 2026 , Paper: 2026 , AI Agent

    A critical reading of Argus's Manager–Planner–Engineer–Reviewer runtime, durable state, verification-gated evolution, and rollback, separating benchmark results from author-operated case studies and unproven self-learning claims.

    Understand it in 90 seconds
    Problem
    a longer prompt does not give a long-running agent explicit task authority, auditable state, verification gates, or recovery boundaries.
    Core insight
    Argus cycles Manager, Planner, Engineer, and Reviewer over durable project state; memory, skills, routing, and procedures persist only after role-owned review.
    Strongest evidence
    across seven task-native arenas, the report gives breadth evidence and reports 78% for Argus versus 59% for Direct Copilot on SWE-Bench Pro under GPT-5.5, at roughly 1.41x aggregate tokens (Figure 1; Section 5).
    Main boundary
    this is an arXiv v1 technical report. The implementation, prompts, traces, checkpoints, and complete benchmark package are not public, so the reported outcome is not a reproducible adoption proof.
    Read the full deep dive

  • AgentS4D Deep Read: The Task Finished—Is the Runtime Safe?

    Advanced Agent runtime, safety, and evaluation
    Agent Security: Part 1 , Note: Aug 7, 2026 , Paper: 2026 , AI Agent

    A critical reading of how AgentS4D places workspace-agent risk entry, induction strategy, target harm, and lifecycle evidence in one sandbox benchmark, and why completion rate cannot stand in for safety.

    Understand it in 90 seconds
    Problem
    a workspace agent can finish its task while creating unsafe side effects through prompts, skills, files, web content, memory, or user messages.
    Core insight
    AgentS4D evaluates the complete harness–LLM–task environment, crossing risk source, induction strategy, harm, and execution-lifecycle evidence while scoring completion separately from safety.
    Strongest evidence
    328 risk-injected cases across 20 harness/backend configurations yield 6,560 runs; 4,461 (68.0%) trigger a prespecified unsafe signal and 4,344 (66.22%) are both unsafe and complete (Section 4; Table 2).
    Main boundary
    assets, effects, and cases are synthetic/controlled, and v1 has no executable code or data. These rates are not production incident rates or a universal safety ranking.
    Read the full deep dive

  • Real-Time Detection and Repair of LLM Agent Failures: A Deep Read of AgentTrajectorySentinel

    Advanced Agent runtime, safety, and evaluation
    Agent Evaluation: Part 3 , Note: Aug 7, 2026 , Paper: 2026 , AI Agent

    A critical reading of AgentTrajectorySentinel's low-cost healthy-only temporal monitor, deterministic verification, and rollback-and-retry loop, separating measured detection and repair gains from calibration dependence, content blind spots, and reproducibility limits.

    Understand it in 90 seconds
    Problem
    agent failures begin before the final answer; an LLM judge at every step can be too slow and costly.
    Core insight
    a temporal monitor trained on healthy trajectories works with deterministic verification; supported interventions roll state back to a trusted checkpoint for targeted retry.
    Strongest evidence
    across 2,823 committed episodes, three frameworks, and several models, the repair study compares monitor, verifier, and policy and reports task success from 52% to 73% (Section 5; Table 4).
    Main boundary
    healthy-only calibration, short trajectories, injected failures, and weak textual-hallucination detection limit transfer to a new production stack.
    Read the full deep dive

  • OSReward Deep Read: Why Agent Success Cannot Be Judged by Another Model Alone

    Advanced Agent runtime, safety, and evaluation
    Agent Evaluation: Part 1 , Note: Aug 2, 2026 , Paper: 2026 , AI Agent

    A complete reading of OSReward's data construction, 27 VLM judges, Hard and Multi subsets, error and cost analyses, OS-Shepherd-100K training, and a deployable hybrid verification architecture.

    Understand it in 90 seconds
    Previous approach's limitation
    Traditional model judges infer success from compressed screens and text histories, making them vulnerable to an agent's completion narrative.
    Core insight
    Use a human-gold benchmark to expose false-success bias, then separate verifiable state, model judgment, and human arbitration into distinct evidence layers.
    Strongest evidence
    Table 1 and Figures 5–7 show judges near 90% on the full set falling to roughly 70% on the Hard set, with directional failure-recall and cross-platform errors.
    Main boundary
    OS-Shepherd improves cost and some accuracy, but its labels still come from strong-judge agreement and the full artifact and production-verifier stack is incomplete.
    Read the full deep dive

For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.

Speaking & contact