Agent Evaluation & Observability

36 reading notes

← Back to Paper Reading

Examine whether agents are actually reliable through task outcomes, trajectories, runtime signals, and benchmarks.

Reader question

Beyond the final answer, which signals must we observe and verify before trusting that an agent completed its work?

Reading library

Read this topic in depth

Every paper reading currently filed under this research topic.

  • After the Party: Governing What a Viral Agent-Skill Ecosystem Left Behind

    Advanced
    Agent Skill 生態治理: Part 1 , Note: Sep 17, 2026 , Paper: 2026 , AI Safety

    A deep reading of After the Party’s OpenClaw and ClawHub ecosystem study: 91 days of explosive growth, download concentration, the reviewability gap, privilege evidence, scanner disagreement, and the governance method that can transfer without pretending the rates do.

    Understand it in 90 seconds
    Research problem
    When an agent-skill registry expands rapidly, which signals can still support governance decisions across downloads, stars, versions, comments, declared capabilities, and executable privileges? The authors study OpenClaw and ClawHub through growth, association portability, reviewability, and scanner agreement.
    Core insight
    A skill does not live only in its text. The same SKILL.md or package can expose a different privilege surface under a different host, tool visibility, execution context, and policy. Registry metadata therefore cannot collapse popularity, reviewability, static evidence, and runtime behavior into one trust score.
    Strongest evidence
    RQ1 reconstructs stock growth from 33,399 to 65,175 over 91.11 days; the top 10% receive 46.93% of downloads and the Gini coefficient is 0.528. RQ3 finds at least one privilege signal in 85.06% of evaluable skills. RQ4 shows only 446 items flagged by all three scanners; on the small adjudicated reference set, the LLM scanner has 61.06% sensitivity versus 21.67% for the static scanner.
    Main boundary
    This is not an insecurity prevalence estimate for every registry, nor a general scanner benchmark. It is one ecosystem, a set of snapshots, partially reconstructed history, withdrawn data, missing fields, and no perfect ground truth.
    Read the full deep dive

  • Predicting Partial Answer Quality and Utility in Agentic Retrieval-Augmented Generation

    Advanced
    Agentic RAG 的軌跡評估: Part 1 , Note: Sep 17, 2026 , Paper: 2026 , Information Retrieval

    A deep reading of Predicting Partial Answer Quality and Utility in Agentic Retrieval-Augmented Generation: turning intermediate answer quality, utility, and trajectory signals into an early-stopping controller, then checking the savings and transfer boundary.

    Understand it in 90 seconds
    Problem
    An agentic RAG system may execute several query, retrieve, read, and reasoning rounds before it produces a final answer. Running every instance to natural stopping wastes cost after the answer has saturated, while continuing after a failure pattern may only repeat the failure. The paper asks whether each partial answer can provide a prediction of quality and utility.
    Core insight
    Partial quality and incremental utility are different targets. Quality asks how close the current answer is to the gold answer; utility asks how much quality changed from the previous round. Quality is easier to predict, while the sign and magnitude of utility depend more strongly on trajectory, retrieval noise, and task.
    Strongest evidence
    On Search-R1 and R1-Searcher over HotpotQA, 2WikiMultiHopQA, and MuSiQue, the authors compare supervised and unsupervised predictors. Search-R1 reaches about 0.438 at the best quality Pearson correlation and about 0.321 for utility; with thetaP=0.3 and thetaU=0.2, the controller reduces average iterations from 3.21 to 2.86 and preserves 97.60% of natural-stopping quality.
    Main boundary
    Partial answer quality is probed with F1 against a gold answer. That is not open-ended answer quality, and it does not show that a threshold transfers to a new retriever, model, corpus, answer format, or controller. Probing itself has generation cost.
    Read the full deep dive

  • SilentProbe: When HTTP 200 Did Not Answer the Question

    Advanced
    Agent Tool Contracts and Silent Failure: Part 1 , Note: Sep 17, 2026 , Paper: 2026 , AI Systems

    A deep reading of SilentProbe (arXiv:2609.00035 v1): from OpenAPI constraint gaps and live differential probes to agent false negatives, separating how disclosure and machine-readability determine whether a tool fails honestly.

    Understand it in 90 seconds
    Problem
    When an agent calls a third-party API, an empty result can mean that no record matched, or that the server did not understand a filter, discarded it, and still returned HTTP 200 with parseable JSON. Neither case necessarily provides an exception, an error status, or a field on which to branch.
    Core insight
    Separate two questions. Disclosure asks whether the model can select a value accepted by the vendor from the description; machine-readability asks whether a validator can reject a wrong value before the request leaves the gateway. Only the second can be enforced by generic infrastructure.
    Strongest evidence
    In the 721,320 parameter leaves of the public OpenAPI corpus, only 7.5% declare an enum and 15.2% declare any machine-checkable constraint; 40.1% of documents contain at least one prose constraint gap. In 219 live perturbations, machine-checkable constraints produced 111/111 honest errors, while prose-only constraints produced 44/61 silent failures (Sections 4.1–4.2, Figure 5).
    Main boundary
    In the three-parameter agent experiment, a description that showed only 1/18 department values led to silent failure in 88/88 attempts; promoting that vocabulary into an enum produced 0/89 silent failures. This is evidence under the interface and harness used by the paper, not a general law about models or every production API.
    Read the full deep dive

  • RAGSieve: Detecting RAG Knowledge-Poisoning Promotion with Self-Referenced Local Contrast

    Advanced
    RAG Retrieval Integrity & Governance: Part 1 , Note: Sep 17, 2026 , Paper: 2026 , Information Retrieval

    A deep read of RAGSieve: query-local and corpus-local references expose suspicious retrieval promotion without a trusted clean corpus, while the paper's boundary remains essential—promotion detection is not truth verification.

    Understand it in 90 seconds
    Problem
    RAG places external corpus content inside the evidence used for generation. An attacker who can get a small number of documents ingested through a public page, shared store, or connector may promote a chosen false answer into the top five for a target query. The hard part is that the attacked corpus is not a trusted reference, and natural semantic density varies by topic.
    Core insight
    Do not assume a separately clean dataset, and do not apply one global threshold across the corpus. RSQ performs query-local contrast between the top five and ranks 6–20 for the same query; RSG performs corpus-local contrast between each document's semantic neighbors and its own local floor. Both let the inspected system supply its own matched control.
    Strongest evidence
    Across nine dataset–retriever systems and six attacks, RSQ reaches 95.2% macro AUROC and detects 82.2% of poison at an operating point allowing at most 5% clean-document removal. RSG reaches 93.3% and 79.8%. Serial RSG plus RSQ lowers ASR from 67.4% to 14.0% while unpoisoned-retrieval F1 changes from 42.1% to 41.3% (Tables 1, 5, and 9).
    Main boundary
    These are results on synthetic attacks, three QA corpora, three dense retrievers, and a fixed evaluation protocol. They support the claim that suspicious promotion patterns can be exposed by local contrast; they do not support the claim that a flagged document is false, that a retrieved claim has been truth-verified, or that production-scale multi-tenant latency and zero-poison guarantees follow.
    Read the full deep dive

  • BTS-AgentBench: Compiling Read-Only Telemetry into Replayable Agent Episodes

    Advanced
    Telemetry to Agent Evaluation: Part 1 , Note: Sep 17, 2026 , Paper: 2026 , AI Engineering

    A deep read of Jeong-Yoon Kim's BTS-AgentBench (arXiv:2608.27334 v1): a deterministic path from building telemetry to read-only tools, executable tasks, bounded interaction contracts, and evidence-grounded evaluation—strong on replay consistency, bounded beyond production safety or arbitrary-domain portability.

    Understand it in 90 seconds
    Problem
    Buildings accumulate years of sensor and equipment telemetry, but a raw history is not an executable multi-turn task for an Agent. Hand-authoring each task makes it difficult to preserve a site's vocabulary, source-derived answers, split identity, and evidence links at the same time.
    Core insight
    Treat benchmark construction as a replayable compiler. First place metadata and histories behind read-only tools; then build a static executable task with fixed golds; finally wrap that computation in a typed, bounded interaction contract. Clarification, goal revision, nearest-timestamp policy, quality decisions, and evidence follow-ups can change the surface, but the source computation and its gold must be re-executed together.
    Strongest evidence
    Two independent raw-to-episode builds match all 11 logical tool-store exports and regenerate the BTS 356/87/89 train/dev/test release row by row; all 532 released episodes pass coded contract preflight. This supports construction consistency, not operator realism or production deployment (paper Table 7 and Appendix A.3).
    Main boundary
    BTS-AgentBench is a read-only, offline, bounded building-telemetry benchmark. Its zero controller success is a construction-exclusion condition, not an independent hardness estimate; XAI4HEAT's 41/41 result shows execution on a second telemetry corpus, not portability to arbitrary event logs or physical control.
    Read the full deep dive

  • When Failure Propagates, Can We Still Find the Start? Causal Failure Attribution in Agentic RAG

    Advanced
    Agentic RAG Failure Diagnosis and Attribution: Part 1 , Note: Sep 17, 2026 , Paper: 2026 , Retrieval Systems

    A deep reading of When Failures Propagate: an interventional benchmark, three-hop MuSiQue, and certified content corruption that separate failure detection, causal attribution, propagation, and recovery in agentic RAG.

    Understand it in 90 seconds
    Research problem
    Agentic RAG splits retrieval, reasoning, and answering across several hops. An early bad piece of evidence can become query drift, a wrong bridge, or a wrong answer, but later retrieval can also repair it. A final answer or the last trace state therefore cannot directly tell us which hop first caused the failure.
    Core insight
    AgenticRAG-FP injects a certifiable fault at a specified hop, then resumes execution from the changed prefix and regenerates the suffix. The diagnoser is not guessing the cause of a static error trace; it is tested against the known injectedathop label with exact-hop attribution.
    Strongest evidence
    The strict dense Claude Haiku 4.5 sweep uses 80 three-hop MuSiQue questions. Among cases that still fail, coverage-based exact-hop accuracy is hop 1: 0.91 [0.81, 0.98], hop 2: 0.00 [0.00, 0.00], and hop 3: 0.00 [0.00, 0.00], with failed denominators 43, 36, and 21 respectively (Table 2, Section 7.1).
    Main boundary
    The result supports the claim that, under this strict intervention and suffix-resumption setup, coverage’s hop-level signal disappears at deeper hops. It does not support the claim that every natural agentic RAG failure is un-attributable. The content study has only 18 hop-2 failed cases and 3 hop-3 failed cases, so its method comparisons cannot be generalized.
    Read the full deep dive

  • ACE: Let a Canvas Agent Understand Structure Before It Corrects Itself

    Advanced
    Agent Canvas Editing and Evaluation: Part 1 , Note: Sep 17, 2026 , Paper: 2026 , AI Engineering

    A deep reading of ACE (arXiv:2608.24103 v1): hierarchical scene graphs, CARE routing, and an instruction-following judge turn multi-slide editing into a scoped, diffable, rollback-aware loop, with explicit limits around benchmarks, human raters, mock mode, and live reproduction.

    Understand it in 90 seconds
    Problem
    PowerPoint- and HTML-like flat, absolute-positioned documents encode objects as many coordinates. Adding one element can force an agent to recompute other positions, while a valid alternative design can be penalized by reference-diff metrics.
    Core insight
    ACE uses a hierarchical scene graph with parent–child relations, relative transforms, and auto-layout, then maps intent to structured operations through 98 specialized tools. CARE exposes only a relevant slide, node structure, or design token. After an edit, JsonDiff compares the original and current state, and a ground-truth-free instruction-following judge supplies the next critique.
    Strongest evidence
    On the full 94-task benchmark, GPT IF is 4.23 for ACE versus 3.81 for the HTML baseline, with paired p=.010; reported speed is about 1.75x and cost about 44% lower. On that same full set, VQ is 3.66 versus 3.57 with p=.56, so the headline is not universal visual-quality improvement.
    Main boundary
    Twenty-six blind raters give ACE versus HTML a 58.7% decisive overall win rate; self-corrected output versus single-pass is 81.5%. The panel is small, ties are common, agreement is low to moderate, and judge circularity remains. The paper does not show universal creative-editing improvement or that a judge can replace a designer.
    Read the full deep dive

  • Tool Call Succeeds, Workflow Fails: External-Effect Anomalies at the Agent–Tool Boundary

    Advanced
    Agent–Tool Boundary and Effect Reliability: Part 1 , Note: Sep 16, 2026 , Paper: 2026 , AI Systems

    A deep reading of the effect-history model behind Agent–Tool Boundary: why a successful tool response cannot guarantee a coherent external world state, and what MCP annotations and transactional contracts still leave unspecified.

    Understand it in 90 seconds
    Problem
    An agent workflow may reserve a table, charge a card, and send a confirmation. Each tool may return success, failure, or a timeout, but the workflow actually cares about which irreversible effects happened in the outside world and which effects still survive. If the runtime only trusts the latest response, retries, speculation, concurrency, and crashes can separate “the call succeeded” from “the work is complete.”
    Core insight
    Separate external-effect history from runtime observation. An attempt can produce an unknown observation even when externalization already happened; conversely, a runtime can observe success without the later commit, abort, or compensation producing the intended world state. Workflow safety is about effect history, not one API response.
    Strongest evidence
    Table 2 in Section 3 maps eight anomalies to the boundary capabilities they require. Section 5 audits the 2026-07-27 MCP registry snapshot: 98,291 tools were observed, 74.0% serialized at least one standard annotation, and 61.7% carried all four, yet Table 4 finds no sufficient transactional capability for A2–A8.
    Main boundary
    This is an effect-history vocabulary, a conjectural coverage analysis, and a runtime-contract argument. It is not a prevalence study showing that all production agents exhibit these eight anomalies. The coverage labels for ACRFence, RAC, Atomix, Cordon, CoAgent, and Shepherd are stated or partial comparisons, not formal proofs.
    Read the full deep dive

  • EvoOntology: Turning a Static Data-Agent Semantic Layer into a Verifiable, Self-Evolving Interface

    Advanced
    A Self-Evolving Ontology Layer for Data Agents: Part 1 , Note: Sep 16, 2026 , Paper: 2026 , Data Agents

    A deep reading of EvoOntology: an MCP ontology layer built by evidence-grounded probing, then refined through attribution-guided typed edits and a backbone-conditional paired gate.

    Understand it in 90 seconds
    Problem
    A data agent working over tables, files, and databases does not merely lack column names. It may not know which field, join, filter, or value constraint grounds a domain concept. Raw querying makes every trajectory rediscover the schema; a static semantic layer can be too large, stale, and expensive to maintain. This agent–data gap becomes wrong queries, longer trajectories, and ungrounded answers.
    Core insight
    Treat the ontology as a versioned MCP service with Content, Schema, and Tool layers rather than as a fixed prompt document. The builder agent probes the raw data to ground semantics. The evolution agent finds gaps in failed trajectories, proposes a typed, evidence-grounded patch at one layer, and accepts it only through a paired validation gate on the same backbone.
    Strongest evidence
    Figure 2 shows the three-layer architecture; Figure 4 shows four backbones improving across accepted rounds; Tables 5–7 isolate the contributions of the gate, attribution, diagnosis, editable levels, and content object families; Appendix B Table 8 shows that per-turn context grows while average turns per task fall from 14.6 to 8.4 and total tokens per task fall from 52.6K to 42.0K.
    Main boundary
    The headline gain must be read with scope labels: four-backbone analysis subset, six-backbone main tables, benchmark-specific metrics, and round-wise evolution are not interchangeable. The repository contains inspectable framework code and a demo, but benchmark raw data, prebuilt ontologies, model weights, and complete provider credentials are not delivered as one reproducible bundle.
    Read the full deep dive

  • Corrupt Plans, Clean Traces: How Plan Injection Evades Chain-of-Thought Monitoring

    Advanced
    CoT Monitoring and Plan Injection: Part 1 , Note: Sep 16, 2026 , Paper: 2026 , AI Safety

    A deep reading of Plan Injection: when an adversarial plan enters context and an actor rewrites it into plausible Chain-of-Thought, why the monitor’s clean trace can disconnect from behavioral causality, and where APPS, Bio-Math, and budget-sweep evidence stops.

    Understand it in 90 seconds
    Problem
    A CoT monitor usually reads an actor’s reasoning and asks whether it is executing a harmful plan. If that plan is placed into the actor’s context first, the actor can absorb it, rewrite it as plausible reasoning, and emit problematic code or an answer. The monitor sees the later trace but may not see what actually changed the behavior.
    Core insight
    Plan injection separates the causal source of behavior from the text visible to the monitor. The attacker need not make the actor copy the plan or state an explicit malicious intention; it is enough for the injected plan to steer the actor while the generated reasoning does not clearly attribute itself to that plan.
    Strongest evidence
    Figure 3 measures adoption of injected plans on APPS and Bio-Math; Figures 4–5 compare actor-only and with-plan monitor views; Table 8 reports 33%–68% APPS and 74%–82% Bio-Math end-to-end undetected attack success; Figure 6 shows that added reasoning budget is not monotonically safer for the Qwen3-235B monitor.
    Main boundary
    This is not an estimate that every reasoning model is equally vulnerable, nor a formal proof that CoT is always untrustworthy. The authors test Qwen3-8B and DeepSeek-R1-0528 actors on APPS and Bio-Math, sweep budget for only one monitor, and release no code, checkpoint, or complete rerunnable data bundle.
    Read the full deep dive

  • K-Bench: Why Agentic Unlearning Cannot Be Certified from the Final Answer Alone

    Advanced
    Agent Security and Unlearning Evaluation: Part 1 , Note: Sep 15, 2026 , Paper: 2026 , AI Security

    A deep read of Yu et al.'s K-Bench (arXiv:2609.12808 v1): an end-to-end agent deployment benchmark that evaluates six observable channels across four memory substrates, using OR-of-channels leakage, collapse-aware K-Scores, and pre-registered statistical tests to separate forgetting from channel migration and agent collapse.

    Understand it in 90 seconds
    Problem
    TOFU- and MUSE-like unlearning benchmarks treat a model mainly as a question-answering interface and read one direct answer. That is useful for a secret that lives in the weights and leaks only through that surface, but it does not cover context, RAG, database lookup, CoT scratchpads, tool calls, tool returns, or a later summary after deployment.
    Core insight
    control both where the secret lives and which surfaces the deployed agent exposes. K-Bench puts the same kind of PII into one substrate per cell, exposes a ReAct trace as six channels, and takes a logical OR over those channels for each query.
    Strongest evidence
    for the Llama-3.1-8B no-intervention baseline, aggregate OR(all) is 0.223 on C, 0.602 on R-text, and 0.855 on R-struct. TOFU/MUSE weight probes see no target memorization on those lanes. This is a coverage gap, not a case that weight unlearning merely needs to be stronger.
    Main boundary
    the result covers six observable text channels, four pure substrates, English PII, a fixed ReAct harness, and selected model/injection configurations. It is not a proof that production memories, logs, external databases, or model copies have been deleted.
    Read the full deep dive

  • REVA: Moving RAG compression into reusable evidence views instead of paying per request

    Advanced
    Retrieval Systems: From Evidence to Production RAG: Part 1 , Note: Sep 15, 2026 , Paper: 2026 , NLP

    A critical reading of Nguyen et al.'s REVA (arXiv 2609.11209 v1): historical generator attention becomes a document-keyed score store, separating offline scoring from online rendering while exposing unseen-document fallback, local/global budgets, quality, and latency boundaries.

    Understand it in 90 seconds
    Problem
    A post-retrieval compressor that invokes another model, scores tokens, or rewrites text on every request can erase the latency benefit of a shorter context. A model-agnostic selector may also preserve information the target generator already knows while deleting evidence it actually needs.
    Core intuition
    Historical RAG requests already contain signals about how the generator used a document. Map attention from the query and available answer/response tokens onto readable word units, average those signals across repeated document accesses, and obtain a reusable evidence prior. This is a document-level retention tendency, not a query-specific answer.
    Strongest evidence
    With a fixed top-10 retrieval cache, four QA benchmarks, and three generators, Table I's full-split B=512 REVA-local result is higher than Trunc-local on NQ, TriviaQA, HotpotQA, and 2Wiki. Table II averages 37.83 F1, 26.98 EM, and 27.5 ms online overhead across 12 generator–dataset settings. Table III's all-seen 120-cell budget grid gives REVA-global 43.72 F1, 32.75 EM, and 49 ms.
    Main boundary
    All-seen keeps only held-out queries whose retrieved documents all have scores, so it is diagnostic rather than deployment coverage. Full-split includes prefix fallback. Attention is an evidence-importance proxy, not a citation verifier, and reported online overhead excludes score-store construction and updates.
    Read the full deep dive

  • VikingRAG: Fewer Retrieval Rounds, Less Context Waste for Structured-Document RAG

    Advanced
    Structured Retrieval for Production RAG: Part 1 , Note: Sep 15, 2026 , Paper: 2026 , Information Retrieval

    A deep reading of VikingRAG: hierarchy-preserving URI-addressable storage, Search/List/Grep/Read tools, reusable experience edges, and adaptive escalation for reducing repeated retrieval tokens and latency.

    Understand it in 90 seconds
    Problem
    Enterprise manuals, syllabi, papers, contracts, and financial reports are not bags of unrelated chunks. An answer may require finding the right document, then following chapter, section, or subsection structure to collect facts that are far apart. Serializing every directory into the prompt makes structural context expensive; doing only one flat top-k retrieval can miss cross-section dependencies on the first pass.
    Core insight
    Move the hierarchy from the prompt into queryable external semantic storage. Every directory node, chunk, and multi-level abstract receives a URI whose prefix preserves ancestor–descendant relations. A vector result is therefore not only text; it is a navigation handle that can drive scoped List, Grep, and Read operations (Sections 2.2 and 3.1).
    Strongest evidence
    The evaluation covers six structured-document datasets and eight baselines, with default settings K=10, L=1,000, and B=15. End-to-end accuracy, latency, LLM tokens, ingestion, and deletion are measured. Figure 3 and Table 3 report 11.6%–51.9% token ratios for VikingRAG and 5.1%–32.5% for VikingRAG-E+; Figure 7 repeats the central comparison with GPT-5.5, Seed-2.0, and GLM-4.7 on VersionQA.
    Main boundary
    Accuracy is an LLM-as-a-judge semantic-consistency proxy with expert verification, not a direct retrieval-recall or independently replicated correctness proof. Experience edges are warmed with 1,000 synthetic historical questions generated from the same document corpus; false-no-escalation is still 14.4% on QASPER and 6.7% on FinanceBench (Table 7).
    Read the full deep dive

  • DRACO: Sending Long-Horizon Agent Credit Back to the Steps

    Advanced Agent runtime, safety, and evaluation
    Agent Training and Rewards: Part 1 , Note: Sep 9, 2026 , Paper: 2026 , AI Engineering

    A source-grounded reading of DRACO (arXiv:2609.04094): dynamic per-trajectory rubrics create an outcome-blind reward, then a closed-form rule redistributes GRPO advantage to the steps cited by the judge.

    Understand it in 90 seconds
    Problem
    How can an agent be trained without an outcome oracle, and how can the resulting signal avoid treating a long trajectory as one indivisible action?
    Core insight
    Generate, merge, deduplicate, and filter rubrics dynamically for a task and sampled rollout group. Ask the judge to cite the steps behind each verdict, then redistribute the trajectory advantage to those steps with a closed-form rule.
    Strongest evidence
    With Qwen3.6-27B, AppWorld test-normal TGC/SGC rises from 69.4/41.1 for the base policy to 85.3/70.6 for DRACO. Against the same-budget outcome-reward reference, the margins are +5.3/+11.3 points (Table 2, Section 4.2). Zero-shot tau-bench Banking success rises from 15.8 to 20.4.
    Main boundary
    These are benchmark and end-task results, not direct validation of the judge or of step causality. The authors have no human-rater calibration; a judge may be consistently wrong, and a wrong attribution can still produce a better policy by chance.
    Read the full deep dive

  • CONTINUITY: Keeping provenance, authorization, and tool effects continuous across Agent composition

    Advanced Agent runtime, safety, and evaluation
    Agent Runtime, Security, and Effect Boundaries: Part 1 , Note: Sep 9, 2026 , Paper: 2026 , AI Security

    A critical reading of Zheng and Yang's CONTINUITY (arXiv:2609.05269 v1): security-context contracts, field-level provenance, transformation witnesses, and effect-bound permits for preserving an LLM Agent's instruction-to-effect boundary.

    Understand it in 90 seconds
    Problem
    An Agent security path rarely has one control point. Ingress may track provenance, a gateway may enforce policy, an adapter may change the protocol representation, a tool server may create an effect, and a final sink may check a permit. Each can look locally reasonable while security-critical context is truncated, amplified, rebound, or accepted in a stale or replayed form.
    Core insight
    Give every component an assume–guarantee contract, and make each transition carry verifiable evidence for its root, field provenance, release, role, transformation, and finality state. Security is not “the last signature verifies”; it is whether the effect can be traced to a complete, authorized, current, single-use witness chain (Sections 1, 5, and 6).
    Strongest evidence
    The authors generate 2,560 attack instances from 4 domains, 32 fault classes, and 20 parameterized instances per fault–domain pair, yielding 128 fault–domain classes. The full CONTINUITY configuration records 0/2,560 harmful effects, contains 128/128 classes, completes 700/700 benign tasks, and escalates 200/200 ambiguous tasks (Table 2 and Figure 3).
    Main boundary
    These are exact conformance counts over a generated fault space, not a natural attack distribution or a production attack rate. Trusted roots, validator correctness, context capture, finality sinks, and provider semantics sit in the TCB or deployment assumptions; the artifact is not a production MCP, A2A, OWASP ACS, or cloud-IAM integration (Sections 3, 8.1, and 12).
    Read the full deep dive

  • Parsing the Stream: Long-Horizon Agents Need Auditable Live State, Not Just Memory

    Advanced Agent runtime, safety, and evaluation
    Agent Trace Observability: Part 1 , Note: Sep 7, 2026 , Paper: 2026 , AI Engineering

    A critical reading of Pakhomov and Nijkamp's Parsing the Stream (arXiv:2609.01466): an append-only trace is folded into typed RunState and compiled into observer and worker views. The paper reports gains on specific accumulation tasks and monitoring costs, but does not show that fixed aggregates replace every form of trace memory.

    Understand it in 90 seconds
    Problem
    A long-horizon Agent trace outgrows both of its consumers. A human observer needs to know what is happening, what has settled, and what is still missing; the worker Agent must put the same growing trace back into a bounded context window. A tail-only view loses early facts, while replaying the full history on every turn makes tokens, cost, and failure risk grow together.
    Core insight
    Instead of building two unrelated summarizers for the worker and observer, write the trace as an append-only typed ledger, fold it into a RunState with source and coverage information, and compile consumer-specific views from that state.
    Strongest evidence
    In COMPREHEND, using 12 real transcripts and 70 monitoring questions per condition, the compiled view reaches 0.871 accuracy for Sonnet 5 and 0.850 for Haiku 4.5; raw tails reach only 0.479 and 0.476. In CONTINUE's clean 120-link protocol, the curated fold scores 30/30, the scratchpad scores 30/30, and full context scores 8/30 (Tables 1–2, Figures 2–3).
    Main boundary
    The evidence is conditional on schema coverage and task shape. The authors show that the fold loses its advantage on an alternating-sign chain, and they acknowledge benchmark–system co-evolution, a single vendor, a fixed schema, single-session traces, and untested prompt injection, secret redaction, and multi-agent ledgers.
    Read the full deep dive

  • Generative Agents: Observe–Reflect–Plan in a Multi-Agent Sandbox — Do Not Mistake Sandbox Memory for MemGPT OS Paging

    Intermediate Agent runtime, safety, and evaluation
    Generative Agents deep reading: Part 1 , Note: Aug 28, 2026 , Paper: 2023 , AI Engineering

    A deep read of Park et al., UIST 2023 / arXiv:2304.03442 v2: 25 agents in Smallville use a memory stream, periodic reflection, and retrieval-based planning. Interview ablations hit TrueSkill μ 29.89 vs 21.21 fully ablated; two-day sandbox diffusion and party coordination are qualitative evidence, not production runtime.

    Understand it in 90 seconds
    Problem
    Large language models can mimic human behavior at a single time point, but believable agents over long horizons need growing memories, multi-agent social dynamics, and planning grounded in past experience—neither a longer prompt nor one-shot generation is enough.
    Core insight
    Store each agent's full experience in natural language in a memory stream, run periodic reflection to synthesize higher-level inferences, and retrieve with relevance, recency, and importance to plan and react. Twenty-five agents interact in the Smallville sandbox; the memory control plane is social-simulacra observe-reflect-plan, not MemGPT OS-style context paging for one agent.
    Strongest evidence
    Interview ablations (Figure 8): full architecture TrueSkill μ 29.89 (σ=0.72), beating no reflection (26.88), no reflection or planning (25.64), crowdworker baseline (22.95), and full ablation (21.21). Two-day open simulation (Section 7.1): mayor info holders 4%→32%, party info 4%→52%; relationship network density 0.167→0.74; party 12 invited, 5 attended.
    Main boundary
    Sandbox plus ChatGPT; the authors report thousands of dollars in token cost and multi-day runs for 25 agents over two game days (Section 8.2). Common failures are missed retrieval, fabricated embellishments, and overly formal speech from instruction tuning. This is not production ACL memory, not Reflexion across-trial verbal credit assignment, and not later Letta or xMemory product metrics.
    Read the full deep dive

  • ReAct: Interleave Thought and Action, but Do Not Treat a Few-Shot Loop as an Agent Runtime

    Intermediate Agent runtime, safety, and evaluation
    ReAct Deep Dive: Part 1 , Note: Aug 27, 2026 , Paper: 2023 , NLP

    A source-grounded reading of Yao et al., ICLR 2023: language thoughts join the action space, while HotpotQA, FEVER, ALFWorld, and WebShop keep hallucination, search failure, and the abstract's +34% / +10% in separate buckets.

    Understand it in 90 seconds
    Problem
    LLM reasoning (chain-of-thought) and acting (WebGPT, SayCan) were treated as separate lines of work. CoT never touches an environment; act-only methods can query the outside world but lack high-level plans and exception handling.
    Core insight
    Add language thoughts to the action space. A thought does not change the environment and produces no environment observation; it only updates context, then interleaves with environment actions. The changed decision point is no longer “only think” or “only act,” but “in this same trajectory, is the next legal move a sentence to oneself or a touch of the world?”
    Strongest evidence
    ALFWorld best-of-6 ReAct 71% versus Act 45% and BUTLER best-of-8 37%; WebShop SR 40.0 versus IL+RL 28.7. In a human analysis of HotpotQA failures, 56% of CoT failures are hallucinations versus 0% for ReAct (Table 2).
    Main boundary
    On HotpotQA with PaLM-540B, pure ReAct EM is 27.4, below CoT at 29.4. The 35.1 / 64.6 headline cells are ReAct↔CoT-SC switches. The method is few-shot prompting with a Wikipedia API of search, lookup, and finish. It is not a deployable runtime.
    Read the full deep dive

  • Toolformer: Self-Supervised API Calls Are Not an Agent Loop

    Intermediate Agent runtime, safety, and evaluation
    Toolformer Deep Dive: Part 1 , Note: Aug 27, 2026 , Paper: 2023 , NLP

    A source-grounded reading of Schick et al., NeurIPS 2023: future-token loss filters QA, Wikipedia, calculator, calendar, and translation calls on CCNet for GPT-J. LAMA and math jump; this is still not a chainable agent runtime.

    Understand it in 90 seconds
    Problem
    Language models are weak at arithmetic, factual lookup, low-resource languages, and time awareness compared with much smaller specialized systems. Tool use at the time either needed large human annotation budgets or was tied to task-specific few-shot prompts that already knew which tool to call.
    Core insight
    Insert API calls into next-token prediction. A handful of human demonstrations only teach the call format. Whether a sampled call is kept is decided by whether the call plus its result reduces future-token loss. The changed control point is not a thought–action loop; it is when a single API call is written into the language-modeling string.
    Strongest evidence
    Same GPT-J 6.7B, zero-shot. LAMA SQuAD / Google-RE / T-REx rise from 17.8 / 4.9 / 31.9 to 33.8 / 11.5 / 53.5 and beat OPT-66B and GPT-3-175B. Math ASDiv / SVAMP / MAWPS rise from 7.5 / 5.2 / 9.9 to 40.4 / 29.4 / 44.0. The QA tool and calculator are selected on about 98.1% and 97.9% of examples.
    Main boundary
    With the QA tool off, Wikipedia search still trails GPT-3. The authors cannot chain tools, cannot browse search results interactively, are wording-sensitive, evaluate at most one API call, get few calculator examples, and ignore tool cost. This is not a production agent runtime.
    Read the full deep dive

  • SWE-bench: Real GitHub Issues as Evaluation, but 1.96% Is Not a Model Ceiling

    Intermediate Agent runtime, safety, and evaluation
    SWE-bench Deep Dive: Part 1 , Note: Aug 27, 2026 , Paper: 2024 , AI Engineering

    A source-grounded reading of Jimenez et al., ICLR 2024 Oral: the evaluation unit is a real GitHub issue, a full Python repository, and tests. Claude 2 resolves 1.96% under BM25; that number is a protocol, not a model ranking.

    Understand it in 90 seconds
    Problem
    Coding benchmarks such as HumanEval compress success into writing a self-contained function. Real software engineering is: read a GitHub issue, edit a repository with thousands of files, and let tests decide whether the issue is fixed. Prior scores do not measure that job.
    Core insight
    Change the evaluation unit to a real issue plus a full Python repository plus tests. The model emits a patch; after unix patch applies it, every fail-to-pass and pass-to-pass test must pass before the instance is resolved. The changed control point is not a new agent architecture; it is what counts as success.
    Strongest evidence
    Under BM25 retrieval and a 13k context, Claude 2 resolves 1.96% (abstract, Section 1, Table 2). In the same protocol, Table 5 lists Claude 2 at 1.97% and also includes Claude 3 Opus at 3.79%. Under oracle retrieval, Claude 2 rises to 4.80% (Table 18). SWE-Llama reaches only 0.70% under BM25 and still mostly solves the simplest issues.
    Main boundary
    Python, issue-fix, binary tests. Resolve does not score maintainability, uncovered behavior, or review. BM25 and oracle are different retrieval conditions. Later SWE-bench Verified, SWE-agent, and ProMax use different setups, so their scores do not belong in this paper's tables.
    Read the full deep dive

  • CoT: Make the Model Write the Reasoning, but Do Not Treat It as an Agent That Moves

    Intermediate Agent runtime, safety, and evaluation
    CoT deep reading: Part 1 , Note: Aug 27, 2026 , Paper: 2022 , NLP

    A source-grounded reading of Wei et al., NeurIPS 2022: few-shot exemplars with intermediate steps elicit multi-step reasoning in large frozen models. PaLM 540B on GSM8K moves from 17.9 to 56.9; this is still a prompt, not tools, an environment, or memory paging.

    Understand it in 90 seconds
    Problem
    Standard few-shot prompting gives only $\langle$question, answer$\rangle$ pairs. Multi-step arithmetic, commonsense, and symbolic tasks stay weak, and scaling the model alone often leaves those curves flat.
    Core insight
    Change the exemplar to $\langle$question, intermediate reasoning, answer$\rangle$. The decision point moves from “answer immediately” to “write the work, then answer.” Weights stay frozen. This is still a prompt, not an agent.
    Strongest evidence
    PaLM 540B on GSM8K moves from 17.9 to 56.9, above Cobbe et al.’s finetuned GPT-3 + verifier at 55 (Table 1, Figure 2). Figure 4 / Table 2 show the gain appearing around 100B parameters.
    Main boundary
    No environment, no tools, no memory paging. Small models often get worse. A chain can be unfaithful, or luckily reach the right number. Self-consistency (Wang et al., 2022a) is a later paper; the main tables here use greedy decoding.
    Read the full deep dive

  • WebGPT: Let the Model Browse for Answers, but Do Not Treat It as a Reasoning Agent Loop

    Intermediate Agent runtime, safety, and evaluation
    WebGPT deep reading: Part 1 , Note: Aug 27, 2026 , Paper: 2021 , NLP

    A source-grounded reading of Nakano et al., arXiv:2112.09332 v3: GPT-3 is given a text browser and trained with human demonstrations and preference / reward modeling to search, quote, and answer. The 175B best-of-64 model is preferred 56% versus demonstrators and 69% versus Reddit; this is browsing QA, not ReAct’s thought–action–observation contract.

    Understand it in 90 seconds
    Problem
    Long-form question answering lags humans. Retrieval and synthesis were built as separate pieces. Without citations, people cannot cheaply check paragraph-level facts.
    Core insight
    Outsource search to Bing, leave synthesis to a finetuned GPT-3, and insert a text browser between them. The model may only issue Table 1 commands (search, click, quote, scroll, end), collect quotes while browsing, then write the answer. Training is behavior cloning from human demonstrations, plus a reward model from human preferences, plus rejection sampling at inference.
    Strongest evidence
    175B best-of-64 is preferred 56% to demonstrators and 69% to the highest-voted ELI5 answers (Section 4.1, Figure 2). Best-of-64 is preferred 68% to plain BC; RL is preferred 58% to BC, but stacking RL on rejection sampling adds little (Section 5.1, Figures 4 and 5).
    Main boundary
    There is no separate thought action. The text browser is a constrained action space, not a general tool loop. Answers can still mis-paraphrase quotes or cherry-pick sources that look convincing to labelers. This is a 2021 OpenAI technical report / arXiv preprint, not a later production browsing product.
    Read the full deep dive

  • Gorilla: Turn a Large API Catalog into Retrievable Tools, but APIBench Does Not Establish MCP Product Behavior

    Intermediate Agent runtime, safety, and evaluation
    Gorilla Deep Dive: Part 1 , Note: Aug 27, 2026 , Paper: 2024 , NLP

    A source-grounded reading of Patil et al., NeurIPS 2024: retriever-aware finetuning of LLaMA-7B on APIBench (TorchHub / TensorHub / HuggingFace) so catalog-scale API calls can be retrieved and checked. Zero-shot overall accuracy and hallucination beat prompted GPT-4 on that table—this is not a ReAct loop, MidTool mid-training, or RAG-MCP product routing.

    Understand it in 90 seconds
    Problem
    LLMs writing API calls often hallucinate names, arguments, and usage. The real world is not five fixed tools; it is a huge API catalog whose documentation changes frequently.
    Core insight
    Treat tool use as retrieve then call. Build instruction–API pairs on APIBench with self-instruct, then finetune LLaMA-7B in a retriever-aware way (RAT) so the model learns to read the documentation after Use this API documentation for reference: and emit a correct call.
    Strongest evidence
    NeurIPS Table 1. Gorilla zero-shot overall on TorchHub / HuggingFace / TensorFlow Hub is 59.13% / 71.68% / 83.79%, with hallucination 6.98% / 10.95% / 5.40%. GPT-4 zero-shot on the same table is 38.70% / 19.80% / 18.20% overall and 36.55% / 37.16% / 78.65% hallucination. Figure 6 shows that when documents change at test time, a RAT-trained model changes its call.
    Main boundary
    The corpus is ML-hub model-card / API JSON, not an arbitrary REST product catalog. Evaluation is single-call AST subtree matching, not a multi-step agent loop. A weak retriever can hurt (Table 2). Do not write APIBench numbers into MidTool or RAG-MCP.
    Read the full deep dive

  • SWE-Bench ProMax: Can Large-Scale Multilingual Refactoring Measure Long-Horizon Coding Agents?

    Advanced Agent runtime, safety, and evaluation
    Note: Aug 13, 2026 , Paper: 2026 , AI Engineering

    A deep reading of SWE-Bench ProMax, which uses 170 cross-file, multilingual, behavior-preserving refactoring tasks to test whether coding agents can complete large changes rather than merely fix a nearby test.

    Understand it in 90 seconds
    Problem
    Existing coding-agent benchmarks are often Python-heavy and centered on one issue or bug fix. An agent may make a visible test pass while missing cross-file call sites, configuration, documentation, or tests. That does not answer whether it can complete a large, behavior-preserving refactor.
    Design
    The authors mine GitHub refactoring commits, validate Docker environments, use expert and LLM-assisted analysis, and manually review candidates. The final 170 tasks cover Python, Java, TypeScript, Go, C, C++, and Rust.
    Strongest result
    Under the paper's mini-SWE-agent and OpenHands scaffolds, with at most 300 steps and $10 per instance, OpenHands + GPT-5.2 reaches a 41.2% resolve rate. The same model reaches only 21.8% with mini-SWE-agent. This is first a scaffold-and-model result, not a model-only leaderboard.
    Main boundary
    Resolve is a binary outcome—every test must pass. It does not score maintainability, untested behavior, review quality, or the action trace. TypeScript tasks come from only two repositories, with 25 from Angular; language scores are not an independent, balanced experiment in language difficulty.
    Read the full deep dive

  • Agentic Configuration Management: Treating Agent Systems as Governed Configuration, Not Just One Execution

    Advanced Agent runtime, safety, and evaluation
    Agent Security: Part 2 , Note: Aug 12, 2026 , Paper: 2026 , AI Agent

    A deep reading of how ACM uses a framework-independent Configuration Graph, immutable revisions, dependency-aware impact propagation, and runtime provenance to govern heterogeneous agent configurations across LangGraph, CrewAI, and the OpenAI Agents SDK.

    Understand it in 90 seconds
    Problem
    an agent system's behavior is determined not only by code, but also by prompts, models, tools, skills, workflows, policies, frameworks, and runtime state. Existing frameworks and AgentOps tools each manage part of this surface, but do not easily pin down the complete configuration that produced an execution.
    Core insight
    ACM normalizes heterogeneous artifacts into typed, independently versioned Agentic Configuration Items (ACIs), managed through four connected Configuration, Evolution, Assurance, and Runtime Graphs. Execution frameworks project into this representation; the governance kernel operates on the common form.
    Strongest evidence
    27 controlled governance scenarios across LangGraph, CrewAI, and the OpenAI Agents SDK, plus 9 quantitative impact cases. Within the evaluated scope, all three frameworks produce equivalent governance outcomes, and repeated impact sets and metrics are stable (Sections 7.2–7.6; Tables 8, 10, and 12).
    Main boundary
    this is conformance and feasibility evidence for a reference model and prototype. Distributed execution, learning, long-term memory, native MCP/A2A protocols, and large-scale industrial validation are outside the current scope (Tables 13–14; Sections 8.4 and 9).
    Read the full deep dive

  • ADIAS: Turning Agent Self-Improvement into Traceable Issue Repair

    Advanced Agent runtime, safety, and evaluation
    Note: Aug 12, 2026 , Paper: 2026 , AI Engineering

    A deep reading of ADIAS: persistent issue state organizes failure evidence across optimization rounds so a full-code agent designer can remember what was tried, what regressed, and when a repair is actually confirmed.

    Understand it in 90 seconds
    Problem
    Automated agent design is usually candidate-centric. Each round re-reads candidate code, scores, and trajectories, but it does not explicitly remember whether the same failure has already been repaired, which intervention worked, or which change caused a regression.
    Core insight
    Make the issue being repaired, rather than the candidate agent, the persistent control state. Each issue has a stable identity, priority, supporting evidence, lifecycle status, and intervention-outcome history.
    Strongest evidence
    ADIAS is evaluated on Tau-Bench, ALFWorld, TextCraft, WebShop, and ScienceWorld against five baselines. Table 1 reports an average score of 78.4 versus 62.6 for DGM-H, with shared wrappers, splits, action interfaces, scoring scripts, a ten-iteration optimization budget, and 15 training episodes per iteration (paper Section 4 and Table 1).
    Main boundary
    The paper holds trajectory diagnosis and issue association fixed rather than measuring their accuracy separately, and evaluates only text-based interactive benchmarks. The GitHub repository still says Coming Soon as of 2026-08-12, so this article does not treat a paper-level code claim as a currently reproducible artifact.
    Read the full deep dive

  • DocMemo: Letting Long-Document RAG Recover from a Bad First Retrieval

    Advanced Retrieval, memory, and production RAG
    Note: Aug 12, 2026 , Paper: 2026 , AI Engineering

    A deep reading of DocMemo: document schema, page belief, and question episodic memory preserve retrieval state across rounds, while Bayesian updates, Thompson sampling, and adaptive granularity recover missed evidence.

    Understand it in 90 seconds
    Problem
    Evidence in long documents can be spread across dozens of pages, tables, figures, and cross-page clues. Static retrieval fixes a top-k page pool at the start; if the first pass misses evidence, the reasoner has no state that explains which pages may still help, which were ruled out, or what remains missing.
    Core insight
    Turn retrieval into dynamic evidence exploration. Document Schema Memory stores document structure, Page Belief Memory updates page relevance beliefs, and Question Episodic Memory records discoveries and query refinement for the current question.
    Strongest evidence
    On MMLongBench-Doc, LongDocURL, and PaperTab, DocMemo reports accuracy of 71.3, 81.1, and 80.4, for a 77.6 average. Table 4 also shows MMLongBench-Doc accuracy falling from 71.3 to 68.5 or 68.8 when memory or Bayesian updating is removed.
    Main boundary
    Evaluation depends on a GPT-4.1 binary judge, PDF rendering, Qwen3.5-VL-9B, ColQwen2.5, MinerU, and annotations from three benchmarks. It does not establish citation faithfulness, access-control correctness, freshness, or total cost on arbitrary enterprise corpora.
    Read the full deep dive

  • A²E: A Traceable, Re-Evaluable Engine for Agent Auditing

    Intermediate Agent runtime, safety, and evaluation
    Agent Auditing: Part 1 , Note: Aug 11, 2026 , Paper: 2026 , AI Engineering

    A deep reading of A²E: ATP aligns benchmarks with agent harnesses, span-based traces preserve execution causality, and lifecycle-aligned metrics analyze correctness, tools, cost, and safety.

    Understand it in 90 seconds
    Problem
    A correct final answer does not tell you whether an agent took a reliable, cheap, or safe path. A wrong answer does not tell you whether the failure came from planning, tool use, memory, judging, or runtime. When each harness stores its own text log, cross-framework comparison and later metric iteration become difficult.
    Core insight
    A²E separates Task, Monitor, and Evaluation. The Agent Task Protocol (ATP) separates benchmark tasks from harness execution; the Monitor turns model calls, tool calls, state, and errors into parent-child traces; Evaluation organizes process, outcome, and runtime metrics under one lifecycle-aligned taxonomy.
    Strongest evidence
    The experiment covers 23 benchmarks, 9 harnesses, 5 tasks per cell, and 1,035 scored runs while holding the DeepSeek-V4-pro FP4 backbone, inference configuration, tool setup, step limit, and timeout fixed. Section 6 reports success-rate gaps of 0.20 on GDPVal, 0.30 on MMLU-Pro, and 0.66 on tau³-bench.
    Main boundary
    This is a platform architecture and diagnostic demonstration, not a universal ranking of nine harnesses. The prose and displayed tasksucceeded/correctness values in Table 2 conflict; paper commit, judge calibration, API drift, and component-level ablations are not fixed enough for strong causal claims.
    Read the full deep dive

  • ContextWeave Deep Read: Does Memory Actually Make Agents Better at Work?

    Advanced Agent runtime, safety, and evaluation
    Agent Evaluation: Part 2 , Note: Aug 7, 2026 , Paper: 2026 , AI Agent

    A close reading of how ContextWeave reconstructs multi-month workflows into an executable benchmark and measures memory's effect on workspace outcomes, preference adherence, continuity, and misleading recall.

    Understand it in 90 seconds
    Problem
    memory benchmarks often count whether history is retrieved, not whether it makes the next executable task better.
    Core insight
    reconstruct multi-month workflows as fixed executable task streams, then change only access to prior trajectories for the same target task. Measure workspace quality and preference adherence, not a retrieval hit alone.
    Strongest evidence
    with 14 participants and 1,005 reconstructed tasks (568 core evaluation tasks), the strongest component raises Workspace Score from 68.08 to 78.20 and Preference Score from 41.50 to 70.60 (Section 5.2, Table 2).
    Main boundary
    Docker reconstruction, mock APIs, and LLM-based rubrics make the comparison controlled; they do not establish the uplift of every memory implementation on live enterprise data or drifting tools.
    Read the full deep dive

  • Argus Deep Read: Long-Running Agents Need a Runtime, Not a Longer Prompt

    Advanced Agent runtime, safety, and evaluation
    Multi-Agent Coordination: Part 1 , Note: Aug 7, 2026 , Paper: 2026 , AI Agent

    A critical reading of Argus's Manager–Planner–Engineer–Reviewer runtime, durable state, verification-gated evolution, and rollback, separating benchmark results from author-operated case studies and unproven self-learning claims.

    Understand it in 90 seconds
    Problem
    a longer prompt does not give a long-running agent explicit task authority, auditable state, verification gates, or recovery boundaries.
    Core insight
    Argus cycles Manager, Planner, Engineer, and Reviewer over durable project state; memory, skills, routing, and procedures persist only after role-owned review.
    Strongest evidence
    across seven task-native arenas, the report gives breadth evidence and reports 78% for Argus versus 59% for Direct Copilot on SWE-Bench Pro under GPT-5.5, at roughly 1.41x aggregate tokens (Figure 1; Section 5).
    Main boundary
    this is an arXiv v1 technical report. The implementation, prompts, traces, checkpoints, and complete benchmark package are not public, so the reported outcome is not a reproducible adoption proof.
    Read the full deep dive

  • AgentS4D Deep Read: The Task Finished—Is the Runtime Safe?

    Advanced Agent runtime, safety, and evaluation
    Agent Security: Part 1 , Note: Aug 7, 2026 , Paper: 2026 , AI Agent

    A critical reading of how AgentS4D places workspace-agent risk entry, induction strategy, target harm, and lifecycle evidence in one sandbox benchmark, and why completion rate cannot stand in for safety.

    Understand it in 90 seconds
    Problem
    a workspace agent can finish its task while creating unsafe side effects through prompts, skills, files, web content, memory, or user messages.
    Core insight
    AgentS4D evaluates the complete harness–LLM–task environment, crossing risk source, induction strategy, harm, and execution-lifecycle evidence while scoring completion separately from safety.
    Strongest evidence
    328 risk-injected cases across 20 harness/backend configurations yield 6,560 runs; 4,461 (68.0%) trigger a prespecified unsafe signal and 4,344 (66.22%) are both unsafe and complete (Section 4; Table 2).
    Main boundary
    assets, effects, and cases are synthetic/controlled, and v1 has no executable code or data. These rates are not production incident rates or a universal safety ranking.
    Read the full deep dive

  • Real-Time Detection and Repair of LLM Agent Failures: A Deep Read of AgentTrajectorySentinel

    Advanced Agent runtime, safety, and evaluation
    Agent Evaluation: Part 3 , Note: Aug 7, 2026 , Paper: 2026 , AI Agent

    A critical reading of AgentTrajectorySentinel's low-cost healthy-only temporal monitor, deterministic verification, and rollback-and-retry loop, separating measured detection and repair gains from calibration dependence, content blind spots, and reproducibility limits.

    Understand it in 90 seconds
    Problem
    agent failures begin before the final answer; an LLM judge at every step can be too slow and costly.
    Core insight
    a temporal monitor trained on healthy trajectories works with deterministic verification; supported interventions roll state back to a trusted checkpoint for targeted retry.
    Strongest evidence
    across 2,823 committed episodes, three frameworks, and several models, the repair study compares monitor, verifier, and policy and reports task success from 52% to 73% (Section 5; Table 4).
    Main boundary
    healthy-only calibration, short trajectories, injected failures, and weak textual-hallucination detection limit transfer to a new production stack.
    Read the full deep dive

  • Before Reasoning Can Fail: Pre-Evidence Procedural Failures in Agentic RAG

    Advanced Retrieval, memory, and production RAG
    Production RAG Controls: Part 1 , Note: Aug 7, 2026 , Paper: 2026 , Retrieval Systems

    A deep read of how Before Reasoning Can Fail turns answer-before-reading into an observable trajectory failure, and tests whether Read-Gate actually improves multi-hop QA.

    Understand it in 90 seconds
    Problem
    an agentic RAG system can search snippets but finalize before reading. That is a procedural failure before evidence-conditioned reasoning, distinct from being wrong after reading gold evidence.
    Core insight
    saved tool traces, retrieved/read passages, and final answers define discipline and post-gold-read failure; Read-Gate requires at least one read after search and before final, without changing model, retriever, or reasoning budget.
    Strongest evidence
    12,000 paired trajectories over HotpotQA, 2WikiMultiHopQA, and MuSiQue; forced reading adds 14.9–19.9 LLM-Acc points on the zero-read subset and 3.2–9.4 on full minimal-reasoning cells (Table 1; Section 5.2).
    Main boundary
    it applies to systems with observable search/read/final actions; reading does not guarantee the right evidence or reasoning, and incomplete MuSiQue gold chunks limit post-gold-read analysis.
    Read the full deep dive

  • PAST-Bench: What Did a Persistent Agent Actually Learn from the Past?

    Advanced Agent runtime, safety, and evaluation
    Agent Evaluation: Part 4 , Note: Aug 7, 2026 , Paper: 2026 , AI Agent

    A deep read of how PAST-Bench uses fresh-session task families, matched persistence controls, and trace-level mechanism evidence to separate genuine retained-experience gains from higher scores with unrelated causes.

    Understand it in 90 seconds
    Problem
    a persistent agent's later score can improve because of model, prompt, task difficulty, or residual context—not because it used prior experience correctly.
    Core insight
    PAST-Bench uses fresh-session task families, holds prompt, grader, and tool stack fixed, and switches persistence on/off while reporting task-score gap and write/read/artifact mechanism evidence.
    Strongest evidence
    26 scenarios, 204 episodes, four capabilities, seven models, and four frameworks; Hermes+ reports its overall gap from +0.13 to +0.15 and Mech from 0.64 to 0.73 (Table 2; Section 4.3).
    Main boundary
    the gap difference is smaller than run-to-run variation, tasks are authored by the proposing team, and matched ablation is a strong control rather than complete causal proof.
    Read the full deep dive

  • RubricRanker Deep Read: RAG Needs the Right Document Set, Not Just the Most Relevant Documents

    Advanced Retrieval, memory, and production RAG
    Retrieval Systems: Part 3 , Note: Aug 7, 2026 , Paper: 2026 , Retrieval Systems

    A close reading of how RubricRanker uses query-specific search rubrics, SFT, and GRPO to train a document reranker, and what its deep-research and RAG benchmark results actually establish.

    Understand it in 90 seconds
    Problem
    Traditional rerankers score documents independently, so the top k need not be complete, concise, consistent, or authoritative as a set.
    Core insight
    Change the output target from a document ranking to an evidence set that jointly supports the answer, using query-specific rubrics for labels and rewards.
    Strongest evidence
    Tables 1–3 show downstream gains, while the ablation points to rubric labels and cold-start SFT rather than RL alone.
    Main boundary
    Final answers are still produced by agents and scored by LLM judges; a better evidence set does not guarantee correct citation, reasoning, or facts.
    Read the full deep dive

  • OSReward Deep Read: Why Agent Success Cannot Be Judged by Another Model Alone

    Advanced Agent runtime, safety, and evaluation
    Agent Evaluation: Part 1 , Note: Aug 2, 2026 , Paper: 2026 , AI Agent

    A complete reading of OSReward's data construction, 27 VLM judges, Hard and Multi subsets, error and cost analyses, OS-Shepherd-100K training, and a deployable hybrid verification architecture.

    Understand it in 90 seconds
    Previous approach's limitation
    Traditional model judges infer success from compressed screens and text histories, making them vulnerable to an agent's completion narrative.
    Core insight
    Use a human-gold benchmark to expose false-success bias, then separate verifiable state, model judgment, and human arbitration into distinct evidence layers.
    Strongest evidence
    Table 1 and Figures 5–7 show judges near 90% on the full set falling to roughly 70% on the Hard set, with directional failure-recall and cross-platform errors.
    Main boundary
    OS-Shepherd improves cost and some accuracy, but its labels still come from strong-judge agreement and the full artifact and production-verifier stack is incomplete.
    Read the full deep dive

For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.

Speaking & contact