Agent Evaluation & Observability
78 reading notes
Examine whether agents are actually reliable through task outcomes, trajectories, runtime signals, and benchmarks.
Reader question
Beyond the final answer, which signals must we observe and verify before trusting that an agent completed its work?
Reading library
Read this topic in depth
Every paper reading currently filed under this research topic.
-
TruthInsightBench: Evaluating Whether AI Agents Can Make Evidence-Grounded Scientific Discoveries
Advanced Agent runtime, safety, and evaluationTruthInsightBench shifts evaluation from reproducing a prescribed answer to examining whether an agent's own scientific claims are traceable, robust, controlled, and falsifiable. This reading explains its blind tasks, 29-item scoring rubric, and four-scaffold results, while examining the limits of one frozen base model, single-run evaluations, and an uncalibrated LLM judge.
Understand it in 90 seconds
-
StepGuard Paper Reading: Guarding Agent Actions Step by Step
Intermediate Agent runtime, safety, and evaluationA close reading of how StepGuard trains a 4B runtime guard from prefix-aligned synthetic trajectories and uses Balance-GRPO to adjust safety and task utility. We examine the 77.3% relative ASR reduction and 2.8-point utility decrease on AgentDojo and AgentDyn, along with synthetic supervision, failure modes, latency, and the absence of a formal guarantee.
Understand it in 90 seconds
-
RetrievalRouter Reading: Choosing a Retrieval Route for Each Query
Intermediate Retrieval, memory, and production RAGRetrievalRouter routes each query among BM25, text-vector, multimodal, and reranking pipelines. This reading explains its soft-target policy, accuracy-latency results across 11 datasets, and the deployment costs and generalization questions its offline benchmark leaves open.
Understand it in 90 seconds
-
MLOps-Bench Reading: Can Agents Complete Production ML Operations?
Intermediate Agent runtime, safety, and evaluationMLOps-Bench places coding agents in 31 real open-source ML repositories and tests whether they can make executable changes across data, training, serving, monitoring, and governance. This reading explains how tasks are grounded in repository evidence, what six agents did on 187 held-out tasks, and why a 31.2% task completion rate beside a 68.7% test pass rate matters.
Understand it in 90 seconds
-
Librarian Agents: Preparing Reusable Workspaces Before Questions Arrive
Intermediate Retrieval, memory, and production RAGLibrarian Agents moves part of document retrieval before query time: a reinforcement-trained builder creates a navigable, provenance-linked filesystem for a separate answerer. This reading examines the 50 held-out BrowseComp-Plus corpus tasks, a distinct 443-question synthetic reuse set, BRIGHT transfer, token trade-offs, and the claims the evidence does not yet establish.
Understand it in 90 seconds
-
WetRobo: How Coding Agents Adapt Lab Robots
Advanced Agent runtime, safety, and evaluationWetRobo packages a transferable robot setup, operating rules, and inspectable code so a coding agent can adapt programs in two biological laboratories. This reading examines three narrow tasks, the cross-lab bottle-cap comparison, and what the evidence cannot establish.
Understand it in 90 seconds
-
Ask the Tool, Don't Guess: How Progress Signals Change KV Cache Scheduling
Advanced Agent runtime, safety, and evaluationTool duration depends on the live environment, so names and call histories cannot reliably predict remaining work. This reading explains how Ask the Tool, Don't Guess recovers progress from tool output and execution traces, turns it into KV cache retention, offload, and reload hints, and bounds the reported TTFT gains to the authors' four-H100 experiment.
Understand it in 90 seconds
-
PACT: Can Enterprise AI Assistants Be Trusted Under Pressure? A Reading on Rule Following
Advanced Agent runtime, safety, and evaluationPACT tests whether enterprise AI assistants follow rules through ordinary pressure, multi-turn pushback, and cases where a rule does not apply. This reading explains its six evaluation axes, reported results, and limits as evidence.
Understand it in 90 seconds
-
Agent Planning Benchmark: Diagnosing Planning and Failure in LLM Agents
Advanced Agent runtime, safety, and evaluationA close reading of APB’s 4,209 cases and five planning settings: how it separates long-horizon plans, feedback-conditioned actions, tool noise, tool failure, and refusal—and what constructed tasks, LLM judging, and controlled execution checks leave unresolved.
Understand it in 90 seconds
-
An Empirical Study of Harness Design for Coding Agents: Planning, Tools, and Context Management
Advanced Agent runtime, safety, and evaluationWhen does a coding agent benefit from planning, a structured tool interface, or context management, and when do these components only add cost? Read the conditional effects across four models, two benchmarks, and 176 matched settings.
Understand it in 90 seconds
-
SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness
Advanced Agent runtime, safety, and evaluationSoL-Pi treats harness improvement as a family of auto-research loops with frozen metrics, capability gates, and independent validation. This reading explains the four retained mechanisms, the held-out design, author-reported token and cost estimates, and the limits of transfer and deployment claims.
Understand it in 90 seconds
-
TRACE: Accountable Source Discovery in Digital Archives
Advanced Retrieval, memory, and production RAGTRACE turns retrieval over OCR-degraded French archives into an inspectable agent workflow: corpus-targeted decomposition, fused search, deferred relevance decisions, and cross-question reranking. Read its results on 1,752 questions alongside the limits of the public data version, internal deployment claims, and cost estimate.
Understand it in 90 seconds
-
Refuse, Decompose, Refresh: Claim Boundaries in Closed-Loop AI Evaluation
Advanced Agent runtime, safety, and evaluationA close reading of an evaluation protocol that abstains when comparisons lack observable support, reports separate decisions, and treats distribution shift as reference invalidation. The article explains its simulator thresholds, denominators, held-out evidence, and deployment boundary.
Understand it in 90 seconds
-
ScienceIDE: Scientific Code as Verifiable Agent Environments
Advanced Agent runtime, safety, and evaluationA critical reading of ScienceIDE's 64 environments, 2,812 generated tasks, and separate 85-task hard benchmark: how reusable scientific checks support evaluation and training, and where private verifiers, partial release, task selection, and transfer claims remain bounded.
Understand it in 90 seconds
-
Finding the Right Fit: How Models, Harnesses, and Tasks Change Agent Rankings
Advanced Agent runtime, safety, and evaluationA comparison of 66 model–harness configurations across three task collections, with score reversals, cost trade-offs, and failure-feedback traces. What these observations can—and cannot—tell us about deployment choices.
Understand it in 90 seconds
-
AgentPProf: Profiling Long-Horizon Agent Trajectories Across Runs
Advanced Agent runtime, safety, and evaluationAgentPProf maps agent prompts, tool actions, and system effects into semantic operation stacks that can be aggregated across sessions. The paper reports 0.764 B³ F1 and MAP gains on three localization benchmarks; a flame graph reveals concentrations, not causal proof.
Understand it in 90 seconds
-
EconSkills: How Web-Agent Skills Transfer and Get Retrieved
Advanced Agent runtime, safety, and evaluationEconSkills turns successful economic-data browsing traces into parameterized SOPs with scope, verification, and recovery steps. A known match transfers; library-scale retrieval remains constrained by coverage and approximate mismatches.
Understand it in 90 seconds
-
MAGS Paper Reading: Formal Verification and the Boundary of Safety Specs
Advanced Agent runtime, safety, and evaluationA deep reading of how MAGS translates agent-generated programs into Dafny, proves them against frozen specifications, and compiles them back; the reading separates 220 formal certificates from independent safety and functional results.
Understand it in 90 seconds
-
GameLogicBench: Testing Coding Agents Against Runtime Game Logic
Intermediate Agent runtime, safety, and evaluationGameLogicBench places coding agents inside Godot projects and checks rules at every simulation tick across multiple scenarios and seeds. This reading explains how the benchmark calibrates its judge, what failures it observes, and why its results do not establish reliability for general software agents.
Understand it in 90 seconds
-
Loopjacking: How Human Approval Loses Its Binding to an Action
Advanced Agent runtime, safety, and evaluationA deep reading of Loopjacking: how incomplete approval representations and post-approval state substitution detach a human decision from the executed side effect, and what version-pinned product traces do and do not establish.
Understand it in 90 seconds
-
Agents Are Systems, Not Models: Why Agent Evaluation Must Go Beyond Model Scores
Advanced Agent runtime, safety, and evaluationThis AI-for-science study varies task information, reasoning, verification, runtime budget, and backbone. In its main experiments, repeat-run noise accounts for about 54% of score variance among genuine attempts. The result is important—but covers only four specialist-model tasks, not every kind of agent.
Understand it in 90 seconds
-
When Upstream Messages Override Correct Answers: How Wrong Answers Displace Downstream Judgment
Intermediate Agent runtime, safety, and evaluationA controlled handoff study across five benchmarks and five receivers tests when an incorrect upstream answer displaces a downstream answer that was independently correct—and what the experiment does not establish.
Understand it in 90 seconds
-
Raven Paper Reading: How a Harness of Harnesses Plans Multi-Agent Work
Advanced Agent runtime, safety, and evaluationRaven composes model–harness pairs as specialists and asks a Host Agent to plan a dependency DAG. This reading unpacks harness composition, the MAOB benchmark, and what its +10.4/+10.5 percentage-point result actually measures: graph planning match, not successful worker execution.
Understand it in 90 seconds
-
EfficientAgent Reading: KV-Cache Offloading for Concurrent Agents
Advanced Agent runtime, safety, and evaluationEfficientAgent asks when moving a KV cache from GPU memory to host memory actually pays off. The answer depends not on one request alone, but on the reuse working set created by concurrent agents between uses. This reading examines capacity prediction, write admission, SWE-bench replay, and hardware boundaries.
Understand it in 90 seconds
-
Stale-Document Poisoning: How RAG Can Recognize Evidence That No Longer Applies
Advanced Retrieval, memory, and production RAGAuthentic, once-correct documents can make RAG overturn an answer the model already got right. This reading examines 317 cross-domain knowledge reversals, the difference between dates and validity intervals, causal interventions, and a reranker whose gains depend on trustworthy metadata.
Understand it in 90 seconds
-
Completed Pairs Hide Capped Failures: Stopping Rules and Unknown Outcomes in Paired Evaluation
Advanced Agent runtime, safety, and evaluationA single ReVerPi source-reading campaign shows how a runner can suppress a companion arm after the first arm reaches its request cap. Finite-frame bounds and stratified cost accounting reveal what completed-pair summaries omit, without making a population claim about context projection.
Understand it in 90 seconds
-
Who Signs Off? Let Specifications, Not Agents, Decide When a Task Is Done
Intermediate Agent runtime, safety, and evaluationA deep reading of how SpecHarness compiles agent-visible instructions into source-linked obligations and commits state only from qualified evidence, with an examination of SkillsBench and GuideBench results, runtime costs, and limits on attributing gains to sign-off alone.
Understand it in 90 seconds
-
Can LLM Agents Tamper With Their Own Traces? Who Controls the Evidence Boundary
Intermediate Agent runtime, safety, and evaluationA close reading of arXiv v1: ten local coding-agent harness pairs, four controlled routes to trace tampering, and the distinct evidentiary roles of native traces, an external file observer, and tool-effect records.
Understand it in 90 seconds
-
Do Agent Skills Help with Version-Specific Plugin Migration? Scores and Contract Audits in a Retrospective Study
Intermediate Agent runtime, safety, and evaluationA close reading of the arXiv v1 comparison of a dsh plugin migration skill: what the static diagnostic scores show, how a contract counterexample qualifies them, and where task concentration, LLM judges, and artifact coverage set limits.
Understand it in 90 seconds
-
The Source Link Is Intact, but the Rule Is Stale: How Agents Miss Updates Under a Verification Budget
Advanced Agent runtime, safety, and evaluationA deep reading of When Stale Constraints Go Unchecked: immutable provenance, append-only supersession, and stale-consistent decisions under a two-record verification budget. Audits four runs, the held-out correction, denominators, and the Zenodo artifact while bounding the synthetic evidence.
Understand it in 90 seconds
-
Can Authorization Hold After an Agent Requests Payment? A Deep Reading of APort Vault
Advanced Agent runtime, safety, and evaluationA deep reading of APort Vault that separates payment requests, successful payments, policy decisions, recipient membership, and unpermitted transfers while examining what a pre-action authorization layer does and does not establish.
Understand it in 90 seconds
-
How to Shrink an Agent Regression Benchmark Without Distorting It
Advanced Agent runtime, safety, and evaluationA deep reading of Trajectory-Aware Benchmark Subset Selection: historical outcome strata, trajectory sanitization, geometric selection, 76 configurations, temporal validation, cost, and transfer limits.
Understand it in 90 seconds
-
How Can a Vulnerability Report Become Executable Evidence? A Deep Read of MobileCybench
Advanced Agent runtime, safety, and evaluationA deep read of MobileCybench: 13 Android apps and 495 executable security probes used to evaluate five coding agents, with careful distinctions among probe triggers, vulnerability attribution, maintainer confirmation, and the limits of the evidence.
Understand it in 90 seconds
-
XYEval: Why agents follow bad advice
Advanced Agent runtime, safety, and evaluationA critical reading of Wu et al.'s XYEval (arXiv 2609.23939 v1): controlled XY mutations across five models and six benchmark suites, with trace analyses of how misleading suggestions affect task completion, communication, and tool trajectories—and the limits of generators and judges.
Understand it in 90 seconds
-
Does a Correct RAG Answer Mean the System Behaved Correctly? Reading The RAT
Advanced Retrieval, memory, and production RAGA deep read of The RAT: how a joint Bayesian model separates retrieval, abstention, and answer correctness, and what its annotation-budget results, judge calibration, partial retrieval analysis, and controlled Wikipedia scope do and do not establish.
Understand it in 90 seconds
-
Prime Agent: How a Self-Improving RLM Harness Keeps Long-Run State Outside the Model
Advanced Agent runtime, safety, and evaluationA deep read of Prime Agent (arXiv 2608.23552 v1): its L0–L3 state hierarchy, persistent REPL, recursive subagents, and Continual Harness—alongside author-reported evaluations, a specification exploit retained through refinement, and the unsandboxed execution boundary.
Understand it in 90 seconds
-
SIFT: Turning Expensive Self-Improvement Evaluation into Cheap Ranking and Deferred Verification
Advanced Agent runtime, safety, and evaluationA deep reading of Self Improvement via Fast Tree-search (arXiv:2609.19526): pairwise LLM judging, regularized Bradley–Terry ranking, and asynchronous tree search decide which agent patches deserve expensive benchmark verification.
Understand it in 90 seconds
-
BioPhys-Bridge: Making Scientific RAG Walk from Evidence to Physics, Mechanism, and Decision
Advanced Retrieval, memory, and production RAGA deep reading of BioPhys-Bridge (arXiv:2609.19180 v1): a benchmark that makes evidence IDs, values and units, physical equations, assumptions, biological mechanisms, and next decisions explicit, while separating attribution from scientific correctness.
Understand it in 90 seconds
-
RAFT: Making Troubleshooting RAG Retrieve the State That Led to a Fix
Advanced Retrieval, memory, and production RAGA deep reading of RAFT: A Stateful Retrieval-Augmented Framework for Troubleshooting Agents (arXiv:2609.20754): closed support cases become state-transition trajectories, intermediate entries are retrieved, and the parent case is returned with its matched anchor.
Understand it in 90 seconds
-
Learning When to Trust: Measuring and Training Selective Trust with Selective Context Preference Optimization
Advanced Retrieval, memory, and production RAGA deep read of Learning When to Trust via Selective Context Preference Optimization: MIST holds each reasoning item fixed while varying clean, misleading, correct-context, and irrelevant-context signals; SC2W isolates clean-correct flips, and SCOPE reduces them with balanced DPO preference quartets while retaining the paper's text-only, contamination, and deployment-prevalence boundaries.
Understand it in 90 seconds
-
Bounded Agents: When Agent Security Is an Authorization-Architecture Problem
Advanced Agent runtime, safety, and evaluationA deep read of Bounded Agents (arXiv:2608.15888 v1): the Agentic Principal Chain, six conjunctive authorization conditions, and composition closure for constraining delegated authority and cross-step side effects—together with the complete-restriction, serialized-admission, enforcement, and utility boundaries.
Understand it in 90 seconds
-
HyPE: Moving the RAG Question–Document Gap into Indexing
Intermediate Retrieval, memory, and production RAGA deep reading of Bridging the Question-Answer Gap in Retrieval-Augmented Generation: Hypothetical Prompt Embeddings. HyPE precomputes hypothetical questions for every chunk and turns query-to-document retrieval into question-to-question matching, with strong six-dataset results but real indexing, freshness, chunking, and single-generator boundaries.
Understand it in 90 seconds
-
After the Party: Governing What a Viral Agent-Skill Ecosystem Left Behind
Advanced Agent runtime, safety, and evaluationA deep reading of After the Party’s OpenClaw and ClawHub ecosystem study: 91 days of explosive growth, download concentration, the reviewability gap, privilege evidence, scanner disagreement, and the governance method that can transfer without pretending the rates do.
Understand it in 90 seconds
-
Predicting Partial Answer Quality and Utility in Agentic Retrieval-Augmented Generation
Advanced Retrieval, memory, and production RAGA deep reading of Predicting Partial Answer Quality and Utility in Agentic Retrieval-Augmented Generation: turning intermediate answer quality, utility, and trajectory signals into an early-stopping controller, then checking the savings and transfer boundary.
Understand it in 90 seconds
-
SilentProbe: When HTTP 200 Did Not Answer the Question
Advanced Agent runtime, safety, and evaluationA deep reading of SilentProbe (arXiv:2609.00035 v1): from OpenAPI constraint gaps and live differential probes to agent false negatives, separating how disclosure and machine-readability determine whether a tool fails honestly.
Understand it in 90 seconds
-
RAGSieve: Detecting RAG Knowledge-Poisoning Promotion with Self-Referenced Local Contrast
Advanced Retrieval, memory, and production RAGA deep read of RAGSieve: query-local and corpus-local references expose suspicious retrieval promotion without a trusted clean corpus, while the paper's boundary remains essential—promotion detection is not truth verification.
Understand it in 90 seconds
-
BTS-AgentBench: Compiling Read-Only Telemetry into Replayable Agent Episodes
Advanced Agent runtime, safety, and evaluationA deep read of Jeong-Yoon Kim's BTS-AgentBench (arXiv:2608.27334 v1): a deterministic path from building telemetry to read-only tools, executable tasks, bounded interaction contracts, and evidence-grounded evaluation—strong on replay consistency, bounded beyond production safety or arbitrary-domain portability.
Understand it in 90 seconds
-
When Failure Propagates, Can We Still Find the Start? Causal Failure Attribution in Agentic RAG
Advanced Retrieval, memory, and production RAGA deep reading of When Failures Propagate: an interventional benchmark, three-hop MuSiQue, and certified content corruption that separate failure detection, causal attribution, propagation, and recovery in agentic RAG.
Understand it in 90 seconds
-
ACE: Let a Canvas Agent Understand Structure Before It Corrects Itself
Advanced Agent runtime, safety, and evaluationA deep reading of ACE (arXiv:2608.24103 v1): hierarchical scene graphs, CARE routing, and an instruction-following judge turn multi-slide editing into a scoped, diffable, rollback-aware loop, with explicit limits around benchmarks, human raters, mock mode, and live reproduction.
Understand it in 90 seconds
-
Tool Call Succeeds, Workflow Fails: External-Effect Anomalies at the Agent–Tool Boundary
Advanced Agent runtime, safety, and evaluationA deep reading of the effect-history model behind Agent–Tool Boundary: why a successful tool response cannot guarantee a coherent external world state, and what MCP annotations and transactional contracts still leave unspecified.
Understand it in 90 seconds
-
EvoOntology: Turning a Static Data-Agent Semantic Layer into a Verifiable, Self-Evolving Interface
Advanced Retrieval, memory, and production RAGA deep reading of EvoOntology: an MCP ontology layer built by evidence-grounded probing, then refined through attribution-guided typed edits and a backbone-conditional paired gate.
Understand it in 90 seconds
-
Corrupt Plans, Clean Traces: How Plan Injection Evades Chain-of-Thought Monitoring
Advanced Agent runtime, safety, and evaluationA deep reading of Plan Injection: when an adversarial plan enters context and an actor rewrites it into plausible Chain-of-Thought, why the monitor’s clean trace can disconnect from behavioral causality, and where APPS, Bio-Math, and budget-sweep evidence stops.
Understand it in 90 seconds
-
K-Bench: Why Agentic Unlearning Cannot Be Certified from the Final Answer Alone
Advanced Agent runtime, safety, and evaluationA deep read of Yu et al.'s K-Bench (arXiv:2609.12808 v1): an end-to-end agent deployment benchmark that evaluates six observable channels across four memory substrates, using OR-of-channels leakage, collapse-aware K-Scores, and pre-registered statistical tests to separate forgetting from channel migration and agent collapse.
Understand it in 90 seconds
-
REVA: Moving RAG compression into reusable evidence views instead of paying per request
Advanced Retrieval, memory, and production RAGA critical reading of Nguyen et al.'s REVA (arXiv 2609.11209 v1): historical generator attention becomes a document-keyed score store, separating offline scoring from online rendering while exposing unseen-document fallback, local/global budgets, quality, and latency boundaries.
Understand it in 90 seconds
-
VikingRAG: Fewer Retrieval Rounds, Less Context Waste for Structured-Document RAG
Advanced Retrieval, memory, and production RAGA deep reading of VikingRAG: hierarchy-preserving URI-addressable storage, Search/List/Grep/Read tools, reusable experience edges, and adaptive escalation for reducing repeated retrieval tokens and latency.
Understand it in 90 seconds
-
DRACO: Sending Long-Horizon Agent Credit Back to the Steps
Advanced Agent runtime, safety, and evaluationA source-grounded reading of DRACO (arXiv:2609.04094): dynamic per-trajectory rubrics create an outcome-blind reward, then a closed-form rule redistributes GRPO advantage to the steps cited by the judge.
Understand it in 90 seconds
-
CONTINUITY: Keeping provenance, authorization, and tool effects continuous across Agent composition
Advanced Agent runtime, safety, and evaluationA critical reading of Zheng and Yang's CONTINUITY (arXiv:2609.05269 v1): security-context contracts, field-level provenance, transformation witnesses, and effect-bound permits for preserving an LLM Agent's instruction-to-effect boundary.
Understand it in 90 seconds
-
Parsing the Stream: Long-Horizon Agents Need Auditable Live State, Not Just Memory
Advanced Agent runtime, safety, and evaluationA critical reading of Pakhomov and Nijkamp's Parsing the Stream (arXiv:2609.01466): an append-only trace is folded into typed RunState and compiled into observer and worker views. The paper reports gains on specific accumulation tasks and monitoring costs, but does not show that fixed aggregates replace every form of trace memory.
Understand it in 90 seconds
-
Generative Agents: Observe–Reflect–Plan in a Multi-Agent Sandbox — Do Not Mistake Sandbox Memory for MemGPT OS Paging
Intermediate Agent runtime, safety, and evaluationA deep read of Park et al., UIST 2023 / arXiv:2304.03442 v2: 25 agents in Smallville use a memory stream, periodic reflection, and retrieval-based planning. Interview ablations hit TrueSkill μ 29.89 vs 21.21 fully ablated; two-day sandbox diffusion and party coordination are qualitative evidence, not production runtime.
Understand it in 90 seconds
-
ReAct: Interleave Thought and Action, but Do Not Treat a Few-Shot Loop as an Agent Runtime
Intermediate Agent runtime, safety, and evaluationA source-grounded reading of Yao et al., ICLR 2023: language thoughts join the action space, while HotpotQA, FEVER, ALFWorld, and WebShop keep hallucination, search failure, and the abstract's +34% / +10% in separate buckets.
Understand it in 90 seconds
-
Toolformer: Self-Supervised API Calls Are Not an Agent Loop
Intermediate Agent runtime, safety, and evaluationA source-grounded reading of Schick et al., NeurIPS 2023: future-token loss filters QA, Wikipedia, calculator, calendar, and translation calls on CCNet for GPT-J. LAMA and math jump; this is still not a chainable agent runtime.
Understand it in 90 seconds
-
SWE-bench: Real GitHub Issues as Evaluation, but 1.96% Is Not a Model Ceiling
Intermediate Agent runtime, safety, and evaluationA source-grounded reading of Jimenez et al., ICLR 2024 Oral: the evaluation unit is a real GitHub issue, a full Python repository, and tests. Claude 2 resolves 1.96% under BM25; that number is a protocol, not a model ranking.
Understand it in 90 seconds
-
CoT: Make the Model Write the Reasoning, but Do Not Treat It as an Agent That Moves
Intermediate Agent runtime, safety, and evaluationA source-grounded reading of Wei et al., NeurIPS 2022: few-shot exemplars with intermediate steps elicit multi-step reasoning in large frozen models. PaLM 540B on GSM8K moves from 17.9 to 56.9; this is still a prompt, not tools, an environment, or memory paging.
Understand it in 90 seconds
-
WebGPT: Let the Model Browse for Answers, but Do Not Treat It as a Reasoning Agent Loop
Intermediate Agent runtime, safety, and evaluationA source-grounded reading of Nakano et al., arXiv:2112.09332 v3: GPT-3 is given a text browser and trained with human demonstrations and preference / reward modeling to search, quote, and answer. The 175B best-of-64 model is preferred 56% versus demonstrators and 69% versus Reddit; this is browsing QA, not ReAct’s thought–action–observation contract.
Understand it in 90 seconds
-
Gorilla: Turn a Large API Catalog into Retrievable Tools, but APIBench Does Not Establish MCP Product Behavior
Intermediate Agent runtime, safety, and evaluationA source-grounded reading of Patil et al., NeurIPS 2024: retriever-aware finetuning of LLaMA-7B on APIBench (TorchHub / TensorHub / HuggingFace) so catalog-scale API calls can be retrieved and checked. Zero-shot overall accuracy and hallucination beat prompted GPT-4 on that table—this is not a ReAct loop, MidTool mid-training, or RAG-MCP product routing.
Understand it in 90 seconds
-
SWE-Bench ProMax: Can Large-Scale Multilingual Refactoring Measure Long-Horizon Coding Agents?
Advanced Agent runtime, safety, and evaluationA deep reading of SWE-Bench ProMax, which uses 170 cross-file, multilingual, behavior-preserving refactoring tasks to test whether coding agents can complete large changes rather than merely fix a nearby test.
Understand it in 90 seconds
-
ADIAS: Turning Agent Self-Improvement into Traceable Issue Repair
Advanced Agent runtime, safety, and evaluationA deep reading of ADIAS: persistent issue state organizes failure evidence across optimization rounds so a full-code agent designer can remember what was tried, what regressed, and when a repair is actually confirmed.
Understand it in 90 seconds
-
DocMemo: Letting Long-Document RAG Recover from a Bad First Retrieval
Advanced Retrieval, memory, and production RAGA deep reading of DocMemo: document schema, page belief, and question episodic memory preserve retrieval state across rounds, while Bayesian updates, Thompson sampling, and adaptive granularity recover missed evidence.
Understand it in 90 seconds
-
Agentic Configuration Management: Treating Agent Systems as Governed Configuration, Not Just One Execution
Advanced Agent runtime, safety, and evaluationA deep reading of how ACM uses a framework-independent Configuration Graph, immutable revisions, dependency-aware impact propagation, and runtime provenance to govern heterogeneous agent configurations across LangGraph, CrewAI, and the OpenAI Agents SDK.
Understand it in 90 seconds
-
A²E: A Traceable, Re-Evaluable Engine for Agent Auditing
Intermediate Agent runtime, safety, and evaluationA deep reading of A²E: ATP aligns benchmarks with agent harnesses, span-based traces preserve execution causality, and lifecycle-aligned metrics analyze correctness, tools, cost, and safety.
Understand it in 90 seconds
-
ContextWeave Deep Read: Does Memory Actually Make Agents Better at Work?
Advanced Agent runtime, safety, and evaluationA close reading of how ContextWeave reconstructs multi-month workflows into an executable benchmark and measures memory's effect on workspace outcomes, preference adherence, continuity, and misleading recall.
Understand it in 90 seconds
-
Argus Deep Read: Long-Running Agents Need a Runtime, Not a Longer Prompt
Advanced Agent runtime, safety, and evaluationA critical reading of Argus's Manager–Planner–Engineer–Reviewer runtime, durable state, verification-gated evolution, and rollback, separating benchmark results from author-operated case studies and unproven self-learning claims.
Understand it in 90 seconds
-
AgentS4D Deep Read: The Task Finished—Is the Runtime Safe?
Advanced Agent runtime, safety, and evaluationA critical reading of how AgentS4D places workspace-agent risk entry, induction strategy, target harm, and lifecycle evidence in one sandbox benchmark, and why completion rate cannot stand in for safety.
Understand it in 90 seconds
-
Real-Time Detection and Repair of LLM Agent Failures: A Deep Read of AgentTrajectorySentinel
Advanced Agent runtime, safety, and evaluationA critical reading of AgentTrajectorySentinel's low-cost healthy-only temporal monitor, deterministic verification, and rollback-and-retry loop, separating measured detection and repair gains from calibration dependence, content blind spots, and reproducibility limits.
Understand it in 90 seconds
-
Before Reasoning Can Fail: Pre-Evidence Procedural Failures in Agentic RAG
Advanced Retrieval, memory, and production RAGA deep read of how Before Reasoning Can Fail turns answer-before-reading into an observable trajectory failure, and tests whether Read-Gate actually improves multi-hop QA.
Understand it in 90 seconds
-
PAST-Bench: What Did a Persistent Agent Actually Learn from the Past?
Advanced Agent runtime, safety, and evaluationA deep read of how PAST-Bench uses fresh-session task families, matched persistence controls, and trace-level mechanism evidence to separate genuine retained-experience gains from higher scores with unrelated causes.
Understand it in 90 seconds
-
RubricRanker Deep Read: RAG Needs the Right Document Set, Not Just the Most Relevant Documents
Advanced Retrieval, memory, and production RAGA close reading of how RubricRanker uses query-specific search rubrics, SFT, and GRPO to train a document reranker, and what its deep-research and RAG benchmark results actually establish.
Understand it in 90 seconds
-
OSReward Deep Read: Why Agent Success Cannot Be Judged by Another Model Alone
Advanced Agent runtime, safety, and evaluationA complete reading of OSReward's data construction, 27 VLM judges, Hard and Multi subsets, error and cost analyses, OS-Shepherd-100K training, and a deployable hybrid verification architecture.
Understand it in 90 seconds
For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.
Speaking & contact