Tool Use & Coding Agents
43 reading notes
Study how tool discovery, schema context, coding tasks, and verifiable execution jointly shape agent performance.
Reader question
When an agent must choose tools, edit code, and execute actions, how can we reduce context and operational errors?
Reading library
Read this topic in depth
Every paper reading currently filed under this research topic.
-
TruthInsightBench: Evaluating Whether AI Agents Can Make Evidence-Grounded Scientific Discoveries
Advanced Agent runtime, safety, and evaluationTruthInsightBench shifts evaluation from reproducing a prescribed answer to examining whether an agent's own scientific claims are traceable, robust, controlled, and falsifiable. This reading explains its blind tasks, 29-item scoring rubric, and four-scaffold results, while examining the limits of one frozen base model, single-run evaluations, and an uncalibrated LLM judge.
Understand it in 90 seconds
-
MLOps-Bench Reading: Can Agents Complete Production ML Operations?
Intermediate Agent runtime, safety, and evaluationMLOps-Bench places coding agents in 31 real open-source ML repositories and tests whether they can make executable changes across data, training, serving, monitoring, and governance. This reading explains how tasks are grounded in repository evidence, what six agents did on 187 held-out tasks, and why a 31.2% task completion rate beside a 68.7% test pass rate matters.
Understand it in 90 seconds
-
WetRobo: How Coding Agents Adapt Lab Robots
Advanced Agent runtime, safety, and evaluationWetRobo packages a transferable robot setup, operating rules, and inspectable code so a coding agent can adapt programs in two biological laboratories. This reading examines three narrow tasks, the cross-lab bottle-cap comparison, and what the evidence cannot establish.
Understand it in 90 seconds
-
Ask the Tool, Don't Guess: How Progress Signals Change KV Cache Scheduling
Advanced Agent runtime, safety, and evaluationTool duration depends on the live environment, so names and call histories cannot reliably predict remaining work. This reading explains how Ask the Tool, Don't Guess recovers progress from tool output and execution traces, turns it into KV cache retention, offload, and reload hints, and bounds the reported TTFT gains to the authors' four-H100 experiment.
Understand it in 90 seconds
-
Agent Planning Benchmark: Diagnosing Planning and Failure in LLM Agents
Advanced Agent runtime, safety, and evaluationA close reading of APB’s 4,209 cases and five planning settings: how it separates long-horizon plans, feedback-conditioned actions, tool noise, tool failure, and refusal—and what constructed tasks, LLM judging, and controlled execution checks leave unresolved.
Understand it in 90 seconds
-
SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness
Advanced Agent runtime, safety, and evaluationSoL-Pi treats harness improvement as a family of auto-research loops with frozen metrics, capability gates, and independent validation. This reading explains the four retained mechanisms, the held-out design, author-reported token and cost estimates, and the limits of transfer and deployment claims.
Understand it in 90 seconds
-
ScienceIDE: Scientific Code as Verifiable Agent Environments
Advanced Agent runtime, safety, and evaluationA critical reading of ScienceIDE's 64 environments, 2,812 generated tasks, and separate 85-task hard benchmark: how reusable scientific checks support evaluation and training, and where private verifiers, partial release, task selection, and transfer claims remain bounded.
Understand it in 90 seconds
-
MAGS Paper Reading: Formal Verification and the Boundary of Safety Specs
Advanced Agent runtime, safety, and evaluationA deep reading of how MAGS translates agent-generated programs into Dafny, proves them against frozen specifications, and compiles them back; the reading separates 220 formal certificates from independent safety and functional results.
Understand it in 90 seconds
-
GameLogicBench: Testing Coding Agents Against Runtime Game Logic
Intermediate Agent runtime, safety, and evaluationGameLogicBench places coding agents inside Godot projects and checks rules at every simulation tick across multiple scenarios and seeds. This reading explains how the benchmark calibrates its judge, what failures it observes, and why its results do not establish reliability for general software agents.
Understand it in 90 seconds
-
Closed-World Resolution: Why Tool Identity Must Precede Authorization
Intermediate Agent runtime, safety, and evaluationA close reading of Closed-World Resolution Against Tool Hallucination in LLM Agents: how fabricated tools, invalid arguments, and MCP namespace collisions can bypass ordinary gates, and where the evidence stops.
Understand it in 90 seconds
-
EfficientAgent Reading: KV-Cache Offloading for Concurrent Agents
Advanced Agent runtime, safety, and evaluationEfficientAgent asks when moving a KV cache from GPU memory to host memory actually pays off. The answer depends not on one request alone, but on the reuse working set created by concurrent agents between uses. This reading examines capacity prediction, write admission, SWE-bench replay, and hardware boundaries.
Understand it in 90 seconds
-
Completed Pairs Hide Capped Failures: Stopping Rules and Unknown Outcomes in Paired Evaluation
Advanced Agent runtime, safety, and evaluationA single ReVerPi source-reading campaign shows how a runner can suppress a companion arm after the first arm reaches its request cap. Finite-frame bounds and stratified cost accounting reveal what completed-pair summaries omit, without making a population claim about context projection.
Understand it in 90 seconds
-
Can LLM Agents Tamper With Their Own Traces? Who Controls the Evidence Boundary
Intermediate Agent runtime, safety, and evaluationA close reading of arXiv v1: ten local coding-agent harness pairs, four controlled routes to trace tampering, and the distinct evidentiary roles of native traces, an external file observer, and tool-effect records.
Understand it in 90 seconds
-
Do Agent Skills Help with Version-Specific Plugin Migration? Scores and Contract Audits in a Retrospective Study
Intermediate Agent runtime, safety, and evaluationA close reading of the arXiv v1 comparison of a dsh plugin migration skill: what the static diagnostic scores show, how a contract counterexample qualifies them, and where task concentration, LLM judges, and artifact coverage set limits.
Understand it in 90 seconds
-
Can Authorization Hold After an Agent Requests Payment? A Deep Reading of APort Vault
Advanced Agent runtime, safety, and evaluationA deep reading of APort Vault that separates payment requests, successful payments, policy decisions, recipient membership, and unpermitted transfers while examining what a pre-action authorization layer does and does not establish.
Understand it in 90 seconds
-
How to Shrink an Agent Regression Benchmark Without Distorting It
Advanced Agent runtime, safety, and evaluationA deep reading of Trajectory-Aware Benchmark Subset Selection: historical outcome strata, trajectory sanitization, geometric selection, 76 configurations, temporal validation, cost, and transfer limits.
Understand it in 90 seconds
-
XYEval: Why agents follow bad advice
Advanced Agent runtime, safety, and evaluationA critical reading of Wu et al.'s XYEval (arXiv 2609.23939 v1): controlled XY mutations across five models and six benchmark suites, with trace analyses of how misleading suggestions affect task completion, communication, and tool trajectories—and the limits of generators and judges.
Understand it in 90 seconds
-
Prime Agent: How a Self-Improving RLM Harness Keeps Long-Run State Outside the Model
Advanced Agent runtime, safety, and evaluationA deep read of Prime Agent (arXiv 2608.23552 v1): its L0–L3 state hierarchy, persistent REPL, recursive subagents, and Continual Harness—alongside author-reported evaluations, a specification exploit retained through refinement, and the unsandboxed execution boundary.
Understand it in 90 seconds
-
SIFT: Turning Expensive Self-Improvement Evaluation into Cheap Ranking and Deferred Verification
Advanced Agent runtime, safety, and evaluationA deep reading of Self Improvement via Fast Tree-search (arXiv:2609.19526): pairwise LLM judging, regularized Bradley–Terry ranking, and asynchronous tree search decide which agent patches deserve expensive benchmark verification.
Understand it in 90 seconds
-
Bounded Agents: When Agent Security Is an Authorization-Architecture Problem
Advanced Agent runtime, safety, and evaluationA deep read of Bounded Agents (arXiv:2608.15888 v1): the Agentic Principal Chain, six conjunctive authorization conditions, and composition closure for constraining delegated authority and cross-step side effects—together with the complete-restriction, serialized-admission, enforcement, and utility boundaries.
Understand it in 90 seconds
-
After the Party: Governing What a Viral Agent-Skill Ecosystem Left Behind
Advanced Agent runtime, safety, and evaluationA deep reading of After the Party’s OpenClaw and ClawHub ecosystem study: 91 days of explosive growth, download concentration, the reviewability gap, privilege evidence, scanner disagreement, and the governance method that can transfer without pretending the rates do.
Understand it in 90 seconds
-
SilentProbe: When HTTP 200 Did Not Answer the Question
Advanced Agent runtime, safety, and evaluationA deep reading of SilentProbe (arXiv:2609.00035 v1): from OpenAPI constraint gaps and live differential probes to agent false negatives, separating how disclosure and machine-readability determine whether a tool fails honestly.
Understand it in 90 seconds
-
BTS-AgentBench: Compiling Read-Only Telemetry into Replayable Agent Episodes
Advanced Agent runtime, safety, and evaluationA deep read of Jeong-Yoon Kim's BTS-AgentBench (arXiv:2608.27334 v1): a deterministic path from building telemetry to read-only tools, executable tasks, bounded interaction contracts, and evidence-grounded evaluation—strong on replay consistency, bounded beyond production safety or arbitrary-domain portability.
Understand it in 90 seconds
-
ACE: Let a Canvas Agent Understand Structure Before It Corrects Itself
Advanced Agent runtime, safety, and evaluationA deep reading of ACE (arXiv:2608.24103 v1): hierarchical scene graphs, CARE routing, and an instruction-following judge turn multi-slide editing into a scoped, diffable, rollback-aware loop, with explicit limits around benchmarks, human raters, mock mode, and live reproduction.
Understand it in 90 seconds
-
Tool Call Succeeds, Workflow Fails: External-Effect Anomalies at the Agent–Tool Boundary
Advanced Agent runtime, safety, and evaluationA deep reading of the effect-history model behind Agent–Tool Boundary: why a successful tool response cannot guarantee a coherent external world state, and what MCP annotations and transactional contracts still leave unspecified.
Understand it in 90 seconds
-
EvoOntology: Turning a Static Data-Agent Semantic Layer into a Verifiable, Self-Evolving Interface
Advanced Retrieval, memory, and production RAGA deep reading of EvoOntology: an MCP ontology layer built by evidence-grounded probing, then refined through attribution-guided typed edits and a backbone-conditional paired gate.
Understand it in 90 seconds
-
Corrupt Plans, Clean Traces: How Plan Injection Evades Chain-of-Thought Monitoring
Advanced Agent runtime, safety, and evaluationA deep reading of Plan Injection: when an adversarial plan enters context and an actor rewrites it into plausible Chain-of-Thought, why the monitor’s clean trace can disconnect from behavioral causality, and where APPS, Bio-Math, and budget-sweep evidence stops.
Understand it in 90 seconds
-
DRACO: Sending Long-Horizon Agent Credit Back to the Steps
Advanced Agent runtime, safety, and evaluationA source-grounded reading of DRACO (arXiv:2609.04094): dynamic per-trajectory rubrics create an outcome-blind reward, then a closed-form rule redistributes GRPO advantage to the steps cited by the judge.
Understand it in 90 seconds
-
CONTINUITY: Keeping provenance, authorization, and tool effects continuous across Agent composition
Advanced Agent runtime, safety, and evaluationA critical reading of Zheng and Yang's CONTINUITY (arXiv:2609.05269 v1): security-context contracts, field-level provenance, transformation witnesses, and effect-bound permits for preserving an LLM Agent's instruction-to-effect boundary.
Understand it in 90 seconds
-
Parsing the Stream: Long-Horizon Agents Need Auditable Live State, Not Just Memory
Advanced Agent runtime, safety, and evaluationA critical reading of Pakhomov and Nijkamp's Parsing the Stream (arXiv:2609.01466): an append-only trace is folded into typed RunState and compiled into observer and worker views. The paper reports gains on specific accumulation tasks and monitoring costs, but does not show that fixed aggregates replace every form of trace memory.
Understand it in 90 seconds
-
Indirect Prompt Injection: Web Pages and Tool Returns Become Instruction Channels, but 2023 Cases Do Not Represent Later Guard Products
Intermediate Agent runtime, safety, and evaluationA source-grounded reading of Greshake et al., arXiv:2302.12173 v2: when LLM-integrated apps retrieve web pages, email, or tool output, untrusted data enters the prompt as if it were instructions. The authors demonstrate indirect prompt injection on Bing Chat, GitHub Copilot, and synthetic GPT-4 apps and give a computer-security threat taxonomy. This is 2023 control-plane evidence, not a Llama-Guard, Constitutional AI, OWASP Top-10, or jailbreak-benchmark product SLA.
Understand it in 90 seconds
-
ReAct: Interleave Thought and Action, but Do Not Treat a Few-Shot Loop as an Agent Runtime
Intermediate Agent runtime, safety, and evaluationA source-grounded reading of Yao et al., ICLR 2023: language thoughts join the action space, while HotpotQA, FEVER, ALFWorld, and WebShop keep hallucination, search failure, and the abstract's +34% / +10% in separate buckets.
Understand it in 90 seconds
-
Toolformer: Self-Supervised API Calls Are Not an Agent Loop
Intermediate Agent runtime, safety, and evaluationA source-grounded reading of Schick et al., NeurIPS 2023: future-token loss filters QA, Wikipedia, calculator, calendar, and translation calls on CCNet for GPT-J. LAMA and math jump; this is still not a chainable agent runtime.
Understand it in 90 seconds
-
SWE-bench: Real GitHub Issues as Evaluation, but 1.96% Is Not a Model Ceiling
Intermediate Agent runtime, safety, and evaluationA source-grounded reading of Jimenez et al., ICLR 2024 Oral: the evaluation unit is a real GitHub issue, a full Python repository, and tests. Claude 2 resolves 1.96% under BM25; that number is a protocol, not a model ranking.
Understand it in 90 seconds
-
Reflexion: Write Verbal Reflections into Memory — Do Not Mistake Retries for Weight Learning
Intermediate Agent runtime, safety, and evaluationA deep read of Shinn et al., NeurIPS 2023: frozen weights plus linguistic feedback stored in episodic memory for across-trial verbal credit assignment. HumanEval pass@1 91.0 vs GPT-4 80.1 is a programming setup with self-tests and retries; WebShop and MBPP mark the boundary.
Understand it in 90 seconds
-
MemGPT: Treat Context as Paged Memory — Do Not Mistake the OS Metaphor for Enterprise Memory
Intermediate Agent runtime, safety, and evaluationA deep read of Packer et al., arXiv:2310.08560 v2: finite context as RAM, OS-like tiers, and function-mediated paging. DMR moves GPT-4 from 32.1% to 92.5%; nested KV shows multi-hop lookup—not ACL memory governance.
Understand it in 90 seconds
-
CoT: Make the Model Write the Reasoning, but Do Not Treat It as an Agent That Moves
Intermediate Agent runtime, safety, and evaluationA source-grounded reading of Wei et al., NeurIPS 2022: few-shot exemplars with intermediate steps elicit multi-step reasoning in large frozen models. PaLM 540B on GSM8K moves from 17.9 to 56.9; this is still a prompt, not tools, an environment, or memory paging.
Understand it in 90 seconds
-
WebGPT: Let the Model Browse for Answers, but Do Not Treat It as a Reasoning Agent Loop
Intermediate Agent runtime, safety, and evaluationA source-grounded reading of Nakano et al., arXiv:2112.09332 v3: GPT-3 is given a text browser and trained with human demonstrations and preference / reward modeling to search, quote, and answer. The 175B best-of-64 model is preferred 56% versus demonstrators and 69% versus Reddit; this is browsing QA, not ReAct’s thought–action–observation contract.
Understand it in 90 seconds
-
Gorilla: Turn a Large API Catalog into Retrievable Tools, but APIBench Does Not Establish MCP Product Behavior
Intermediate Agent runtime, safety, and evaluationA source-grounded reading of Patil et al., NeurIPS 2024: retriever-aware finetuning of LLaMA-7B on APIBench (TorchHub / TensorHub / HuggingFace) so catalog-scale API calls can be retrieved and checked. Zero-shot overall accuracy and hallucination beat prompted GPT-4 on that table—this is not a ReAct loop, MidTool mid-training, or RAG-MCP product routing.
Understand it in 90 seconds
-
MidTool: Does Teaching Tool Use During Mid-Training Make Agents More Reliable?
Advanced Agent runtime, safety, and evaluationA deep reading of MidTool, which moves schema grounding, workflow composition, and recovery under incomplete information into a 20.3B-token mid-training mixture—while web search remains at 0%.
Understand it in 90 seconds
-
SWE-Bench ProMax: Can Large-Scale Multilingual Refactoring Measure Long-Horizon Coding Agents?
Advanced Agent runtime, safety, and evaluationA deep reading of SWE-Bench ProMax, which uses 170 cross-file, multilingual, behavior-preserving refactoring tasks to test whether coding agents can complete large changes rather than merely fix a nearby test.
Understand it in 90 seconds
-
Argus Deep Read: Long-Running Agents Need a Runtime, Not a Longer Prompt
Advanced Agent runtime, safety, and evaluationA critical reading of Argus's Manager–Planner–Engineer–Reviewer runtime, durable state, verification-gated evolution, and rollback, separating benchmark results from author-operated case studies and unproven self-learning claims.
Understand it in 90 seconds
-
RAG-MCP: Retrieve Tool Discovery, but Account for Routing Failure
Intermediate Retrieval, memory, and production RAGA source-grounded reading of RAG-MCP's tool-routing pipeline, 11,100-tool stress test, MCPBench result, scale failure, and incomplete artifacts.
Understand it in 90 seconds
For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.
Speaking & contact