Agent Auditing

Series · 1 posts

← Back to Paper Reading

  • A²E: A Traceable, Re-Evaluable Engine for Agent Auditing

    Intermediate Agent runtime, safety, and evaluation
    Agent Auditing · Part 1 · Note · Aug 11, 2026 · Paper · 2026 · AI Engineering

    A deep reading of A²E: ATP aligns benchmarks with agent harnesses, span-based traces preserve execution causality, and lifecycle-aligned metrics analyze correctness, tools, cost, and safety.

    Understand it in 90 seconds
    Problem
    A correct final answer does not tell you whether an agent took a reliable, cheap, or safe path. A wrong answer does not tell you whether the failure came from planning, tool use, memory, judging, or runtime. When each harness stores its own text log, cross-framework comparison and later metric iteration become difficult.
    Core insight
    A²E separates Task, Monitor, and Evaluation. The Agent Task Protocol (ATP) separates benchmark tasks from harness execution; the Monitor turns model calls, tool calls, state, and errors into parent-child traces; Evaluation organizes process, outcome, and runtime metrics under one lifecycle-aligned taxonomy.
    Strongest evidence
    The experiment covers 23 benchmarks, 9 harnesses, 5 tasks per cell, and 1,035 scored runs while holding the DeepSeek-V4-pro FP4 backbone, inference configuration, tool setup, step limit, and timeout fixed. Section 6 reports success-rate gaps of 0.20 on GDPVal, 0.30 on MMLU-Pro, and 0.66 on tau³-bench.
    Main boundary
    This is a platform architecture and diagnostic demonstration, not a universal ranking of nine harnesses. The prose and displayed tasksucceeded/correctness values in Table 2 conflict; paper commit, judge calibration, API drift, and component-level ablations are not fixed enough for strong causal claims.
    Read the full deep dive

For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.

Speaking & contact