Agent Evaluation

Series · 4 posts

← Back to Paper Reading

  • OSReward Deep Read: Why Agent Success Cannot Be Judged by Another Model Alone

    Advanced Agent runtime, safety, and evaluation
    Agent Evaluation · Part 1 · Note · Aug 2, 2026 · Paper · 2026 · AI Agent

    A complete reading of OSReward's data construction, 27 VLM judges, Hard and Multi subsets, error and cost analyses, OS-Shepherd-100K training, and a deployable hybrid verification architecture.

    Understand it in 90 seconds
    Previous approach's limitation
    Traditional model judges infer success from compressed screens and text histories, making them vulnerable to an agent's completion narrative.
    Core insight
    Use a human-gold benchmark to expose false-success bias, then separate verifiable state, model judgment, and human arbitration into distinct evidence layers.
    Strongest evidence
    Table 1 and Figures 5–7 show judges near 90% on the full set falling to roughly 70% on the Hard set, with directional failure-recall and cross-platform errors.
    Main boundary
    OS-Shepherd improves cost and some accuracy, but its labels still come from strong-judge agreement and the full artifact and production-verifier stack is incomplete.
    Read the full deep dive

  • ContextWeave Deep Read: Does Memory Actually Make Agents Better at Work?

    Advanced Agent runtime, safety, and evaluation
    Agent Evaluation · Part 2 · Note · Aug 7, 2026 · Paper · 2026 · AI Agent

    A close reading of how ContextWeave reconstructs multi-month workflows into an executable benchmark and measures memory's effect on workspace outcomes, preference adherence, continuity, and misleading recall.

    Understand it in 90 seconds
    Problem
    memory benchmarks often count whether history is retrieved, not whether it makes the next executable task better.
    Core insight
    reconstruct multi-month workflows as fixed executable task streams, then change only access to prior trajectories for the same target task. Measure workspace quality and preference adherence, not a retrieval hit alone.
    Strongest evidence
    with 14 participants and 1,005 reconstructed tasks (568 core evaluation tasks), the strongest component raises Workspace Score from 68.08 to 78.20 and Preference Score from 41.50 to 70.60 (Section 5.2, Table 2).
    Main boundary
    Docker reconstruction, mock APIs, and LLM-based rubrics make the comparison controlled; they do not establish the uplift of every memory implementation on live enterprise data or drifting tools.
    Read the full deep dive

  • Real-Time Detection and Repair of LLM Agent Failures: A Deep Read of AgentTrajectorySentinel

    Advanced Agent runtime, safety, and evaluation
    Agent Evaluation · Part 3 · Note · Aug 7, 2026 · Paper · 2026 · AI Agent

    A critical reading of AgentTrajectorySentinel's low-cost healthy-only temporal monitor, deterministic verification, and rollback-and-retry loop, separating measured detection and repair gains from calibration dependence, content blind spots, and reproducibility limits.

    Understand it in 90 seconds
    Problem
    agent failures begin before the final answer; an LLM judge at every step can be too slow and costly.
    Core insight
    a temporal monitor trained on healthy trajectories works with deterministic verification; supported interventions roll state back to a trusted checkpoint for targeted retry.
    Strongest evidence
    across 2,823 committed episodes, three frameworks, and several models, the repair study compares monitor, verifier, and policy and reports task success from 52% to 73% (Section 5; Table 4).
    Main boundary
    healthy-only calibration, short trajectories, injected failures, and weak textual-hallucination detection limit transfer to a new production stack.
    Read the full deep dive

  • PAST-Bench: What Did a Persistent Agent Actually Learn from the Past?

    Advanced Agent runtime, safety, and evaluation
    Agent Evaluation · Part 4 · Note · Aug 7, 2026 · Paper · 2026 · AI Agent

    A deep read of how PAST-Bench uses fresh-session task families, matched persistence controls, and trace-level mechanism evidence to separate genuine retained-experience gains from higher scores with unrelated causes.

    Understand it in 90 seconds
    Problem
    a persistent agent's later score can improve because of model, prompt, task difficulty, or residual context—not because it used prior experience correctly.
    Core insight
    PAST-Bench uses fresh-session task families, holds prompt, grader, and tool stack fixed, and switches persistence on/off while reporting task-score gap and write/read/artifact mechanism evidence.
    Strongest evidence
    26 scenarios, 204 episodes, four capabilities, seven models, and four frameworks; Hermes+ reports its overall gap from +0.13 to +0.15 and Mech from 0.64 to 0.73 (Table 2; Section 4.3).
    Main boundary
    the gap difference is smaller than run-to-run variation, tasks are authored by the proposing team, and matched ablation is a strong control rather than complete causal proof.
    Read the full deep dive

For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.

Speaking & contact