Tag: Evaluation
Posts with this tag
- Gemini Decisions API: Routing AI Agents with Probabilities and Confidence
Explore Gemini Decision 1's three question types, a Python routing example, confidence thresholds, and multimodal limits.
- Jev Beat Pokémon Red: Was It the Model or the Harness?
Using Jev's Pokémon Red run and public code, this article separates the decision model's role from game state, pathfinding, and recovery mechanisms in a long-running agent.
- AI Cosmos: Is Getting It Right with AI the Same as Learning It?
Use AI Cosmos's legal and programming studies to examine situated participation, executable workbenches, and the difference between earning points and mastering a target skill.
- OpenGray: Why Decision Accuracy Above 90% Can Hide Failures to Stop
Use OpenGray's known-answer tasks and planning logs to distinguish aggregate decision scores, escalation, dose-goal attainment, and deliverability.
- GravityOCR: Why 3.94× Faster Decoding Becomes a 1.32× Page Speedup
Understand GravityOCR's shared diffusion drafter and AR verifier through three timing boundaries, output-equivalence conditions, and OCR pipeline bottlenecks.
- How to Read the Terminal-Bench 4.0 Leaderboard: Scores, Agents, Cost, and Limits
Use a BenchLM snapshot to interpret Terminal-Bench configurations, resolution rates, confidence intervals, evaluation costs, and cross-leaderboard comparisons.
- Tencent’s Evaluation Agents Kept Trying Amap: Why Didn’t They Stop?
Swarmchasers tracked third-party model evaluation agents on Tencent Cloud attempting to query Amap; the episode exposes the boundary between network tools, stop conditions, and evaluation governance.
- Anthropic’s IoT Agent Replication Test: Eight Simulated Nodes, No Field Worm
What Anthropic’s isolated Android-emulator evaluation establishes about agent replication, what its unauthenticated ADB starting point leaves untested, and why the year-end capability estimate is only a projection.
- LLM Wiki Knowledge Updates: Turning Agent Proposals into Verifiable Knowledge Change PRs
A synthetic enterprise knowledge update exercise shows how to combine source tracking, impact analysis, answer regression, and human approval into a governed knowledge change lifecycle.
- Anthropic's OSS Scanner: Who Triages Unreviewed Vulnerability Reports?
Anthropic's opt-in OSS Scanner sends model-generated vulnerability reports to maintainers without human review, shifting first-line validation and triage to participating projects.
- Do Clinical AI Benchmarks Measure Exam Questions or Everyday Work?
A single-system study at NYU Langone compares 127,833 clinical AI requests with 58 public benchmarks and finds that their task mix differs from clinicians’ day-to-day work.
- Anthropic's Unintended Model Actions: Real Forms, Delayed Detection, and Agent Boundaries
Anthropic's account of Claude reaching real websites during evaluations shows why agent testing needs network isolation, action gates, and incident reporting.
- ParanoiaEval: When a Coding Agent Treats a Resolved Risk Again
ParanoiaEval uses paired tasks to test whether coding agents choose risk responses from evidence; this article examines its results, limits, and practical evaluation lessons.
- SWE-CC: Passing Tests Can Still Break Repository Rules
SWE-CC turns 823 contribution policies from 12 open-source projects into executable checks, showing what test results and resolve rates leave out of coding-agent evaluations.
- When AI Agents Turn Public Data Retrieval into Website Probing
Transluce’s archived requests to U.S. and Canadian government sites show how ordinary retrieval can become high-volume or exploit-shaped traffic, and what stop controls operators can test.
- Holo4: One Agent Across GUIs, Code, and Tools—with Different Licenses
A closer look at H Company’s cross-interface agent and long-horizon harness, its benchmark claims, public traces, and the licensing split between checkpoints.
- Target Retail Product Search: Constrain Semantic Recall with Precision and Experiments
An engineering analysis of Target's lexical and vector retrieval, attribute controls, weighted interleaving, and the evidence limits behind its reported gains.
- Turning Agent Risks into Runtime Policy: ASSERT Evaluation and ACS Enforcement
How Microsoft’s run-assert-eval connects Clarity risk discovery, ASSERT behavior tests, and ACS runtime policy, while measuring unsafe behavior separately from over-refusal.
- Project Swap: Agents Can Trade Without Knowing What Their People Want
Anthropic's low-stakes employee book exchange found that preference estimates constrained outcomes more than bargaining rules, making preference understanding a separate test for delegated agents.
- AI Agents Should Do More Than Agree: How the XY Problem Derails a Fix
XYEval shows how a plausible but misplaced user suggestion can lower agent success; the practical response is to verify the goal, then explain a better path with evidence.
- What Evidence Should an AI Agent's Vulnerability Report Include? MobileCybench Replays Executable Probes
MobileCybench replays Android agent exploits in an isolated environment, then checks trusted state with executable probes to see which security properties were violated.
- RSIAgent: Can an Agent Improve Without Updating Model Weights?
A focused engineering reading of RSIAgent's curriculum, actor, verifier, and broad-to-deep exploration loop, with a careful audit of frozen experience, benchmark reporting, and reproducibility limits.
- What Should We Measure When AI Starts Doing AI R&D? Anthropic's Three Dashboards
Anthropic proposes three measurements for AI-led R&D: automation, agent oversight, and safety compute; this article separates its internal self-report from methodology limits and cross-lab comparability.
- GitHub Agentic Workflows: Turning Runnable Agent CI into a Runtime Contract
A focused engineering reading of gh-aw v0.89.17: how log audits, MCP Gateway and firewall boundaries, grading, model-cost signals, and incident monitoring move Agentic CI beyond a demo.
- Pydantic AI v2.45: Durable Agent Reliability Is a Session and Trace Contract
A technical reading of how Pydantic AI v2.45.0 aligns DynamicToolset, MCP sessions, tool history, and usage spans with durable runs, plus the adoption boundaries for TypeSafeModel and Bedrock effort handling.
- Gemini Antigravity Agent 09-2026: An Agent Runtime Protocol Migration
A practical breakdown of Antigravity Agent 09-2026's remote/local compatibility boundary, built-in tool contract changes, adapter design, contract tests, and the migration risk before 05-2026 shuts down on October 5, 2026.
- Amazon SageMaker HyperPod Inference Gateway: Where GPU-Aware Routing Helps
An engineering analysis of how Amazon SageMaker HyperPod Inference Gateway uses KV cache, queue depth, LoRA, and prefix-cache signals, with the EKS add-on, CRD, failure, benchmark, and vendor-claim boundaries made explicit.
- How Claude Speeds Up Biomolecular Models: FlashPairformer and Reversible Inference Kits
An engineering reading of Anthropic's Claude-assisted optimization of more than 30 biomolecular and genomics models, from FlashPairformer and Big mode to stock/exact/fast contracts, cost curves, and evidence limits.
- TypeSafe AI and Jev: Turning AI into a Calibrated Decision Primitive
An engineering reading of TypeSafe AI's System One model and Jev: typed decisions, probability-aware workflows, evaluation claims, and the limits of replacing text generation with decision primitives.
- Jev in the Agent Runtime: Confidence-Gated Routing, Fan-Out, and Community Experiments
A practical architecture guide to placing Jev between agents, tools, and human review through confidence-gated routing, speculative fan-out, composite scoring, and careful evaluation.
- Gemini 3.8 Flash Coding-Agent Workflow: Routing Uncertainty from Planning to Execution
Using Astra planning and Flash execution as an example, this article adds a SPEC verification gate, escalation rules, and a careful reading of DeepSWE costs.
- Unified Knowledge Graph RAG: GraphRAG and LightRAG Are Query Policies, Not Global Switches
A systems reading of AWS's Unified Knowledge Graph RAG reference stack: how GraphRAG and LightRAG share ingestion, graph, hybrid retrieval, and lineage infrastructure while selecting a query strategy per question.
- Automated Alignment Researchers: Why Agentic Post-Training Needs Integrity Gates
Anthropic's automated alignment researcher experiment shows how agents can search and iterate on post-training methods while benchmarks, capability floors, data isolation, and integrity review remain outside the agent's authority.
- What LLM Inference Costs: DeepSeek-R1 Benchmarks and GPU Rental Prices
Read public DeepSeek-R1 MLPerf logs for 8 B200 and B300 GPUs, then apply sourced GPU rental quotes while separating measured latency, arithmetic cost scenarios, and API prices.
- How to Read AI Agent Papers: From CoT and WebGPT to ReAct
One diagram shows how CoT and WebGPT merge into ReAct, then connects Gorilla and IPI to the rest of the agent-systems reading path.
- How to Read RAG Papers: From Dense Retrieval (DPR) to Lewis RAG
One diagram shows how DPR and Lewis RAG connect to the retrieval papers already on this site.
- TREC RAG 2026: Why RAG Evaluation Is Adding Agents
Use TREC RAG 2026 to explain how RAG evaluation moved from document QA to agent-in-the-loop. This article covers direction and task design, not enterprise harness implementation.
- How to Build an Enterprise RAG Evaluation Harness (TREC RAG 2026)
Using TREC RAG 2026 and RAGDoll as references, design a replayable enterprise RAG evaluation harness: data model, citations, agent traces, judge calibration, and launch gates.
- What Is Anthropic Agent Memory: Cross-Session Memory vs Dreaming
Untangle Anthropic Agent Memory vs Dreaming: which handles cross-session recall, which runs overnight batches—and do not treat them as the same thing.
- Why Reinforcement Learning Breaks Small Language Models
Gradient freezing, numerical overflow, and policy collapse when aligning 70M–500M SLMs with PPO, and what the capacity-headroom hypothesis explains. This article covers failure modes—not a slogan about moving toward robustness.
- What Is AgentEscapeBench: Measuring Out-of-Domain Tool Reasoning
A deep read of AgentEscapeBench: why agents fail on out-of-domain, long tool chains, and what this benchmark can and cannot show.
- GPT-5.6 Prompting Checklist: From Shorter Prompts to Tool Orchestration
Turns GPT-5.6 official prompting into a checklist: lean prompts, tool orchestration, and when not to encode policy in prompts. This is the checklist—not the 15% article.
- Self-Scaffolding for Agentic Coding: Ornith 1.0 Training and Evaluation Limits
Read Ornith 1.0's self-scaffolding: what scaffolding solves in agentic coding, what benchmarks can prove, and where trust boundaries sit. Ornith is the case study, not the search entry point.
- AI Agent Guide: Architecture, Tools, Evaluation, and Production
A practical guide to agents versus workflows, single- and multi-agent architecture, tools and MCP, state and memory, evaluation, security, and the path from PoC to production.
- Enterprise RAG Guide: Retrieval Architecture, Evaluation, and Production Delivery
A practical framework for enterprise RAG data pipelines, hybrid search, reranking, GraphRAG, agentic RAG, evaluation, access governance, failure diagnosis, and operations.
- GPT-5.6 Sol Prompting: Why Shorter Prompts Score Higher
Reads OpenAI's official GPT-5.6 Sol prompting guidance: what the 15% shorter-prompt figure can and cannot support. This article explains why to write less—not a rules checklist.
- What Is GPT-5.6 Sol: Routing, Pricing, and Benchmarks
A product-oriented read of GPT-5.6 Sol: tiered pricing, model routing, and how to interpret benchmarks. This is not the architecture paper.
- OpenAI Deployment Simulation: Predicting LLM Safety Before Launch
Read OpenAI Deployment Simulation: why offline eval and real deployment diverge, and what this simulation can and cannot predict.
- Phil Schmid: Why Agent Harness Is More Important Than Model Leaderboards in 2026
Deep dive into the Jan 2026 article: durability, OS analogies, system evaluation gaps, lightweight Harness under the Bitter Lesson, and hill climbing alongside training-inference convergence.
- Harness Design for Long-Running AI Engineering: Generation, Evaluation, and Verification Chains
A source-grounded engineering guide to Anthropic's long-running application harness, including the planner-generator-evaluator contract, context reset trade-offs, and reported costs.
- Why Reasoning Models 'Cannot Control Their Own Train of Thought' — And Why That's Good News for AI Safety
OpenAI's latest research reveals that current frontier reasoning models are almost completely unable to hide or alter their Chain of Thought (CoT) based on instructions, with maximum controllability at only 15.4%. This 'flaw' is not a problem, but rather the key reason why current CoT monitoring mechanisms can be trusted.
For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.
Speaking & contact