Tag: Evaluation
Posts with this tag
- TREC RAG 2026: When RAG Evaluation Becomes an Agent Workflow
A concise entry point to TREC RAG 2026, ClimbMix-400b, and RAGDoll, with a bridge to a technical guide for building a replayable enterprise RAG evaluation harness.
- TREC RAG 2026 Technical Deep Dive: From Evidence Lineage to a Replayable RAG Evaluation Harness
A practical design for an enterprise RAG evaluation harness based on TREC RAG 2026 and RAGDoll: data models, execution stages, citation support, agent traces, judge calibration, and production gates.
- Anthropic's Memory and Dreaming for Continuous Agent Learning
Exploring Anthropic's underlying memory systems and the asynchronous Dreaming process for Claude agents to solve context management and continuous learning in multi-agent environments.
- Towards Robust Reinforcement Learning for Small-Scale Language Model Agents: Architecture and Practical Insights
An in-depth analysis of three common PPO failure modes (gradient freezing, numerical overflow, policy collapse) for 70M-500M SLMs, and the Capacity-Headroom Hypothesis (PPL < 20).
- AgentEscapeBench Deep Dive: Benchmarking Out-of-Domain Tool-Grounded Reasoning in LLM Agents
A technical analysis of arXiv:2605.07926 (AgentEscapeBench). Exploring how LLM agent performance degrades across long-range DAG tool dependency graphs, exposing bottlenecks in clue adherence, intermediate output propagation, and state tracking.
- OpenAI GPT-5.6 Prompting Guidance: From Lean Prompts to Programmatic Tool Orchestration
Unpacking OpenAI's official GPT-5.6 model guidance. Discover how leaner system prompts boost evaluation scores by 15% and cut costs by 67%, alongside autonomy boundaries, text.verbosity, Programmatic Tool Calling, and Pro mode.
- Ornith 1.0 and Self-Scaffolding: Training, Evaluation, and Trust Boundaries for Agentic Coding
A structured look at Ornith-1.0's Self-Scaffolding, anti-reward-hacking controls, and Pipeline-RL design, plus the evaluation and trust boundaries needed for agentic coding systems.
- AI Agent Guide: Architecture, Tools, Evaluation, and Enterprise Delivery
A practical guide to agents versus workflows, single- and multi-agent architecture, tools and MCP, state and memory, evaluation, security, and the path from PoC to production.
- Enterprise RAG Guide: Retrieval Architecture, Evaluation, and Production Delivery
A practical framework for enterprise RAG data pipelines, hybrid search, reranking, GraphRAG, agentic RAG, evaluation, access governance, failure diagnosis, and operations.
- Architecture and Practical Judgment! OpenAI Officially Releases GPT-5.6 Sol Prompting Guidance: Trimming Prompts Unexpectedly Yields a 15% Performance Boost
In-depth analysis of the latest "GPT-5.6 Sol Prompting Guidance" officially released by OpenAI. Officials confirmed for the first time: deleting lengthy rules and examples and streamlining the System Prompt not only reduces costs by 67% but also increases task evaluation scores by 10-15%. This article breaks down the brand new Model + Harness practices, Programmatic Tool Calling, and validation workflows.
- GPT-5.6 Sol Is Generally Available: Routing, Pricing, and Benchmark Caveats
An updated guide to Sol, Terra, and Luna after GPT-5.6 moved from limited preview to general availability, covering API specifications, pricing, benchmark limits, and adoption decisions.
- OpenAI Publishes "Deployment Simulation": Solving Evaluation Awareness and Better Predicting LLM Safety Before Release
An in-depth analysis of OpenAI's latest large language model safety evaluation method, "Deployment Simulation." This article explores how replaying historical prefixes of real user conversations can eliminate the "evaluation awareness" and test-taking behaviors of models found in traditional red-teaming, achieving highly accurate risk prediction for the GPT-5 series models. It provides a complete explanation using concise flowcharts and prediction graphs.
- Phil Schmid: Why Agent Harness Is More Important Than Model Leaderboards in 2026
Deep dive into the Jan 2026 article: durability, OS analogies, system evaluation gaps, lightweight Harness under the Bitter Lesson, and hill climbing alongside training-inference convergence.
- Harness Design for Long-Running AI Engineering: Generation, Evaluation, and Verification Chains
Based on Anthropic's 'Harness design for long-running application development': Improving the reliability and controllability of long-running tasks through generator-evaluator separation, external evaluation, and QA contracts.
- Why Reasoning Models 'Cannot Control Their Own Train of Thought' — And Why That's Good News for AI Safety
OpenAI's latest research reveals that current frontier reasoning models are almost completely unable to hide or alter their Chain of Thought (CoT) based on instructions, with maximum controllability at only 15.4%. This 'flaw' is not a problem, but rather the key reason why current CoT monitoring mechanisms can be trusted.
For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.
Speaking & contact