Paper reading

BTS-AgentBench: Compiling Read-Only Telemetry into Replayable Agent Episodes

AI Engineering Advanced Published Sep 17, 2026 Reading time 25 min read Updated Sep 17, 2026

Telemetry to Agent Evaluation · Part 1 /1

At a glance

TL;DR

The paper's primary contribution is not a claim that one model is best. It is a construction method that compiles fixed building telemetry and metadata into executable, replayable, evidence-bearing multi-turn Agent benchmark episodes.
The central layering is read-only tool store → static executable task → typed interaction contract → operator-facing episode → deterministic verifier. Every clarification, revision, timestamp policy, or quality decision that adds telemetry work is re-executed against the source store.
The 532-row BTS release passes contract preflight with zero findings; two independent raw-to-episode builds match 11 logical tool-store exports and the 356/87/89 split. One-shot test runs report 79/89 for GPT-5.5, 71/89 for Gemini 3.1 Pro, and 58/89 for Claude Opus 4.7.
The decisive boundary is read-only, offline, bounded building telemetry. The paper does not evaluate write-side control, safety-critical actuation, maintenance planning, or long-horizon troubleshooting, and benchmark scores are not production safety evidence.

The paper's primary contribution is not a claim that one model is best. It is a construction method that compiles fixed building telemetry and metadata into executable, replayable, evidence-bearing multi-turn Agent benchmark episodes.
The central layering is read-only tool store → static executable task → typed interaction contract → operator-facing episode → deterministic verifier. Every clarification, revision, timestamp policy, or quality decision that adds telemetry work is re-executed against the source store.

Show all points

The 532-row BTS release passes contract preflight with zero findings; two independent raw-to-episode builds match 11 logical tool-store exports and the 356/87/89 split. One-shot test runs report 79/89 for GPT-5.5, 71/89 for Gemini 3.1 Pro, and 58/89 for Claude Opus 4.7.
The decisive boundary is read-only, offline, bounded building telemetry. The paper does not evaluate write-side control, safety-critical actuation, maintenance planning, or long-horizon troubleshooting, and benchmark scores are not production safety evidence.

Who this is for

AI engineers designing Agent benchmarks, tool-use harnesses, data agents, or enterprise evaluation workflows
Technical owners who need source data, interaction contracts, evidence, replay, and failure diagnosis joined into an auditable pipeline

Who this is for

AI engineers designing Agent benchmarks, tool-use harnesses, data agents, or enterprise evaluation workflows
Technical owners who need source data, interaction contracts, evidence, replay, and failure diagnosis joined into an auditable pipeline

The paper in 90 seconds

Problem: Buildings accumulate years of sensor and equipment telemetry, but a raw history is not an executable multi-turn task for an Agent. Hand-authoring each task makes it difficult to preserve a site’s vocabulary, source-derived answers, split identity, and evidence links at the same time.
Core insight: Treat benchmark construction as a replayable compiler. First place metadata and histories behind read-only tools; then build a static executable task with fixed golds; finally wrap that computation in a typed, bounded interaction contract. Clarification, goal revision, nearest-timestamp policy, quality decisions, and evidence follow-ups can change the surface, but the source computation and its gold must be re-executed together.
Strongest evidence: Two independent raw-to-episode builds match all 11 logical tool-store exports and regenerate the BTS 356/87/89 train/dev/test release row by row; all 532 released episodes pass coded contract preflight. This supports construction consistency, not operator realism or production deployment (paper Table 7 and Appendix A.3).
Main boundary: BTS-AgentBench is a read-only, offline, bounded building-telemetry benchmark. Its zero controller success is a construction-exclusion condition, not an independent hardness estimate; XAI4HEAT’s 41/41 result shows execution on a second telemetry corpus, not portability to arbitrary event logs or physical control.

My bounded verdict is: the durable idea is to make the source of an answer, the interaction obligations, and the replayable scorer one executable contract. That is useful for building and auditing a constrained read-only telemetry benchmark. Once the question becomes device writes, authorization, or real-site safety, this paper does not provide enough evidence.

What exactly does the paper introduce? / Paper identity and scope

This reading uses the complete arXiv 2608.27334 v1 HTML and v1 PDF. The version was submitted on 2026-08-27 by Jeong-Yoon Kim. The arXiv record identifies it as a preprint; it should not be described as peer-reviewed. In paper type, it is both a dataset/evaluation paper and a systems paper: it describes a method for constructing a benchmark from source telemetry and releases one instance called BTS-AgentBench. The released benchmark is not a separate model algorithm.

The reader question is: how can an already collected, site-specific telemetry corpus become an executable, verifiable, replayable Agent benchmark with bounded operator context? Answering it requires keeping four objects separate:

The source corpus is BTS metadata plus timestamp/value histories. It carries site, equipment, zone, point-class, and stream identifiers.
A static task is a fixed query, tool call, gold, evidence bundle, verifier, and split produced by the read-only runtime.
An episode places that static task inside a finite user simulator and adds trackable clarification, revision, policy, commitment, or evidence phases.
A model trace is the Agent’s tool calls and answers produced during evaluation. It does not participate in task retention and cannot repair the benchmark after the fact.

The paper calls the source task $r$ and the episode $r^{\star}$ : the latter changes the Agent-facing interface while preserving the source computation and target. The star does not guarantee that the episode is more realistic; it marks an interface transformation whose source semantics should remain visible. Each episode combines a deterministic user simulator, read-only tools, an Agent trace, and a programmatic evaluator. Success requires the telemetry target and the interaction obligations, with no mutable final database state (Section 3.1).

Why the prior approach is insufficient / Why the obvious alternatives are insufficient

Rich telemetry is not an executable task

BTS contains long time series and standardized Brick metadata, but “the data exists” does not mean there is a queryable, answerable, scoreable episode. An Agent answering a point lookup, daily mean, window comparison, or quality decision needs stable stream binding, tool arguments, a timestamp policy, acceptable answer forms, and evidence naming the contributing stream.

Hand-writing these conversations creates two scaling problems. First, generic Agent data cannot supply the local vocabulary of a particular site. Second, if every row is authored independently, adding a clarification or revision turn can make the user prompt, tool call, gold, evidence, and verifier drift apart. The construction method avoids asking a model to generate prettier prose for every row; it keeps those fields inside a replayable contract.

The candidate space is large, so retention must be fixed first

In paper Table 1, BTS metadata contains 19,665 streams, of which 14,422 match raw histories and become tool-ready points. The candidate space includes 2,193,431 day-mean candidates, 315,929 window-mean candidates, and 5,989,083 window-pairwise candidates. The final diversity-capped release keeps 60 rows for point disambiguation, day mean, relative 24-hour mean, window pairwise comparison, window rank, timestamp value, and timestamp nearest; 53 window-mean rows; and 59 quality-gate rows, for 532 total.

Those candidate counts describe the search space induced by the tool store and task generator, not task counts in the original BTS release. BTS_C is held out for testing. Other candidates are diversity-capped by point class and calendar quarter, then a fixed ordering assigns every fifth non-held-out candidate to development. Model output and a descriptive difficulty proxy do not affect retention or split assignment (Section 3.2 and Table 1). This is what makes the evaluation a test on a fixed release rather than a split tuned after seeing model answers.

Core intuition: compile the contract before the Agent interacts

The easiest misreading is to treat the paper as a templating tool for synthetic dialogue. A better mental model is:

BTS metadata + raw histories
  → normalized read-only tool store
  → static executable task
  → typed interaction contract
  → deterministic operator surface
  → programmatic verifier and replay report

The changed control point is the boundary between source computation and the interaction contract. A surface renderer may hide an exact timestamp behind a recoverable clarification slot, or extend a lookup into “use the nearest observation and explain the quality.” But when a new turn introduces telemetry work, that operation is executed against the same read-only store, and its phase gold, final target, evidence, and verifier are updated together. Language presents the contract; it does not replace the computation.

The episode contract in Section 4.2 is:

$C=(Q,\Phi,A,E,V), \qquad \phi_i=(f_i,g_i,R_i)$

Here $Q$ stores the interaction mode, missing slots, and operator turns; $\Phi=(\phi_1,\ldots,\phi_n)$ is the phase sequence; $f_i$ is a phase type; $g_i$ is its structured gold map; $R_i$ lists the scoring fields; $A$ contains canonical and acceptable read-only calls; $E$ identifies contributing streams; and $V$ contains milestones, tolerances, and protocol conditions. The value of this notation is operational: what a turn says and what the evaluator must check remain one object.

The contract transformation can be summarized as: