Agent Tool Contracts and Silent Failure
Series · 1 posts
-
SilentProbe: When HTTP 200 Did Not Answer the Question
AdvancedA deep reading of SilentProbe (arXiv:2609.00035 v1): from OpenAPI constraint gaps and live differential probes to agent false negatives, separating how disclosure and machine-readability determine whether a tool fails honestly.
Understand it in 90 seconds
- Problem
- When an agent calls a third-party API, an empty result can mean that no record matched, or that the server did not understand a filter, discarded it, and still returned HTTP 200 with parseable JSON. Neither case necessarily provides an exception, an error status, or a field on which to branch.
- Core insight
- Separate two questions. Disclosure asks whether the model can select a value accepted by the vendor from the description; machine-readability asks whether a validator can reject a wrong value before the request leaves the gateway. Only the second can be enforced by generic infrastructure.
- Strongest evidence
- In the 721,320 parameter leaves of the public OpenAPI corpus, only 7.5% declare an enum and 15.2% declare any machine-checkable constraint; 40.1% of documents contain at least one prose constraint gap. In 219 live perturbations, machine-checkable constraints produced 111/111 honest errors, while prose-only constraints produced 44/61 silent failures (Sections 4.1–4.2, Figure 5).
- Main boundary
- In the three-parameter agent experiment, a description that showed only 1/18 department values led to silent failure in 88/88 attempts; promoting that vocabulary into an enum produced 0/89 silent failures. This is evidence under the interface and harness used by the paper, not a general law about models or every production API.
For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.
Speaking & contact