Comparing agent harnesses and tools for incident investigation

Cursor diagnosed 12 incidents with repository tools alone, 14 with Foam’s tool, and 9 with Sentry’s tool. Foam’s separate agent scored 18 of 22. The Cursor results compare specific tool setups. The Foam agent result also reflects its harness, prompts, and evidence access.^[1]

Comparison design and scoring

For each incident, an agent received a prompt and an isolated copy of the relevant codebase. It investigated without a user supplying the solution and wrote a root cause analysis. Each setup ran once per incident. A GPT-4o judge compared the answer with a human-written key and counted it as correct only if it identified the same incident, the same precise technical cause, and a cause that would lead an engineer toward the same fix. The evaluation measured diagnoses, not whether an agent produced a working code change.^[2]

Cursor ran opus-4.6 in all three of its setups: repository access alone, repository access with a Sentry tool, and repository access with Foam’s telemetry tool. The fourth setup used Foam’s agent, with Pi as its harness, terminal access, and direct Foam telemetry access. Foam tailored its prompts to favor using Foam’s tool and used a recursive investigation strategy informed by Recursive Language Models.^[3] This fourth setup changes several parts of the system at once.

Results

Within the Cursor harness, the Foam tool setup scored two more incidents than Cursor alone, a difference of 9.1 percentage points. The Sentry tool setup scored three fewer than Cursor alone. Foam’s agent had the highest score, but its 18 correct diagnoses include the effects of its harness, prompts, terminal access, and telemetry together.

What changed at the incident level

The net scores conceal cases that changed in both directions. Relative to Cursor alone, the Foam tool setup correctly diagnosed four additional incidents and missed two that Cursor had previously solved. The Sentry tool setup gained three and missed six.^[1]

One of the newly correct answers involved a ClickHouse syntax error. Cursor had sent DESCRIBE otel_logs, but its query tool appended LIMIT 500, turning a valid schema inspection into an invalid statement. Without the Foam tool, Cursor attributed the error to a duplicated output-format clause. With Foam, it identified the appended LIMIT as the fault and pointed to the query tool’s unconditional rewrite. The production log showed the original query, and the trace linked it to the resulting error.^[1]

The Foam agent’s 18 correct answers are a separate result because its harness and prompts also differed. One incident it solved was a MongoDB duplicate-key exception during distributed lock acquisition. The exception looked like a failure in isolation. The logs and traces showed that the application caught it, waited, retried, and completed the request. The correct diagnosis was that the lock was working as designed and required no bug fix. The other three setups did not receive a correct score on that incident.^[1]

What the tool and harness comparisons show

The three Cursor runs shared a harness and model, so their scores describe differences between tool configurations on this incident set. The Foam setup scored two more correct answers overall, while losing two incidents Cursor had solved alone. The Foam agent’s 18 correct answers reflect a separate system: Pi as the harness, its investigation prompts, terminal access, and telemetry. Its six-answer gap over Cursor alone cannot be assigned to any one of those components.

The evaluation contains 22 incidents drawn from Foam’s production experience, with one run per setup and automated judging against human-written keys. Repeated runs, an independent review of the judgments, and incidents from other codebases would give a stronger estimate of how well these results generalize.

Conclusion

On these 22 incidents, Cursor diagnosed 14 root causes with Foam’s tool and 12 with its repository tools alone. The separate Foam agent identified 18, but it also used a different harness and prompts. The incident-level results show why the sequence around an error matters: a syntax error can originate in a query tool, and a database exception can be an expected step in a successful request.

Agents sometimes need the sequence of events around a failure to reach the right diagnosis. This benchmark shows that benefit on specific incidents and provides the outputs and scoring code to examine where each setup succeeded or failed.

References