Reducing issue volume with message clustering

Error monitors group events into issues for engineers to review. We compared Sentry’s issues with Foam’s message-based clusters in migration data from 11 production environments: 126,651 Sentry issues and 5,671 Foam clusters.

The count measures issue volume. It does not establish whether the clusters preserve every distinct root cause.

The grouping problem

One underlying problem can generate thousands of events. It can also appear under different stack traces or with different values in the error message. A useful grouping system puts events that call for the same fix together and keeps unrelated problems apart.

Grouping more aggressively can reduce duplicates while increasing the chance of hiding a separate bug.

How Sentry groups events

Sentry is an error monitoring platform. Its grouping system first computes a fingerprint from an error’s structure, generally prioritizing the stack trace. It normalizes frames using rules that vary by platform. When a stack trace is unavailable or insufficient, it can use exception types, values, and normalized messages. For an error that does not match an existing fingerprint, Sentry also uses a model to compare its stack trace with existing issues and merge likely duplicates. ^[1]

This approach preserves information about where an error occurred. It can also split one cause when the execution path changes, or group different causes whose traces look similar. Sentry’s model is intended to reduce these failures, and Sentry has continued to update it. The counts below come from Foam’s migration data; they do not describe every Sentry version or customer environment. ^[2]

How Foam groups events

Foam starts with the exception message. Drain3, a streaming template miner, identifies text that remains stable as values change. The diagram below uses illustrative messages, not a labeled sample from the migration. ^[3]

Drain3 routes a message through a fixed-depth tree, first by token count and then by leading tokens. At a leaf, it compares the message with candidate templates using positional token similarity. If the best score falls below a threshold, it creates a new cluster. Otherwise it updates the matched template, replacing changing tokens with wildcards. ^[3]

A template can continue to describe an error after a code change moves it to another function. Similar messages with different token counts or different paths through the tree, however, may become separate clusters.

Foam adds a second pass for clusters with the same first three tokens. It compares their token sets using the overlap coefficient and merges clusters that reach a configured threshold:

Here, A and B are the token sets of two candidate templates. This pass can join templates that Drain3 created independently. It can also merge messages that share wording but represent different causes, a risk that needs to be measured separately.

Side-by-side comparison

The comparison measures issue counts in this migration dataset. It does not measure which method more accurately separates root causes.

Reduction across production environments

These counts came from Foam’s migration away from Sentry grouping. The reduction varies substantially by environment.

The median environment saw a 10.9× reduction. Across all environments, Foam produced 95.5% fewer issue records. The difference between the median and the total reflects how much the largest environments contribute to the overall count.

Conclusion

Across 11 production environments, Foam reduced 126,651 Sentry issues to 5,671 clusters. The median environment had 10.9× fewer issues, and the full dataset had 95.5% fewer. That left a much shorter issue list in this dataset. Message templates can keep related errors together when values or stack traces change. Foam’s second pass merges additional templates with overlapping tokens.

These counts do not measure review time or the rate at which distinct causes were merged. A labeled sample of error events would make those tradeoffs measurable.

References