Managing Combinatorial Explosion In High Dimensional Time Series Anomaly Detection

This article discusses the challenges of combinatorial explosion in high dimensional time series anomaly detection. It highlights techniques to manage complexity and align monitoring with leadership priorities to improve data observability and operational efficiency.

Managing Combinatorial Explosion In High Dimensional Time Series Anomaly Detection

Enterprises now depend on telemetry streams to decide how they price products, route fleets, tune recommendations, and prove compliance. Each “metric” arrives with attributes such as device type, cohort, geography, feature flag, and product variant. That complexity shows up in spending too, with the data observability market projected to grow from about $2.37 billion in 2024 to roughly $4.73 billion by 2030. In that environment, anomaly detection on time series is no longer only about spotting a spike; it is about managing the combinatorial explosion created when every dimension and attribute can become part of the story. Suvodeep Pyne, a Staff Software Engineer II at StarTree Inc. and an ACM member, has spent his career building systems that keep those high dimensional datasets usable and fast.

Seeing Metrics As Families, Not Lines

“Most teams still start with a single graph,” Pyne says. “But the moment they ask 'for which users, where, and under what conditions,’ they are already in high dimensional territory.” In modern telemetry, a simple time series like “checkout conversion” often stands for thousands of micro-series distinguished by country, device, experiment cell, and customer segment. Practitioners expect anomaly detection to handle that reality without flooding them with noise or hiding important patterns. That expectation creates an immediate combinatorial problem. Every new dimension multiplies the number of potential metric slices, and each slice can behave differently over time. In his anomaly detection work, Pyne designs pipelines that treat dimensions as first class selectors rather than afterthought filters. Instead of scraping every possible combination, detectors operate on a curated family of metric–dimension pairs that reflect how the business actually reasons about value and risk.

Turning Combinatorial Space Into A Tractable Search Problem

As data grows, the sheer number of potential views can overwhelm both humans and systems. Industry forecasts project the anomaly detection market to reach about $21.21 billion by 2032, which shows how many organizations now treat the problem as core infrastructure. In high dimensional settings, the real risk is not a single outlier; it is the subtle pattern that only appears when specific attributes move together. For Pyne, managing combinatorial explosion starts by admitting that the search space must be shaped.

In the platforms he works on, detection engines let teams declare which dimensions may participate in which models and at what granularity. Lightweight heuristics scan broad populations of combinations to identify promising candidates, while more expensive models are reserved for high value regions of the search space. New experiments introduce flags and cohorts, new channels add attributes like fulfillment partner or network hop, and regulatory policies force data to be segmented by residency or consent. “Combinatorial explosion is not just a math phrase,” Pyne says. “It is what your on-call engineer feels when they open an alert and see twenty different slices moving at once.”

Aligning High Dimensional Monitoring With Leadership Priorities

At that point, combinatorial explosion stops being only a modeling detail and becomes a leadership issue. Leaders care about which combinations of attributes define “healthy” behavior, how much they are spending to monitor those combinations, and whether the organization can explain why certain slices are watched closely while others are sampled or ignored. Pyne’s earlier work on Cubert at LinkedIn illustrates how much these choices matter. By inventing a new join algorithm and building operators for large scale data handling, he and his collaborators reduced People You May Know runtime by more than 70 percent, cut around 1,000 node-hours of compute per day, and improved engagement on that feature by 22 percent. Those gains translated into estimated compute savings near $1.8 million per year and freed capacity for additional machine learning workflows that also depended on high dimensional joins and aggregations. As a CIKM 2025 program committee member, he sees more teams treating combinatorial structure as a first class design concern.

Techniques That Will Tame Combinatorial Explosion

Looking forward, time series intelligence is set to play an even larger role in how organizations coordinate and act. One forecast estimates the time series intelligence software market at about $6 billion in 2023 and projects it to reach roughly $15 billion by 2031, driven by real time analytics in finance, logistics, and operations. That expansion will not only increase the volume of time series; it will also increase their dimensionality as more context is logged by default. To cope, Pyne expects systems to rely more heavily on techniques that compress, summarise, and prioritise combinations instead of enumerating them. Dimensionality reduction methods can map complex attribute sets into lower dimensional representations where distance still reflects similarity of behavior. Embedding-based detectors can look for unusual trajectories in this compressed space rather than running independent models for every slice, while policies encode how often certain combinations should be sampled based on their historical stability and potential impact. “In the future, systems will have to negotiate constantly between the size of the space, the speed of decisions, and the cost of computation,” he says.

Why Combinatorial Thinking Needs To Reach Beyond The Data Team

The consequences of getting this wrong are not limited to infrastructure teams. In education, healthcare, logistics, and public services, people’s experiences are shaped by combinations of factors: who they are, where they are, which channel they use, and which policy applies. At Quasix, Pyne helped build Realia, a Learning Management System that grew to more than 100k users across Android, iOS, and web. By digitizing attendance, quizzes, announcements, and fee payments across students, parents, teachers, and administrators, Realia reduced operational workload for institutions by roughly 90 percent and lifted engagement and completion metrics across multiple cohorts. From his perspective, that lesson now applies everywhere.

As more telemetry becomes high dimensional by default, organizations need clear answers to three questions: which combinations define success, which combinations define unacceptable risk, and how their systems will behave when both the number of signals and the number of dimensions double. As a Globee Awards judge, Pyne looks for that clarity when reviewing technology narratives. “The teams that win are the ones that treat combinatorial explosion as a design constraint, not an excuse,” he says. “If you can explain your space, how you search it, and how you will scale that search over the next decade, you are not just doing anomaly detection. You are building a discipline.”