Sherlock: Automated Anomaly Detector on Druid

Sherlock is an anomaly detection service built on top of Druid, especially for time-series data. It leverages EGADS (Extensible Generic Anomaly Detection System) to learn the data patterns and automatically detect anomalies in large-scale time-series data. Data Quality monitoring of product-specific KPIs is a problem common all over the company. Speedy, accurate detection with minimal false alerts can lead to spotting and fixing problems before they have a large impact on business. Typically, there are two solutions to the problem: manually eyeball graphs every day to look for irregularities or set up upper and lower bound thresholds for automated alerting. While eyeballing is the more accurate of the two, it's not a scalable approach. Automated thresholding allows monitoring many metrics at once; however, threshold-based alerting can lead to too many missed issues or, conversely, too many false alerts. Neither is perfect. Sherlock combines the best of both approaches by building models based on historical time-series, which can be as accurate as manual eyeballing while having the ability to scale out and monitor as many metrics as required.