Sherlock: Automated Anomaly Detector on Druid

Logo
Presented by

Guruganesh Kotta | Senior Software Engineer | Yahoo; Jigar Patel | Senior Software Development Engineer | Yahoo

About this talk

Sherlock is an anomaly detection service built on top of Druid, especially for time-series data. It leverages EGADS (Extensible Generic Anomaly Detection System) to learn the data patterns and automatically detect anomalies in large-scale time-series data. Data Quality monitoring of product-specific KPIs is a problem common all over the company. Speedy, accurate detection with minimal false alerts can lead to spotting and fixing problems before they have a large impact on business. Typically, there are two solutions to the problem: manually eyeball graphs every day to look for irregularities or set up upper and lower bound thresholds for automated alerting. While eyeballing is the more accurate of the two, it's not a scalable approach. Automated thresholding allows monitoring many metrics at once; however, threshold-based alerting can lead to too many missed issues or, conversely, too many false alerts. Neither is perfect. Sherlock combines the best of both approaches by building models based on historical time-series, which can be as accurate as manual eyeballing while having the ability to scale out and monitor as many metrics as required.

Related topics:

More from this channel

Upcoming talks (0)
On-demand talks (34)
Subscribers (623)
Imply, founded by the original creators of Apache Druid®, develops an innovative database purpose-built for modern analytics applications. Imply is driving a new era in data analytics, where interactive queries, real-time and historical data at unlimited scale, combine with the best price/performance, to realize the full potential of data.