Name: Reducing Dashboard Latency by 1000% with Apache Druid
Start: 2021-11-23T04:45:00Z
End: 2021-11-23T04:45:43.000Z
Location: BrightTALK

Imply, founded by the original creators of Apache Druid®, develops an innovative database purpose-built for modern analytics applications. Imply is driving a new era in data analytics, where interactive queries, real-time and historical data at unlimited scale, combine with the best price/performance, to realize the full potential of data. 

More than ever, the financial industry needs real-time insights delivered through internal and external analytics applications to stay competitive. Apache Druid and Imply are designed to provide the flexibility, efficiency, and resilience needed to support your analytics applications at scale for streaming and batch data. With Apache Druid and Imply, leading financial institutes and fintechs like Paypal, DBS, Paytm, and many more drive business values with innovative use cases, which we will explore in this talk.

Key takeaways: 

- Key challenges for building analytics architecture for scale
- How Druid and Imply enable Paytm to gain real-time insights for the growth team to drive revenue growth
- How Fintechs drive extra revenue with data applications powered by Druid and Imply
- How Druid and Imply enable you to minimize revenue lost by delivering real-time insights 
- How to reduce fraud with real-time fraud processing, fraud analytics, and fraud notifications

Modern Analytics Applications in the Financial Industry

They both say they’re fast; they both say they scale. Should you flip a coin or are there real technical differences that matter? The developers at Netflix, Twitter, Confluent, Salesforce, and many others chose Druid for good reason.

Watch this webinar and get the facts that lay out the differences. And if you’re a ClickHouse user looking for a change, we’ll show you an easy path to Druid via Imply.

Here’s what we will cover:
- Architecture: the differences in each system’s design
- Flexibility: how you scale out on both systems
- Streaming: how each system handles high-volume event data
- Resilience: what you do on each system to avoid losing data

The New Database Battle: Apache Druid vs. ClickHouse

Why Kafka and Druid are the Perfect Match for Real-Time Analytics Applications?

To succeed in today’s increasingly competitive environment, organizations need real-time, granular insights into their business and operations to grow revenues, fight fraud, and deliver a great customer experience over digital channels. At the same time, share these real-time operational and business insights externally with the customers and partners they serve.

To gain these real-time operational and business insights and share them with customers and partners, requires developers to build apps that bring together streaming data and interactive analytics to push real-time visibility to the next level. This leaves developers trying to figure out what is the right architecture.

Many are asking - how can I ingest millions of events per second? How can I analyze billions of rows interactively in seconds? How can I scale to thousands of end-users concurrently without prohibitive costs? That’s where the K2D (Kafka-to-Druid) stack comes in.

Companies like Netflix, Pinterest, Citrix - as well as Confluent and Imply - have built analytics applications leveraging the K2D stack for real-insights for internal and external users. Join Confluent and Imply, where we will take attendees on a journey to explore:

- What problems does Kafka solve and how does Confluent make it easier?
- Key challenges for building real-time analytics applications and how Imply and Druid fit into the world of analytics applications
- Live demo + use cases:
   ~ how to build an architecture to capture and transform streaming data to analytics applications
   ~ what are some of the use cases from digital native organizations and how these leading companies use Kaffa and Druid to achieve their business goals

Confluent + Imply: Monetize streaming data with real-time analytics applications

It can be challenging to model 1-to-n data relationships in Druid. If you’ve tried, you probably did tons of JOIN operations to build an array. Did you know that with multi-value dimensions you can filter and group complex data quickly and easily?

In this webinar you will:
- Learn the basics of multi-value dimensions and when they are useful
- Get a step-by-step guide to setting up registered lists of data as multi-value string dimensions
- See how a major telecommunications provider is using multi-value dimensions to analyze complex data

'How-to' Half-hour: Using Multivalue Dimensions in Apache Druid

데이터의 차세대 변화는 최신 분석 애플리케이션의 개발입니다. 지금까지 분석은 소수의 분석가가 실행하는 빈도가 낮은 쿼리를 처리했다면, 이제는 수 많은 최종 사용자를 위한 대화형 데이터 경험이 중요합니다. 선두기업들은  비즈니스를 강화를 위해 기업 내부 및 외부의 사용자를 위한 분석 애플리케이션을 구축하고 있습니다. 

이 웨비나에서는 선두기업들이 Imply를 활용하여 데이터 규모 및 동시사용자 수에 제한없이 스트리밍 및 배치데이터에 대한 대화형 데이터 경험을 제공하는 분석 애플리케이션을 구축한 방법을 소개할 예정입니다. 

아젠다:
- Imply 소개: 최신 분석 애플리케이션을 위한 데이터베이스 
- 데모

데이터의 차세대 혁신 : 최신 분석 애플리케이션 구축

PayPal의 사용자 행동 데이터는 제품 성능을 분석하고 더 나은 고객경험을 제공하는데 활용되는 경쟁 차별화요소입니다. 경쟁우위를 유지하려면 사용자 분석 플랫폼이 강력하고 유용해야하며 애널리스트들의 즉각적인 통찰력을 필요로하는 요구를 충족시킬 수 있어야 합니다. 
이 세션에서는 Paypal에서의 다양한 행동분석 유즈케이스 및 Druid가 실시간으로 이 유즈케이스들을 어떻게 강화하고 있는지에 대해서 설명합니다.

Behavior Analytics at PayPal Powered by Druid (한국어 자막 지원)

As one of the biggest data-driven companies in Japan, Rakuten ingests and processes huge amounts of event data for multiple business units and needs to present this data in a timely, responsive manner to accelerate insights. The Rakuten Analytics Platform team tackles this challenge and uses Druid as a central technology in serving this data to its customers. This presentation aims to describe how Druid was selected to solve those challenges, the considerations taken, and how they leverage Druid beyond their original use case.

Rakuten Analytics and their Journey with Druid

기업은 수천개의 자동화된 워크플로와 머신러닝 모델을 적용하여 특정 이벤트 또는 트랜잭션 프로세스를 승인, 거절 또는 지속적으로 모니터링할지 여부와 같이 데이터 스트림의 보안을 자동으로 결정합니다. 워크플로의 사소한 프로세스 변경으로도 의사 결정에 영향을 미치거나 머신러닝 모델에 새로운 변수를 도입할 수 있습니다. 

이러한 변수는 모델 또는 시스템 내의 변경으로 인해 발생할 수 있습니다.  또한 고객측의 변화 (통합 또는 의사결정 행동)으로 인해 발생할 수도 있습니다.  사기공격, 새로운 시장 진입 또는 계절행사로 인한 변경 등의 상황에 따라 적절히 조정해야 하는 경우도 있고, 변경사항을 지켜봐야 하는 경우도 있습니다. 따라서 사용자가 이러한 변경 사항을 신속하게 이해하고 해당 조정 여부를 결정하도록 돕는 것이 중요합니다. 

Imply는 사용자가 이상징후를 실시간으로 모니터링 하고, 실시간으로 자동 경고를 정확하게 생성할 수 있도록 머신러닝 학습 모델의 실시간 모니터링을 가능하게 하는 확장성이 뛰어난 플랫폼을 제공합니다. 

주요 내용: 
- Imply를 사용하여 기계 학습 프로세스 최적화
- Imply Pivot 및 원격 측정 데이터를 사용하여 기계 학습 성능 모니터링 
- Imply를 학습 데이터 소스로 사용하여 모델 정확도를 최적화하고 학습시간 단축
- Imply를 사용하여 모델 예측 프로세스 최적화

라이브 데모: Druid로 머신러닝 파이프라인 최적화 (한국어 자막 지원)

실시간 이벤트/데이터를 수집/처리하는 플랫폼인 Confluent와 연계하여 Imply Druid에 실시간으로 데이터를 수집하고 Imply Pivot을 활용하여 시각화하는 데모를 시연할 예정입니다.

컨플루언트와 임플라이 연동 및 데모

Naver 의 Biz CIC 그룹에서 기존의 광고데이터 분석시스템의 한계점과 Imply를 도입하게된 배경, 사용사례 및 효과를 이야기 합니다.

Naver에서 Imply를 활용한 광고데이터 분석사례

이 세션에서는 Apache Druid의 개발자이자 Imply의 공동창립자인 CTO Gian Merlino가 Project Shapeshift를 소개합니다. Shapeshift는 Druid 사용자 경험의 재구성 및 클라우드 네이티브, 개발자 중심으로 최신 분석 애플리케이션을 구축하는 프로세스입니다.

Apache Druid 및 Imply 제품 전망 (한국어 자막 지원)

企业和组织依赖成千上万机器学习所建立的模型数据来自动决定数据流的安全性，对于特定的事件或者交易流程是否接受、禁止或者持续监控。即便是细微的流程改变可能都会影响到决策模型或者是引入模型中新的变量。

这些变量可能来自模型或系统内部的改变，也可能来自用户端的行为改变。有时我们需要对此做出相应的调整 —— 譬如当有可疑的欺诈行为，进入新的市场 或者是一个节日性的事件造成的变化等等；而有时我们又需要静观其变。因而快速帮助用户理解这些改变从而决定是否要做出相应的调整变得至关重要。

Imply 提供了一个高度可扩展的平台来帮助用户实时监测机器学习模型中可能出现的异常事件并能发出实时警报。

关键知识点
- 使用 Imply 优化机器学习流程
- 使用 Imply Pivot 及遥测数据监控机器学习性能
- 使用 Imply 作为训练样本数据优化模型准确性和缩短训练时间
- 使用 Imply 优化模型预测流程

现场演示: 使用 Imply Druid 优化机器学习数据流程(英语中字)

Apache Druid로 최신 분석 애플리케이션 구축 (한국어 자막 지원)

在本次演示中，我们会展示如何利用 Apache Kafka 和 Imply Druid 构建基于交互式查询和流数据的大规模应用服务

现场演示: 从数据流 到 流数据大规模应用服务(英语中字)

由 Apache Druid 原创者尽心打造, Imply 产品为现代分析型应用而生。Goden Yao, 大中华区业务总监 将在本讲座内介绍数据分析应用的底层架构演变以及当代大型组织是如何构建交互式、大规模、高性价比并兼容实时数据和历史数据的分析应用体验。

Imply 产品介绍—为现代数据分析应用而生

随着数字化转型的深入，企业需要收集、分析更多数据，以确保在变化发生时可以快速做出良好决策。然而，传统本地解决方案已经无法跟上数据的快速发展步伐。在本次演讲中，将为听众分享如何在云中构建数据分析平台。

使用AWS构建无服务器数据分析平台

Imply公司的 CTO, 创始人以及Apache Druid的原作者之一 - Gian Merlino 将会介绍 "项目: Shapeshift" - 对下一代 Druid 用户体验的重构，如何在云原生条件下帮助用户更好的构建现代分析型应用服务

Apache Druid 和 Imply 产品前景展望(英语中字)

使用 Apache Druid 构建现代分析型应用服务 (英语中字)

Organizations are applying hundreds to thousands of automated workflows and machine learning models to automatically determine whether to block, accept or watch a specific transaction or an event. Even a slight change in one of these workflows may impact decision rates or introduce anomalies to your Ml models.

These anomalies in decision rates can be caused by internal changes in models and system components. It can also be caused by changes on the customers’ side: integration or decision behavior. Sometimes a change in decision rates is desirable – such as when there’s a fraud attack, entering into a new market, or a seasonal event, but sometimes it doesn’t. Therefore, it is essential to immediately identify and triage changes in decision rates to ensure users get accurate results.

Imply provides a highly scalable platform that enables real-time monitoring of ML models by allowing users to identify anomalies and generate automated alerts in real-time accurately. 

Key takeaways:
- How you can optimize your machine learning using Imply:
- Monitor machine learning as an application using open telemetry and Imply pivot                                      
- Improve model accuracy and training time using Imply as a source of training data                                  
- Use Imply in a model prediction pipeline

Live demo: Optimizing your machine learning pipeline with Druid

The Indian Express, the largest daily news portal in India, has 500+ million monthly page views across its seven entities. In this talk, Vishal Saxena, Technical Architect of The Indian Express Online, will share various analytics use cases at The Indian Express, how Imply is powering those use cases in real-time, and the challenges faced.

Imply Druid Story at The Indian Express

Sherlock is an anomaly detection service built on top of Druid, especially for time-series data. It leverages EGADS (Extensible Generic Anomaly Detection System) to learn the data patterns and automatically detect anomalies in large-scale time-series data. Data Quality monitoring of product-specific KPIs is a problem common all over the company. Speedy, accurate detection with minimal false alerts can lead to spotting and fixing problems before they have a large impact on business. Typically, there are two solutions to the problem: manually eyeball graphs every day to look for irregularities or set up upper and lower bound thresholds for automated alerting. While eyeballing is the more accurate of the two, it's not a scalable approach. Automated thresholding allows monitoring many metrics at once; however, threshold-based alerting can lead to too many missed issues or, conversely, too many false alerts. Neither is perfect. Sherlock combines the best of both approaches by building models based on historical time-series, which can be as accurate as manual eyeballing while having the ability to scale out and monitor as many metrics as required.

Sherlock: Automated Anomaly Detector on Druid

Confluent Cloud operates a multi-tenant Kafka service, spanning multiple cloud providers and regions across the globe. Offering a cloud service means operational visibility needs to go beyond their internal teams and extend to their customers, who depend on Confluent for a critical piece of their infrastructure. 

This talk will cover what it takes to build large-scale cloud infrastructure and how Apache Druid has helped us push real-time operational and business visibility to the next level. We'll also cover lessons learned from scaling to large data and query volumes, and optimizing our clusters when onboard new customers and use-cases. Operating multi-tenant services requires fine-grained visibility down to the individual tenant, user, or application behavior, where most traditional monitoring stacks fail to scale or become cost-prohibitive. Leveraging Apache Druid means we don't shy away from high-cardinality data, so our teams not only can quickly troubleshoot issues but also glean detailed understanding to help improve the product. That gives us the flexibility to expose the same data to engineering, product teams, and our customers, exposing the insights they need.

Confluent Cloud: Operational Insights at Scale

The way we work has drastically changed. Now more than ever, employees need a reliable way to work from any location - on any device, at any time. Unfortunately, accessing a company’s resources outside of an organization introduces new forms of security risks. 

In this talk, Arun Janarthnam, Senior Principal Architect of Citrix, will share how Citrix mitigates security threats and stops malicious attacks with their real-time analytics platform:

- Respond to security threats in under a minute
- Support self-service analytics
- Build multiple data applications across Citrix

How Citrix uses Druid to Prevent Threats and Stop Malicious Attacks in Real-time

Games24x7 offers multiple casual as well as real money cash games to its users. To maintain a healthy environment for their players onboard, they constantly look out for any elements who would try to win by unethical means in the game. 

In this talk, Kundan Jha, SDE3 of PlayGames24x7, will share how they use Druid to perform fraud analysis to identify fraud users.

Real-Time, High Concurrency Use Case in Fraud Detection

In this talk, Gian Merlino, Imply CTO and co-founder, and one of the original creators of Apache Druid, will introduce Project Shapeshift to the world. Shapeshift is a reimagination of the Druid experience and the process of building modern analytics applications to fit our cloud-native, developer-centric world.

Unveiling What’s Next for Apache Druid and Imply

In this presentation, Sundeep Yedida and Qasim Zeeshan, Software Engineers of Reddit, will talk about how they leverage Apache Druid in conjunction with Apache Datasketches to serve advertisers' needs for forecasting reach and frequency for advertisement campaigns.

Advertiser Audience Forecasting at Reddit with Druid

Here at Ippen Digital we build scalable software solutions for a continuously growing network of over 80 regional news publishers in Germany. This network delivers content for over 100 million monthly active users across various different platforms. Our mission is clear: we want to nurture pluralistic journalism and give our publishers the possibility to voice their opinions openly, while still being profitable for them to continue doing what they are meant to do. On the other hand we want to provide our audience with a great user experience and the content that is really relevant to them. Therefore we aim to truly understand our users and give them a reason to adapt their habits. Together with psychologists we are opening up the field of behavioral science and contextual optimization. The software ecosystem we have built to achieve these goals is sophisticated and state-of-the-art with Apache Druid as one of its core elements. Our data journey with Apache Druid started 3.5 years ago. As our central facts and KPI storage, Druid allows us to generate valuable insights for management and editors alike through dashboards and reports, it provides our data science teams with explorative datasets, and it serves as an integral part for several products – for instance, editorial news assistance in the newsrooms or a personal news assistant – as well as machine learning algorithms. We are here to share our experiences and learnings, and we want to give a high-level demonstration of the potential and benefits of running Apache Druid in a complex data environment while also showing two concrete examples.

Running Druid: From Opinions to Insights, to Action and Impact

In this demo, we will demonstrate how to build an architecture to capture and surface streaming data through interactive queries and unlimited scale with Apache Kafka and Imply Druid.

Live demo: transform streaming data to data streaming applications at scale

As a thought leader and pioneer of distributed energy system technology, SwitchDin ingests and processes a massive amount of streaming data from their distributed energy systems and translates the data into actionable insights for their users. This talk will cover how SwitchDin leverage Imply Druid to solve their data challenge.

Druid/Imply In Distributed Energy Systems

User behavior data at PayPal is a competitive differentiator, leveraged for analyzing product performance, and for providing a better customer experience. To sustain the competitive advantage, it is necessary for the behavior analytics platform to be robust and usable while meeting the ambitious time-to-insights demands of the analysts. The analytics platform at PayPal consists of various phases – namely data collection, data transformation, and insights generation. Druid is the mainstay of the insights generation phase enabling data-driven decision making in real-time. It is being used to support multiple analytics use cases and is deployed in Lambda Architecture mode. Key Takeaways: This talk will cover various behavior analytics use cases at PayPal, how Druid is powering those use cases in real-time, and challenges faced.

Behavior Analytics at PayPal Powered by Druid

Poshmark is a leading social marketplace for new and secondhand styles for women, men, kids, pets, homes, and more. We use Apache Druid for several use cases to monitor business, analyze the trends, perform A/B tests analysis, and share insights across organizations. Druid allowed us to democratize data across the company for both tech and non-technical team members. In this talk, we will talk about our homegrown solution – Yaga, before Druid was adopted and the challenges with Yaga. We will also deep dive into our Druid/Grafana deployment architecture and how we created standardized reporting, which pushed us to develop meta-queries plug-in for Grafana and open-sourcing it. We will also walk through the performance/cost optimization techniques we have developed and the initial struggles of developing a reliable self-serving analytics platform.

Realtime Analytics for Overall Platform Monitoring @ Poshmark

Building Modern Analytics Applications with Apache Druid

Analytics in Motion

In this talk, Gabriel Garcia, Lead Software Engineer of Cisco ThousandEyes, will share their adoption of Apache Druid for customer-facing real-time analytics and how they reduced their dashboards load time by over 1000% with Druid. This talk will also cover how CiscoThousandEyes optimized their production clusters and Kafka ingestions and some difficult problems they solved like deduplication at scale and re-enrichment of recently ingested data.

Reducing Dashboard Latency by 1000% with Apache Druid

Data Analysis

Data Analytics

Data Science

Big Data

Data Best Practices

Analytics

Real Time Analytics

Cisco

Deduplication

Data Quality

Enterprise applications have become a crucial piece of infrastructure for many businesses. The enterprise applications community on BrightTALK features the latest insights in enterprise application integration, enterprise application architecture and EAS software. Join the community to gain access to thousands of videos and webinars presented by recognized enterprise information systems experts and arm yourself with the knowledge you need to succeed.

Enterprise Applications

Practicing business intelligence allows your company to transform raw data into sets of insights for targeted business growth. The business intelligence and analytics community on BrightTALK is made up of thousands of data scientists, database administrators, business analysts and other data professionals. Find relevant webinars and videos on business analytics, business intelligence, data analysis and more presented by recognized thought leaders. Join the conversation by participating in live webinars and round table discussions.

Business Intelligence and Analytics

As an IT professional, many of the problems you face are multifaceted, complex and don’t lend themselves to simple solutions. The information technology community features useful and free information technology resources. Join to browse thousands of videos and webinars on ITIL best practices, IT security strategy and more presented by leading CTOs, CIOs and other technology experts.

Reducing Dashboard Latency by 1000% with Apache Druid

Presented by

About this talk

Imply APAC