Confluent Cloud: Operational Insights at Scale

Logo
Presented by

Xavier Léauté | Principal Engineer | Confluent; Zohreh Karimi | Software Engineer | Confluent

About this talk

Confluent Cloud operates a multi-tenant Kafka service, spanning multiple cloud providers and regions across the globe. Offering a cloud service means operational visibility needs to go beyond their internal teams and extend to their customers, who depend on Confluent for a critical piece of their infrastructure. This talk will cover what it takes to build large-scale cloud infrastructure and how Apache Druid has helped us push real-time operational and business visibility to the next level. We'll also cover lessons learned from scaling to large data and query volumes, and optimizing our clusters when onboard new customers and use-cases. Operating multi-tenant services requires fine-grained visibility down to the individual tenant, user, or application behavior, where most traditional monitoring stacks fail to scale or become cost-prohibitive. Leveraging Apache Druid means we don't shy away from high-cardinality data, so our teams not only can quickly troubleshoot issues but also glean detailed understanding to help improve the product. That gives us the flexibility to expose the same data to engineering, product teams, and our customers, exposing the insights they need.

Related topics:

More from this channel

Upcoming talks (0)
On-demand talks (33)
Subscribers (604)
Imply, founded by the original creators of Apache Druid®, develops an innovative database purpose-built for modern analytics applications. Imply is driving a new era in data analytics, where interactive queries, real-time and historical data at unlimited scale, combine with the best price/performance, to realize the full potential of data.