Confluent Cloud: Operational Insights at Scale

Confluent Cloud operates a multi-tenant Kafka service, spanning multiple cloud providers and regions across the globe. Offering a cloud service means operational visibility needs to go beyond their internal teams and extend to their customers, who depend on Confluent for a critical piece of their infrastructure. This talk will cover what it takes to build large-scale cloud infrastructure and how Apache Druid has helped us push real-time operational and business visibility to the next level. We'll also cover lessons learned from scaling to large data and query volumes, and optimizing our clusters when onboard new customers and use-cases. Operating multi-tenant services requires fine-grained visibility down to the individual tenant, user, or application behavior, where most traditional monitoring stacks fail to scale or become cost-prohibitive. Leveraging Apache Druid means we don't shy away from high-cardinality data, so our teams not only can quickly troubleshoot issues but also glean detailed understanding to help improve the product. That gives us the flexibility to expose the same data to engineering, product teams, and our customers, exposing the insights they need.