Hi [[ session.user.profile.firstName ]]

How to Fail at Kafka

Apache Kafka® is used by thousands of companies across the world but, how difficult is it to operate? Which parameters do you need to set? What can go wrong? This online talk is based on real-world experience of Kafka deployments and explores a collection of common mistakes that are made when running Kafka in production and some best practices to avoid them.

Watch now to learn:

-How to ensure your Kafka data is never lost
-How to write code to cope when things go wrong
-How to ensure data governance between producers and consumers
-How to monitor your cluster

Join Apache Kafka expert, Pete Godfrey, for this engaging talk and delve into best practice ideas and insights.
Recorded Oct 2 2019 19 mins
Your place is confirmed,
we'll send you email reminders
Presented by
Pete Godfrey, Systems Engineer, Confluent
Presentation preview: How to Fail at Kafka

Network with like-minded attendees

  • [[ session.user.profile.displayName ]]
    Add a photo
    • [[ session.user.profile.displayName ]]
    • [[ session.user.profile.jobTitle ]]
    • [[ session.user.profile.companyName ]]
    • [[ userProfileTemplateHelper.getLocation(session.user.profile) ]]
  • [[ card.displayName ]]
    • [[ card.displayName ]]
    • [[ card.jobTitle ]]
    • [[ card.companyName ]]
    • [[ userProfileTemplateHelper.getLocation(card) ]]
  • Channel
  • Channel profile
  • Building an Event Driven Global Data Fabric with Apache Kafka Feb 27 2020 4:00 pm UTC 40 mins
    Will LaForest, CTO Confluent Public Sector
    Government agencies are grappling with a growing challenge of distributing data across a geographically diverse set of locations around the US and globally. In order to ensure mission success, data needs to flow to all of these locations rapidly. Additionally, latency, bandwidth and reliability of communication can prove to be a challenge for agencies. A global data fabric is an emerging approach to help connect mission to data across multiple locations and deliver uniformity and consistency at scale.

    This on-demand webinar will cover:

    An overview of Apache Kafka and and how an event streaming platform can support your agencies mission
    Considerations around handling varying quality communication links
    Synchronous vs asynchronous data replication
    New multi-region capabilities in Confluent Platform for Global Data Fabric
  • Enabling Event Streaming at AO.com Feb 19 2020 11:00 am UTC 50 mins
    Jon Vines, AO.com
    Learn how AO.com are enabling real-time event-driven applications to improve customer experience using Confluent Platform.
    The introduction of Apache Kafka and the Confluent platform is supporting AO.com in modernizing the technical approach to delighting its customers. A key part of this enablement is the introduction of an event-streaming eco-system enabling event-driven applications and architecture.
    Throughout this session, we’ll look at the challenges AO.com faced when looking to adopt Kafka, their use of Confluent Platform including Kafka Connect and KSQL and the adoption of Confluent Cloud. We’ll look at the first steps, where the team are at now and what the future looks like.
  • What's New in Confluent Platform 5.4 Recorded: Feb 6 2020 57 mins
    Mau Barra, Group Manager, Product Marketing, Confluent
    Join the Confluent Product team as we provide a technical overview of Confluent Platform 5.4, which delivers groundbreaking enhancements in the areas of security, disaster recovery and scalability.

    Building an event-driven architecture with Apache Kafka allows you to transition from traditional silos and monolithic applications to modern microservices and event streaming applications.

    However, large enterprises need to complement Kafka with foundational architectural attributes required for production, along with tools that help them run Kafka efficiently at scale.

    This session will cover the key features of this latest release, including:
    -Role-Based Access Control
    -Structured Audit Logs
    -Multi-Region Clusters
    -Schema Validation
    -Tiered Storage
  • SIEM Modernization: Build a Situationally Aware Organization with Apache Kafka® Recorded: Jan 30 2020 35 mins
    Jeffrey Needham, Confluent
    Of all security breaches, 85% are conducted with compromised credentials, often at the administration level or higher. A lot of IT groups think “security” means authentication, authorization and encryption (AAE), but these are often tick-boxes that rarely stop breaches. The internal threat surfaces of data streams or disk drives in a raidset in a data center are not the threat surface of interest.

    Cyber or Threat organizations must conduct internal investigations of IT, subcontractors and supply chains without implicating the innocent. Therefore, they are organizationally air-gapped from IT. Some surveys indicate up to 10% of IT is under investigation at any given time.

    Deploying a signal processing platform, such as Confluent Platform, allows organizations to evaluate data as soon as it becomes available enabling them to assess and mitigate risk before it arises. In Cyber or Threat Intelligence, events can be considered signals, and when analysts are hunting for threat actors, these don't appear as a single needle in a haystack, but as a series of needles. In this paradigm, streams of signals aggregate into signatures. This session shows how various sub-systems in Apache Kafka can be used to aggregate, integrate and attribute these signals into signatures of interest.

    Watch now to learn:
    -The current threat landscape
    -The difference between Security and Threat Intelligence
    -The value of Confluent Platform as an ideal complement to hardware endpoint detection systems and batch-based SIEM warehouses
  • Apache Kafka Architecture & Fundamentals Explained Recorded: Dec 30 2019 57 mins
    Joe Desmond, Technical Trainer, Confluent
    This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput.

    This talk provides a comprehensive overview of Kafka architecture and internal functions, including:
    -Topics, partitions and segments
    -The commit log and streams
    -Brokers and broker replication
    -Producer basics
    -Consumers, consumer groups and offsets

    This session is part 2 of 4 in our Fundamentals for Apache Kafka series.
  • Benefits of Stream Processing and Apache Kafka® Use Cases Recorded: Dec 19 2019 56 mins
    Mark Fei, Technical Trainer, Confluent
    This talk explains how companies are using event-driven architecture to transform their business and how Apache Kafka serves as the foundation for streaming data applications.

    Learn how major players in the market are using Kafka in a wide range of use cases such as microservices, IoT and edge computing, core banking and fraud detection, cyber data collection and dissemination, ESB replacement, data pipelining, ecommerce, mainframe offloading and more.

    Also discussed in this talk are the differences between Apache Kafka and Confluent Platform.
  • Apache Kafka Architecture & Fundamentals Explained Recorded: Dec 16 2019 57 mins
    Joe Desmond, Technical Trainer, Confluent
    This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput.

    This talk provides a comprehensive overview of Kafka architecture and internal functions, including:
    -Topics, partitions and segments
    -The commit log and streams
    -Brokers and broker replication
    -Producer basics
    -Consumers, consumer groups and offsets

    This session is part 2 of 4 in our Fundamentals for Apache Kafka series.
  • Enabling Event Streaming at AO.com Recorded: Dec 10 2019 50 mins
    Jon Vines, AO.com
    Learn how AO.com are enabling real-time event-driven applications to improve customer experience using Confluent Platform.
    The introduction of Apache Kafka and the Confluent platform is supporting AO.com in modernizing the technical approach to delighting its customers. A key part of this enablement is the introduction of an event-streaming eco-system enabling event-driven applications and architecture.
    Throughout this session, we’ll look at the challenges AO.com faced when looking to adopt Kafka, their use of Confluent Platform including Kafka Connect and KSQL and the adoption of Confluent Cloud. We’ll look at the first steps, where the team are at now and what the future looks like.
  • On Track with Apache Kafka®: Building a Streaming ETL Solution with Rail Data Recorded: Dec 9 2019 59 mins
    Robin Moffatt, Developer Advocate, Confluent
    As data engineers, we frequently need to build scalable systems working with data from a variety of sources and with various ingest rates, sizes, and formats. This talk takes an in-depth look at how Apache Kafka can be used to provide a common platform on which to build data infrastructure driving both real-time analytics as well as event-driven applications.

    Using a public feed of railway data it will show how to ingest data from message queues such as ActiveMQ with Kafka Connect, as well as from static sources such as S3 and REST endpoints. We'll then see how to use stream processing to transform the data into a form useful for streaming to analytics in tools such as Elasticsearch and Neo4j. The same data will be used to drive a real-time notifications service through Telegram.

    If you're wondering how to build your next scalable data platform, how to reconcile the impedance mismatch between stream and batch, and how to wrangle streams of data—this talk is for you!
  • Being an Apache Kafka® Expert in a Multi-Cloud World Recorded: Dec 3 2019 31 mins
    Ricardo Ferreira, Developer Advocate, Confluent
    Apache Kafka is an amazing piece of technology, that has been furiously adopted by companies all around the world to implement event-driven architectures. While its adoption continues to increase, the reality is that most developers often complain about the complexity of managing the clusters by themselves, which seriously decreases their ability to be agile.

    This 30-minute demo will introduce Confluent Cloud, a service that offers Apache Kafka and the Confluent Platform so developers can focus on what they do best: the coding part. We will show you how to quickly reuse code written for standard Kafka APIs to connect to Confluent Cloud and how an event-streaming application is built and deployed.
  • How Apache Kafka® Works Recorded: Nov 27 2019 62 mins
    Michael Bingham, Technical Trainer, Confluent
    Pick up best practices for developing applications that use Apache Kafka, beginning with a high level code overview for a basic producer and consumer. From there we’ll cover strategies for building powerful stream processing applications, including high availability through replication, data retention policies, producer design and producer guarantees.

    We’ll delve into the details of delivery guarantees, including exactly-once semantics, partition strategies and consumer group rebalances. The talk will finish with a discussion of compacted topics, troubleshooting strategies and a security overview.

    This session is part 3 of 4 in our Fundamentals for Apache Kafka series.
  • Best Practices for Streaming IoT Data with MQTT and Apache Kafka® Recorded: Nov 14 2019 60 mins
    Kai Waehner, Technology Evangelist, Confluent + Dominik Obermaier, CTO, HiveMQ
    Organizations today are looking to stream IoT data to Apache Kafka. However, connecting tens of thousands or even millions of devices over unreliable networks can create some architecture challenges.

    In this session, we will identify and demo some best practices for implementing a large scale IoT system that can stream MQTT messages to Apache Kafka.
  • Integrating Apache Kafka Into Your Environment Recorded: Nov 4 2019 52 mins
    Russ Sayers, Sr. Technical Trainer, Confluent
    Integrating Apache Kafka with other systems in a reliable and scalable way is a key part of an event streaming platform. This session will show you how to get streams of data into and out of Kafka with Kafka Connect and REST Proxy, maintain data formats and ensure compatibility with Schema Registry and Avro, and build real-time stream processing applications with Confluent KSQL and Kafka Streams.

    This session is part 4 of 4 in our Fundamentals for Apache Kafka series.
  • How Apache Kafka® Works Recorded: Oct 28 2019 63 mins
    Michael Bingham, Technical Trainer, Confluent
    Pick up best practices for developing applications that use Apache Kafka, beginning with a high level code overview for a basic producer and consumer. From there we’ll cover strategies for building powerful stream processing applications, including high availability through replication, data retention policies, producer design and producer guarantees.

    We’ll delve into the details of delivery guarantees, including exactly-once semantics, partition strategies and consumer group rebalances. The talk will finish with a discussion of compacted topics, troubleshooting strategies and a security overview.

    This session is part 3 of 4 in our Fundamentals for Apache Kafka series.
  • Design and Implementation of Incremental Cooperative Rebalancing Recorded: Oct 24 2019 58 mins
    Speaker: Konstantine Karantasis, Software Engineer, Confluent
    Since its initial release, the Kafka group membership protocol has offered Connect, Streams and Consumer applications an ingenious and robust way to balance resources among distributed processes. The process of rebalancing, as it’s widely known, allows Kafka APIs to define an embedded protocol for load balancing within the group membership protocol itself.

    Until now, rebalancing has been working under the simple assumption that every time a new group generation is created, the members join after first releasing all of their resources, getting a whole new load assignment by the time the new group is formed. This allows Kafka APIs to provide task fault-tolerance and elasticity on top of the group membership protocol.

    However, due to its side-effects on multi-tenancy and scalability this simple approach in rebalancing, also known as stop-the-world effect, is limiting larger scale deployments. Because of stop-the-world, application tasks get interrupted only for most of them to receive the same resources after rebalancing. In this technical deep dive, we’ll discuss the proposition of Incremental Cooperative Rebalancing as a way to alleviate stop-the-world and optimize rebalancing in Kafka APIs.

    This talk will cover:
    -The internals of Incremental Cooperative Rebalancing
    -Uses cases that benefit from Incremental Cooperative Rebalancing
    -Implementation in Kafka Connect
    -Performance results in Kafka Connect clusters
  • Apache Kafka Architecture & Fundamentals Explained Recorded: Oct 21 2019 57 mins
    Joe Desmond, Technical Trainer, Confluent
    This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput.

    This talk provides a comprehensive overview of Kafka architecture and internal functions, including:
    -Topics, partitions and segments
    -The commit log and streams
    -Brokers and broker replication
    -Producer basics
    -Consumers, consumer groups and offsets

    This session is part 2 of 4 in our Fundamentals for Apache Kafka series.
  • Scaling Security on 100s of Millions of Mobile Devices Using Kafka & Scylla Recorded: Oct 16 2019 48 mins
    Richard Ney, Sr. Staff Engineer, Lookout + Eyal Gutkind, VP Solutions, ScyllaDB + Jeff Bean, Solutions Architect, Confluent
    Join mobile cybersecurity leader Lookout as they talk through their data ingestion journey.

    Lookout enables enterprises to protect their data by evaluating threats and risks at post-perimeter endpoint devices and providing access to corporate data after conditional security scans. Their continuous assessment of device health creates a massive amount of telemetry data, forcing new approaches to data ingestion. Learn how Lookout changed its approach in order to grow from 1.5 million devices to 100 million devices and beyond, by implementing Confluent Platform and switching to Scylla.
  • Benefits of Stream Processing and Apache Kafka® Use Cases Recorded: Oct 14 2019 57 mins
    Mark Fei, Technical Trainer, Confluent
    This talk explains how companies are using event-driven architecture to transform their business and how Apache Kafka serves as the foundation for streaming data applications.

    Learn how major players in the market are using Kafka in a wide range of use cases such as microservices, IoT and edge computing, core banking and fraud detection, cyber data collection and dissemination, ESB replacement, data pipelining, ecommerce, mainframe offloading and more.

    Also discussed in this talk are the differences between Apache Kafka and Confluent Platform.
  • How to Unlock your Mainframe Data with Confluent, Attunity and Apache Kafka Recorded: Oct 3 2019 51 mins
    Simon Leigh, Confluent. Martin Hamilton, Attunity
    Large enterprises, government agencies, and many other organisations rely on mainframe computers to deliver the core systems managing some of their most valuable and sensitive data. However, the processes and cultures around a mainframe often prevent the adoption of the agile, born-on-the web practices that have become essential to developing cutting edge internal and customer-facing applications. Mainframes also represent significant, long-term investments in terms of time, money, people and possibly even decades worth of stored data. This webinar will help you understand that you can offload and unlock your mainframe data and equip your business for the modern data-driven environment.

    By attending this webinar, you will learn:

    1. How to access the depth and richness of insights held in the data within your mainframe
    2. How to bring real-time data from mainframes efficiently with CDC technology partners for Confluent Enterprise and Apache Kafka
    3. How to reduce the costs and complexity of querying a mainframe database using the unique change data capture function
    4. How to leverage Apache Kafka’s modern distributed architecture to move mainframe data in real-time
    5. How Attunity Replicate software is leveraged to stream data changes to Kafka
  • How to Fail at Kafka Recorded: Oct 2 2019 19 mins
    Pete Godfrey, Systems Engineer, Confluent
    Apache Kafka® is used by thousands of companies across the world but, how difficult is it to operate? Which parameters do you need to set? What can go wrong? This online talk is based on real-world experience of Kafka deployments and explores a collection of common mistakes that are made when running Kafka in production and some best practices to avoid them.

    Watch now to learn:

    -How to ensure your Kafka data is never lost
    -How to write code to cope when things go wrong
    -How to ensure data governance between producers and consumers
    -How to monitor your cluster

    Join Apache Kafka expert, Pete Godfrey, for this engaging talk and delve into best practice ideas and insights.
We provide a central nervous system for streaming real-time data.
Confluent, founded by the creators of open source Apache Kafka®, provides the leading streaming platform that enables enterprises to maximize the value of data. Confluent Platform empowers leaders in industries such as retail, logistics, manufacturing, financial services, technology and media, to move data from isolated systems into a real-time data pipeline where they can act on it immediately.

Backed by Benchmark, Index Ventures and Sequoia, Confluent is based in Palo Alto, California. To learn more, please visit www.confluent.io.

Embed in website or blog

Successfully added emails: 0
Remove all
  • Title: How to Fail at Kafka
  • Live at: Oct 2 2019 1:00 pm
  • Presented by: Pete Godfrey, Systems Engineer, Confluent
  • From:
Your email has been sent.
or close