Hi [[ session.user.profile.firstName ]]

No More Silos: Integrating Databases into Apache Kafka®

Companies new and old are all recognizing the importance of a low-latency, scalable, fault-tolerant data backbone, in the form of the Apache Kafka® streaming platform. With Apache Kafka, developers can integrate multiple sources and systems, which enables low latency analytics, event-driven architectures and the population of multiple downstream systems.

In this talk, we’ll look at one of the most common integration requirements – connecting databases to Apache Kafka. We’ll consider the concept that all data is a stream of events, including that residing within a database. We’ll look at why we’d want to stream data from a database, including driving applications in Apache Kafka from events upstream. We’ll discuss the different methods for connecting databases to Apache Kafka, and the pros and cons of each. Techniques including Change-Data-Capture (CDC) and Apache Kafka Connect will be covered, as well as an exploration of the power of KSQL, streaming SQL for Apache Kafka, for performing transformations such as joins on the inbound data.

Register now to learn:
•Why databases are just a materialized view of a stream of events
•The best ways to integrate databases with Apache Kafka
•Anti-patterns to be aware of
•The power of KSQL for transforming streams of data in Apache Kafka
Recorded Apr 11 2019 57 mins
Your place is confirmed,
we'll send you email reminders
Presented by
Robin Moffatt, Developer Advocate, Confluent
Presentation preview: No More Silos: Integrating Databases into Apache Kafka®

Network with like-minded attendees

  • [[ session.user.profile.displayName ]]
    Add a photo
    • [[ session.user.profile.displayName ]]
    • [[ session.user.profile.jobTitle ]]
    • [[ session.user.profile.companyName ]]
    • [[ userProfileTemplateHelper.getLocation(session.user.profile) ]]
  • [[ card.displayName ]]
    • [[ card.displayName ]]
    • [[ card.jobTitle ]]
    • [[ card.companyName ]]
    • [[ userProfileTemplateHelper.getLocation(card) ]]
  • Channel
  • Channel profile
  • Enabling Event Streaming at AO.com Dec 10 2019 11:00 am UTC 50 mins
    Jon Vines, AO.com
    Learn how AO.com are enabling real-time event-driven applications to improve customer experience using Confluent Platform.
    The introduction of Apache Kafka and the Confluent platform is supporting AO.com in modernizing the technical approach to delighting its customers. A key part of this enablement is the introduction of an event-streaming eco-system enabling event-driven applications and architecture.
    Throughout this session, we’ll look at the challenges AO.com faced when looking to adopt Kafka, their use of Confluent Platform including Kafka Connect and KSQL and the adoption of Confluent Cloud. We’ll look at the first steps, where the team are at now and what the future looks like.
  • How Apache Kafka® Works Nov 27 2019 2:00 pm UTC 62 mins
    Michael Bingham, Technical Trainer, Confluent
    Pick up best practices for developing applications that use Apache Kafka, beginning with a high level code overview for a basic producer and consumer. From there we’ll cover strategies for building powerful stream processing applications, including high availability through replication, data retention policies, producer design and producer guarantees.

    We’ll delve into the details of delivery guarantees, including exactly-once semantics, partition strategies and consumer group rebalances. The talk will finish with a discussion of compacted topics, troubleshooting strategies and a security overview.

    This session is part 3 of 4 in our Fundamentals for Apache Kafka series.
  • Best Practices for Streaming IoT Data with MQTT and Apache Kafka® Recorded: Nov 14 2019 60 mins
    Kai Waehner, Technology Evangelist, Confluent + Dominik Obermaier, CTO, HiveMQ
    Organizations today are looking to stream IoT data to Apache Kafka. However, connecting tens of thousands or even millions of devices over unreliable networks can create some architecture challenges.

    In this session, we will identify and demo some best practices for implementing a large scale IoT system that can stream MQTT messages to Apache Kafka.
  • Integrating Apache Kafka Into Your Environment Recorded: Nov 4 2019 52 mins
    Russ Sayers, Sr. Technical Trainer, Confluent
    Integrating Apache Kafka with other systems in a reliable and scalable way is a key part of an event streaming platform. This session will show you how to get streams of data into and out of Kafka with Kafka Connect and REST Proxy, maintain data formats and ensure compatibility with Schema Registry and Avro, and build real-time stream processing applications with Confluent KSQL and Kafka Streams.

    This session is part 4 of 4 in our Fundamentals for Apache Kafka series.
  • How Apache Kafka® Works Recorded: Oct 28 2019 63 mins
    Michael Bingham, Technical Trainer, Confluent
    Pick up best practices for developing applications that use Apache Kafka, beginning with a high level code overview for a basic producer and consumer. From there we’ll cover strategies for building powerful stream processing applications, including high availability through replication, data retention policies, producer design and producer guarantees.

    We’ll delve into the details of delivery guarantees, including exactly-once semantics, partition strategies and consumer group rebalances. The talk will finish with a discussion of compacted topics, troubleshooting strategies and a security overview.

    This session is part 3 of 4 in our Fundamentals for Apache Kafka series.
  • Design and Implementation of Incremental Cooperative Rebalancing Recorded: Oct 24 2019 58 mins
    Speaker: Konstantine Karantasis, Software Engineer, Confluent
    Since its initial release, the Kafka group membership protocol has offered Connect, Streams and Consumer applications an ingenious and robust way to balance resources among distributed processes. The process of rebalancing, as it’s widely known, allows Kafka APIs to define an embedded protocol for load balancing within the group membership protocol itself.

    Until now, rebalancing has been working under the simple assumption that every time a new group generation is created, the members join after first releasing all of their resources, getting a whole new load assignment by the time the new group is formed. This allows Kafka APIs to provide task fault-tolerance and elasticity on top of the group membership protocol.

    However, due to its side-effects on multi-tenancy and scalability this simple approach in rebalancing, also known as stop-the-world effect, is limiting larger scale deployments. Because of stop-the-world, application tasks get interrupted only for most of them to receive the same resources after rebalancing. In this technical deep dive, we’ll discuss the proposition of Incremental Cooperative Rebalancing as a way to alleviate stop-the-world and optimize rebalancing in Kafka APIs.

    This talk will cover:
    -The internals of Incremental Cooperative Rebalancing
    -Uses cases that benefit from Incremental Cooperative Rebalancing
    -Implementation in Kafka Connect
    -Performance results in Kafka Connect clusters
  • Apache Kafka Architecture & Fundamentals Explained Recorded: Oct 21 2019 57 mins
    Joe Desmond, Technical Trainer, Confluent
    This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput.

    This talk provides a comprehensive overview of Kafka architecture and internal functions, including:
    -Topics, partitions and segments
    -The commit log and streams
    -Brokers and broker replication
    -Producer basics
    -Consumers, consumer groups and offsets

    This session is part 2 of 4 in our Fundamentals for Apache Kafka series.
  • Scaling Security on 100s of Millions of Mobile Devices Using Kafka & Scylla Recorded: Oct 16 2019 48 mins
    Richard Ney, Sr. Staff Engineer, Lookout + Eyal Gutkind, VP Solutions, ScyllaDB + Jeff Bean, Solutions Architect, Confluent
    Join mobile cybersecurity leader Lookout as they talk through their data ingestion journey.

    Lookout enables enterprises to protect their data by evaluating threats and risks at post-perimeter endpoint devices and providing access to corporate data after conditional security scans. Their continuous assessment of device health creates a massive amount of telemetry data, forcing new approaches to data ingestion. Learn how Lookout changed its approach in order to grow from 1.5 million devices to 100 million devices and beyond, by implementing Confluent Platform and switching to Scylla.
  • Benefits of Stream Processing and Apache Kafka® Use Cases Recorded: Oct 14 2019 57 mins
    Mark Fei, Technical Trainer, Confluent
    This talk explains how companies are using event-driven architecture to transform their business and how Apache Kafka serves as the foundation for streaming data applications.

    Learn how major players in the market are using Kafka in a wide range of use cases such as microservices, IoT and edge computing, core banking and fraud detection, cyber data collection and dissemination, ESB replacement, data pipelining, ecommerce, mainframe offloading and more.

    Also discussed in this talk are the differences between Apache Kafka and Confluent Platform.
  • How to Unlock your Mainframe Data with Confluent, Attunity and Apache Kafka Recorded: Oct 3 2019 51 mins
    Simon Leigh, Confluent. Martin Hamilton, Attunity
    Large enterprises, government agencies, and many other organisations rely on mainframe computers to deliver the core systems managing some of their most valuable and sensitive data. However, the processes and cultures around a mainframe often prevent the adoption of the agile, born-on-the web practices that have become essential to developing cutting edge internal and customer-facing applications. Mainframes also represent significant, long-term investments in terms of time, money, people and possibly even decades worth of stored data. This webinar will help you understand that you can offload and unlock your mainframe data and equip your business for the modern data-driven environment.

    By attending this webinar, you will learn:

    1. How to access the depth and richness of insights held in the data within your mainframe
    2. How to bring real-time data from mainframes efficiently with CDC technology partners for Confluent Enterprise and Apache Kafka
    3. How to reduce the costs and complexity of querying a mainframe database using the unique change data capture function
    4. How to leverage Apache Kafka’s modern distributed architecture to move mainframe data in real-time
    5. How Attunity Replicate software is leveraged to stream data changes to Kafka
  • How to Fail at Kafka Recorded: Oct 2 2019 19 mins
    Pete Godfrey, Systems Engineer, Confluent
    Apache Kafka® is used by thousands of companies across the world but, how difficult is it to operate? Which parameters do you need to set? What can go wrong? This online talk is based on real-world experience of Kafka deployments and explores a collection of common mistakes that are made when running Kafka in production and some best practices to avoid them.

    Watch now to learn:

    -How to ensure your Kafka data is never lost
    -How to write code to cope when things go wrong
    -How to ensure data governance between producers and consumers
    -How to monitor your cluster

    Join Apache Kafka expert, Pete Godfrey, for this engaging talk and delve into best practice ideas and insights.
  • SIEM Modernization: Build a Situationally Aware Organization with Apache Kafka® Recorded: Sep 25 2019 35 mins
    Jeffrey Needham, Confluent
    Of all security breaches, 85% are conducted with compromised credentials, often at the administration level or higher. A lot of IT groups think “security” means authentication, authorization and encryption (AAE), but these are often tick-boxes that rarely stop breaches. The internal threat surfaces of data streams or disk drives in a raidset in a data center are not the threat surface of interest.

    Cyber or Threat organizations must conduct internal investigations of IT, subcontractors and supply chains without implicating the innocent. Therefore, they are organizationally air-gapped from IT. Some surveys indicate up to 10% of IT is under investigation at any given time.

    Deploying a signal processing platform, such as Confluent Platform, allows organizations to evaluate data as soon as it becomes available enabling them to assess and mitigate risk before it arises. In Cyber or Threat Intelligence, events can be considered signals, and when analysts are hunting for threat actors, these don't appear as a single needle in a haystack, but as a series of needles. In this paradigm, streams of signals aggregate into signatures. This session shows how various sub-systems in Apache Kafka can be used to aggregate, integrate and attribute these signals into signatures of interest.

    Watch now to learn:
    -The current threat landscape
    -The difference between Security and Threat Intelligence
    -The value of Confluent Platform as an ideal complement to hardware endpoint detection systems and batch-based SIEM warehouses
  • Apache Kafka® + Machine Learning for Supply Chain Recorded: Sep 24 2019 58 mins
    Kai Waehner, Confluent + Graham Ganssle, Expero
    Automating multifaceted, complex workflows requires hybrid solutions like streaming analytics of IoT data, batch analytics like machine learning solutions, and real-time visualizations. Leaders in organizations who are responsible for global supply chain planning are responsible for working with and integrating with data from disparate sources around the world. Many of these data sources output information in real-time, which assists planners in operationalizing plans and interacting with manufacturing output. IoT sensors on manufacturing equipment and inventory control systems feed real-time processing pipelines to match actual production figures against planned schedules to calculate yield efficiency. 

    Using information from both real-time systems and batch optimization, supply chain managers are able to economize operations and automate tedious inventory and manufacturing accounting processes. Sitting on top of all these systems is a supply chain visualization tool, enabling users' visibility over the global supply chain. If you are responsible for key data integration initiatives, join for a detailed walk through of a customer's use of this system built using Confluent and Expero tools. 

    WHAT YOU'LL LEARN: 
    •See different use cases in automation industry and Industrial IoT (IIoT) where an event streaming platform adds business value. 
    •Understand different architecture options to leverage Apache Kafka and Confluent.
    •How to leverage different analytics tools and machine learning frameworks in a flexible and scalable way.
    •How real-time visualization ties together streaming and batch analytics for business users, interpreters, and analysts.
    •Understand how streaming and batch analytics optimize the supply chain planning workflow.
    •Conceptualize the intersection between resource utilization and manufacturing assets with long term planning and supply chain optimization.
  • Building an Enterprise Eventing Framework Recorded: Sep 12 2019 61 mins
    Bryan Zelle, IT Manager, Centene
    Learn how Centene improved their ability to interact and engage with healthcare providers in real time with MongoDB and Confluent Platform.

    Centene is fundamentally modernizing its legacy monolithic systems to support distributed, real-time event-driven healthcare information processing. A key part of their architecture is the development of a universal eventing framework designed to accommodate transformation into an event-driven architecture (EDA).

    The business requirements within Centene's claims adjudication domain were solved leveraging the Kafka Stream DSL, Confluent Platform and MongoDB. Most importantly, Centene discusses how they plan on leveraging this framework to change their culture from batch processing to real-time stream processing.
  • How to Build an Apache Kafka® Connector Recorded: Sep 12 2019 54 mins
    Jeff Bean, Partner Solution Architect, Confluent
    Apache Kafka® is the technology behind event streaming which is fast becoming the central nervous system of flexible, scalable, modern data architectures. Customers want to connect their databases, data warehouses, applications, microservices and more, to power the event streaming platform. To connect to Apache Kafka, you need a connector!

    This online talk dives into the new Verified Integrations Program and the integration requirements, the Connect API and sources and sinks that use Kafka Connect. We cover the verification steps and provide code samples created by popular application and database companies. We will discuss the resources available to support you through the connector development process.

    This is Part 2 of 2 in Building Kafka Connectors - The Why and How
  • Why Build an Apache Kafka® Connector Recorded: Sep 10 2019 38 mins
    Sree Karuthody, Sr. Manager, Technology Partnerships, Confluent + Jeff Bean, Partner Solution Architect, Confluent
    Apache Kafka® is the technology behind event streaming which is fast becoming the central nervous system of flexible, scalable, modern data architectures. Customers want to connect their databases, data warehouses, applications, microservices and more, to power the event streaming platform. To connect to Apache Kafka, you need a connector!

    This online talk focuses on the key business drivers behind connecting to Kafka and introduces the new Confluent Verified Integrations Program. We’ll discuss what it takes to participate, the process and benefits of the program.
  • How to Build an Apache Kafka® Connector Recorded: Aug 29 2019 55 mins
    Jeff Bean, Partner Solution Architect, Confluent
    Apache Kafka® is the technology behind event streaming which is fast becoming the central nervous system of flexible, scalable, modern data architectures. Customers want to connect their databases, data warehouses, applications, microservices and more, to power the event streaming platform. To connect to Apache Kafka, you need a connector!

    This online talk dives into the new Verified Integrations Program and the integration requirements, the Connect API and sources and sinks that use Kafka Connect. We cover the verification steps and provide code samples created by popular application and database companies. We will discuss the resources available to support you through the connector development process.

    This is Part 2 of 2 in Building Kafka Connectors - The Why and How
  • Apache Kafka® + Machine Learning for Supply Chain  Recorded: Aug 23 2019 59 mins
    Kai Waehner, Confluent + Graham Ganssle, Expero
    Automating multifaceted, complex workflows requires hybrid solutions like streaming analytics of IoT data, batch analytics like machine learning solutions, and real-time visualizations. Leaders in organizations who are responsible for global supply chain planning are responsible for working with and integrating with data from disparate sources around the world. Many of these data sources output information in real-time, which assists planners in operationalizing plans and interacting with manufacturing output. IoT sensors on manufacturing equipment and inventory control systems feed real-time processing pipelines to match actual production figures against planned schedules to calculate yield efficiency. 

    Using information from both real-time systems and batch optimization, supply chain managers are able to economize operations and automate tedious inventory and manufacturing accounting processes. Sitting on top of all these systems is a supply chain visualization tool, enabling users' visibility over the global supply chain. If you are responsible for key data integration initiatives, join for a detailed walk through of a customer's use of this system built using Confluent and Expero tools. 

    WHAT YOU'LL LEARN: 
    •See different use cases in automation industry and Industrial IoT (IIoT) where an event streaming platform adds business value. 
    •Understand different architecture options to leverage Apache Kafka and Confluent.
    •How to leverage different analytics tools and machine learning frameworks in a flexible and scalable way.
    •How real-time visualization ties together streaming and batch analytics for business users, interpreters, and analysts.
    •Understand how streaming and batch analytics optimize the supply chain planning workflow.
    •Conceptualize the intersection between resource utilization and manufacturing assets with long term planning and supply chain optimization.
  • Why Build an Apache Kafka® Connector Recorded: Aug 22 2019 39 mins
    Sree Karuthody, Sr. Manager, Technology Partnerships, Confluent + Jeff Bean, Partner Solution Architect, Confluent
    Apache Kafka® is the technology behind event streaming which is fast becoming the central nervous system of flexible, scalable, modern data architectures. Customers want to connect their databases, data warehouses, applications, microservices and more, to power the event streaming platform. To connect to Apache Kafka, you need a connector!

    This online talk focuses on the key business drivers behind connecting to Kafka and introduces the new Confluent Verified Integrations Program. We’ll discuss what it takes to participate, the process and benefits of the program.
  • Introducing Events and Stream Processing into Nationwide Building Society. Recorded: Aug 20 2019 49 mins
    Rob Jackson, Head of Application Architecture at Nationwide Building Society
    Open Banking regulations compel the UK’s largest banks, and building societies to enable their customers to share personal information with other regulated companies securely. As a result companies such as Nationwide Building Society are re-architecting their processes and infrastructure around customer needs to reduce the risk of losing relevance and the ability to innovate.

    In this online talk, you will learn why, when facing Open Banking regulation and rapidly increasing transaction volumes, Nationwide decided to take load off their back-end systems through real-time streaming of data changes into Apache Kafka®. You will hear how Nationwide started their journey with Apache Kafka®, beginning with the initial use case of creating a real-time data cache using Change Data Capture, Confluent Platform and Microservices. Rob Jackson, Head of Application Architecture, will also cover how Confluent enabled Nationwide to build the stream processing backbone that is being used to re-engineer the entire banking experience including online banking, payment processing and mortgage applications.
We provide a central nervous system for streaming real-time data.
Confluent, founded by the creators of open source Apache Kafka®, provides the leading streaming platform that enables enterprises to maximize the value of data. Confluent Platform empowers leaders in industries such as retail, logistics, manufacturing, financial services, technology and media, to move data from isolated systems into a real-time data pipeline where they can act on it immediately.

Backed by Benchmark, Index Ventures and Sequoia, Confluent is based in Palo Alto, California. To learn more, please visit www.confluent.io.

Embed in website or blog

Successfully added emails: 0
Remove all
  • Title: No More Silos: Integrating Databases into Apache Kafka®
  • Live at: Apr 11 2019 9:00 pm
  • Presented by: Robin Moffatt, Developer Advocate, Confluent
  • From:
Your email has been sent.
or close