Hi [[ session.user.profile.firstName ]]

End-to-End Integration from the IoT Edge to Confluent Cloud

This interactive whiteboard presentation discusses use cases leveraging the Apache Kafka® open source ecosystem as a streaming platform to process IoT data. The session shows architectural alternatives of how devices like cars, machines or mobile devices connect to Apache Kafka via IoT standards like MQTT or OPC-UA.

Learn how to analyze the IoT data either natively on Apache Kafka with Kafka Streams / KSQL or other tools leveraging Kafka Connect. Kai Waehner will also discuss the benefits of Confluent Cloud and other tools like Confluent Replicator or MQTT Proxy to build bidirectional real time integration from the edge to the cloud.

Watch now to:
-Understand end-to-end use cases from different industries where you integrate IoT devices with enterprise IT using open source technologies and standards
-See how Apache Kafka enables bidirectional end-to-end integration processing from IoT data to various backend applications in the cloud
-Compare different architectural alternatives and see their benefits and caveats
-Learn about various standards, APIs and tools of integrating and processing IoT data with different open source components of the Apache Kafka ecosystem
-Understand the benefits of Confluent Cloud, which provides a highly available and scalable Apache Kafka ecosystem as a managed service
Recorded Jul 16 2019 28 mins
Your place is confirmed,
we'll send you email reminders
Presented by
Kai Waehner, Technology Evangelist, Confluent + Konstantin Karantasis, Software Engineer, Confluent
Presentation preview: End-to-End Integration from the IoT Edge to Confluent Cloud

Network with like-minded attendees

  • [[ session.user.profile.displayName ]]
    Add a photo
    • [[ session.user.profile.displayName ]]
    • [[ session.user.profile.jobTitle ]]
    • [[ session.user.profile.companyName ]]
    • [[ userProfileTemplateHelper.getLocation(session.user.profile) ]]
  • [[ card.displayName ]]
    • [[ card.displayName ]]
    • [[ card.jobTitle ]]
    • [[ card.companyName ]]
    • [[ userProfileTemplateHelper.getLocation(card) ]]
  • Channel
  • Channel profile
  • Apache Kafka Architecture & Fundamentals Explained Dec 30 2019 2:00 pm UTC 57 mins
    Joe Desmond, Technical Trainer, Confluent
    This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput.

    This talk provides a comprehensive overview of Kafka architecture and internal functions, including:
    -Topics, partitions and segments
    -The commit log and streams
    -Brokers and broker replication
    -Producer basics
    -Consumers, consumer groups and offsets

    This session is part 2 of 4 in our Fundamentals for Apache Kafka series.
  • Benefits of Stream Processing and Apache Kafka® Use Cases Dec 19 2019 2:00 pm UTC 56 mins
    Mark Fei, Technical Trainer, Confluent
    This talk explains how companies are using event-driven architecture to transform their business and how Apache Kafka serves as the foundation for streaming data applications.

    Learn how major players in the market are using Kafka in a wide range of use cases such as microservices, IoT and edge computing, core banking and fraud detection, cyber data collection and dissemination, ESB replacement, data pipelining, ecommerce, mainframe offloading and more.

    Also discussed in this talk are the differences between Apache Kafka and Confluent Platform.
  • Apache Kafka Architecture & Fundamentals Explained Dec 16 2019 11:00 am UTC 57 mins
    Joe Desmond, Technical Trainer, Confluent
    This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput.

    This talk provides a comprehensive overview of Kafka architecture and internal functions, including:
    -Topics, partitions and segments
    -The commit log and streams
    -Brokers and broker replication
    -Producer basics
    -Consumers, consumer groups and offsets

    This session is part 2 of 4 in our Fundamentals for Apache Kafka series.
  • Enabling Event Streaming at AO.com Dec 10 2019 11:00 am UTC 50 mins
    Jon Vines, AO.com
    Learn how AO.com are enabling real-time event-driven applications to improve customer experience using Confluent Platform.
    The introduction of Apache Kafka and the Confluent platform is supporting AO.com in modernizing the technical approach to delighting its customers. A key part of this enablement is the introduction of an event-streaming eco-system enabling event-driven applications and architecture.
    Throughout this session, we’ll look at the challenges AO.com faced when looking to adopt Kafka, their use of Confluent Platform including Kafka Connect and KSQL and the adoption of Confluent Cloud. We’ll look at the first steps, where the team are at now and what the future looks like.
  • Being an Apache Kafka® Expert in a Multi-Cloud World Recorded: Dec 3 2019 31 mins
    Ricardo Ferreira, Developer Advocate, Confluent
    Apache Kafka is an amazing piece of technology, that has been furiously adopted by companies all around the world to implement event-driven architectures. While its adoption continues to increase, the reality is that most developers often complain about the complexity of managing the clusters by themselves, which seriously decreases their ability to be agile.

    This 30-minute demo will introduce Confluent Cloud, a service that offers Apache Kafka and the Confluent Platform so developers can focus on what they do best: the coding part. We will show you how to quickly reuse code written for standard Kafka APIs to connect to Confluent Cloud and how an event-streaming application is built and deployed.
  • How Apache Kafka® Works Recorded: Nov 27 2019 62 mins
    Michael Bingham, Technical Trainer, Confluent
    Pick up best practices for developing applications that use Apache Kafka, beginning with a high level code overview for a basic producer and consumer. From there we’ll cover strategies for building powerful stream processing applications, including high availability through replication, data retention policies, producer design and producer guarantees.

    We’ll delve into the details of delivery guarantees, including exactly-once semantics, partition strategies and consumer group rebalances. The talk will finish with a discussion of compacted topics, troubleshooting strategies and a security overview.

    This session is part 3 of 4 in our Fundamentals for Apache Kafka series.
  • Best Practices for Streaming IoT Data with MQTT and Apache Kafka® Recorded: Nov 14 2019 60 mins
    Kai Waehner, Technology Evangelist, Confluent + Dominik Obermaier, CTO, HiveMQ
    Organizations today are looking to stream IoT data to Apache Kafka. However, connecting tens of thousands or even millions of devices over unreliable networks can create some architecture challenges.

    In this session, we will identify and demo some best practices for implementing a large scale IoT system that can stream MQTT messages to Apache Kafka.
  • Integrating Apache Kafka Into Your Environment Recorded: Nov 4 2019 52 mins
    Russ Sayers, Sr. Technical Trainer, Confluent
    Integrating Apache Kafka with other systems in a reliable and scalable way is a key part of an event streaming platform. This session will show you how to get streams of data into and out of Kafka with Kafka Connect and REST Proxy, maintain data formats and ensure compatibility with Schema Registry and Avro, and build real-time stream processing applications with Confluent KSQL and Kafka Streams.

    This session is part 4 of 4 in our Fundamentals for Apache Kafka series.
  • How Apache Kafka® Works Recorded: Oct 28 2019 63 mins
    Michael Bingham, Technical Trainer, Confluent
    Pick up best practices for developing applications that use Apache Kafka, beginning with a high level code overview for a basic producer and consumer. From there we’ll cover strategies for building powerful stream processing applications, including high availability through replication, data retention policies, producer design and producer guarantees.

    We’ll delve into the details of delivery guarantees, including exactly-once semantics, partition strategies and consumer group rebalances. The talk will finish with a discussion of compacted topics, troubleshooting strategies and a security overview.

    This session is part 3 of 4 in our Fundamentals for Apache Kafka series.
  • Design and Implementation of Incremental Cooperative Rebalancing Recorded: Oct 24 2019 58 mins
    Speaker: Konstantine Karantasis, Software Engineer, Confluent
    Since its initial release, the Kafka group membership protocol has offered Connect, Streams and Consumer applications an ingenious and robust way to balance resources among distributed processes. The process of rebalancing, as it’s widely known, allows Kafka APIs to define an embedded protocol for load balancing within the group membership protocol itself.

    Until now, rebalancing has been working under the simple assumption that every time a new group generation is created, the members join after first releasing all of their resources, getting a whole new load assignment by the time the new group is formed. This allows Kafka APIs to provide task fault-tolerance and elasticity on top of the group membership protocol.

    However, due to its side-effects on multi-tenancy and scalability this simple approach in rebalancing, also known as stop-the-world effect, is limiting larger scale deployments. Because of stop-the-world, application tasks get interrupted only for most of them to receive the same resources after rebalancing. In this technical deep dive, we’ll discuss the proposition of Incremental Cooperative Rebalancing as a way to alleviate stop-the-world and optimize rebalancing in Kafka APIs.

    This talk will cover:
    -The internals of Incremental Cooperative Rebalancing
    -Uses cases that benefit from Incremental Cooperative Rebalancing
    -Implementation in Kafka Connect
    -Performance results in Kafka Connect clusters
  • Apache Kafka Architecture & Fundamentals Explained Recorded: Oct 21 2019 57 mins
    Joe Desmond, Technical Trainer, Confluent
    This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput.

    This talk provides a comprehensive overview of Kafka architecture and internal functions, including:
    -Topics, partitions and segments
    -The commit log and streams
    -Brokers and broker replication
    -Producer basics
    -Consumers, consumer groups and offsets

    This session is part 2 of 4 in our Fundamentals for Apache Kafka series.
  • Scaling Security on 100s of Millions of Mobile Devices Using Kafka & Scylla Recorded: Oct 16 2019 48 mins
    Richard Ney, Sr. Staff Engineer, Lookout + Eyal Gutkind, VP Solutions, ScyllaDB + Jeff Bean, Solutions Architect, Confluent
    Join mobile cybersecurity leader Lookout as they talk through their data ingestion journey.

    Lookout enables enterprises to protect their data by evaluating threats and risks at post-perimeter endpoint devices and providing access to corporate data after conditional security scans. Their continuous assessment of device health creates a massive amount of telemetry data, forcing new approaches to data ingestion. Learn how Lookout changed its approach in order to grow from 1.5 million devices to 100 million devices and beyond, by implementing Confluent Platform and switching to Scylla.
  • Benefits of Stream Processing and Apache Kafka® Use Cases Recorded: Oct 14 2019 57 mins
    Mark Fei, Technical Trainer, Confluent
    This talk explains how companies are using event-driven architecture to transform their business and how Apache Kafka serves as the foundation for streaming data applications.

    Learn how major players in the market are using Kafka in a wide range of use cases such as microservices, IoT and edge computing, core banking and fraud detection, cyber data collection and dissemination, ESB replacement, data pipelining, ecommerce, mainframe offloading and more.

    Also discussed in this talk are the differences between Apache Kafka and Confluent Platform.
  • How to Unlock your Mainframe Data with Confluent, Attunity and Apache Kafka Recorded: Oct 3 2019 51 mins
    Simon Leigh, Confluent. Martin Hamilton, Attunity
    Large enterprises, government agencies, and many other organisations rely on mainframe computers to deliver the core systems managing some of their most valuable and sensitive data. However, the processes and cultures around a mainframe often prevent the adoption of the agile, born-on-the web practices that have become essential to developing cutting edge internal and customer-facing applications. Mainframes also represent significant, long-term investments in terms of time, money, people and possibly even decades worth of stored data. This webinar will help you understand that you can offload and unlock your mainframe data and equip your business for the modern data-driven environment.

    By attending this webinar, you will learn:

    1. How to access the depth and richness of insights held in the data within your mainframe
    2. How to bring real-time data from mainframes efficiently with CDC technology partners for Confluent Enterprise and Apache Kafka
    3. How to reduce the costs and complexity of querying a mainframe database using the unique change data capture function
    4. How to leverage Apache Kafka’s modern distributed architecture to move mainframe data in real-time
    5. How Attunity Replicate software is leveraged to stream data changes to Kafka
  • How to Fail at Kafka Recorded: Oct 2 2019 19 mins
    Pete Godfrey, Systems Engineer, Confluent
    Apache Kafka® is used by thousands of companies across the world but, how difficult is it to operate? Which parameters do you need to set? What can go wrong? This online talk is based on real-world experience of Kafka deployments and explores a collection of common mistakes that are made when running Kafka in production and some best practices to avoid them.

    Watch now to learn:

    -How to ensure your Kafka data is never lost
    -How to write code to cope when things go wrong
    -How to ensure data governance between producers and consumers
    -How to monitor your cluster

    Join Apache Kafka expert, Pete Godfrey, for this engaging talk and delve into best practice ideas and insights.
  • SIEM Modernization: Build a Situationally Aware Organization with Apache Kafka® Recorded: Sep 25 2019 35 mins
    Jeffrey Needham, Confluent
    Of all security breaches, 85% are conducted with compromised credentials, often at the administration level or higher. A lot of IT groups think “security” means authentication, authorization and encryption (AAE), but these are often tick-boxes that rarely stop breaches. The internal threat surfaces of data streams or disk drives in a raidset in a data center are not the threat surface of interest.

    Cyber or Threat organizations must conduct internal investigations of IT, subcontractors and supply chains without implicating the innocent. Therefore, they are organizationally air-gapped from IT. Some surveys indicate up to 10% of IT is under investigation at any given time.

    Deploying a signal processing platform, such as Confluent Platform, allows organizations to evaluate data as soon as it becomes available enabling them to assess and mitigate risk before it arises. In Cyber or Threat Intelligence, events can be considered signals, and when analysts are hunting for threat actors, these don't appear as a single needle in a haystack, but as a series of needles. In this paradigm, streams of signals aggregate into signatures. This session shows how various sub-systems in Apache Kafka can be used to aggregate, integrate and attribute these signals into signatures of interest.

    Watch now to learn:
    -The current threat landscape
    -The difference between Security and Threat Intelligence
    -The value of Confluent Platform as an ideal complement to hardware endpoint detection systems and batch-based SIEM warehouses
  • Apache Kafka® + Machine Learning for Supply Chain Recorded: Sep 24 2019 58 mins
    Kai Waehner, Confluent + Graham Ganssle, Expero
    Automating multifaceted, complex workflows requires hybrid solutions like streaming analytics of IoT data, batch analytics like machine learning solutions, and real-time visualizations. Leaders in organizations who are responsible for global supply chain planning are responsible for working with and integrating with data from disparate sources around the world. Many of these data sources output information in real-time, which assists planners in operationalizing plans and interacting with manufacturing output. IoT sensors on manufacturing equipment and inventory control systems feed real-time processing pipelines to match actual production figures against planned schedules to calculate yield efficiency. 

    Using information from both real-time systems and batch optimization, supply chain managers are able to economize operations and automate tedious inventory and manufacturing accounting processes. Sitting on top of all these systems is a supply chain visualization tool, enabling users' visibility over the global supply chain. If you are responsible for key data integration initiatives, join for a detailed walk through of a customer's use of this system built using Confluent and Expero tools. 

    WHAT YOU'LL LEARN: 
    •See different use cases in automation industry and Industrial IoT (IIoT) where an event streaming platform adds business value. 
    •Understand different architecture options to leverage Apache Kafka and Confluent.
    •How to leverage different analytics tools and machine learning frameworks in a flexible and scalable way.
    •How real-time visualization ties together streaming and batch analytics for business users, interpreters, and analysts.
    •Understand how streaming and batch analytics optimize the supply chain planning workflow.
    •Conceptualize the intersection between resource utilization and manufacturing assets with long term planning and supply chain optimization.
  • Building an Enterprise Eventing Framework Recorded: Sep 12 2019 61 mins
    Bryan Zelle, IT Manager, Centene
    Learn how Centene improved their ability to interact and engage with healthcare providers in real time with MongoDB and Confluent Platform.

    Centene is fundamentally modernizing its legacy monolithic systems to support distributed, real-time event-driven healthcare information processing. A key part of their architecture is the development of a universal eventing framework designed to accommodate transformation into an event-driven architecture (EDA).

    The business requirements within Centene's claims adjudication domain were solved leveraging the Kafka Stream DSL, Confluent Platform and MongoDB. Most importantly, Centene discusses how they plan on leveraging this framework to change their culture from batch processing to real-time stream processing.
  • How to Build an Apache Kafka® Connector Recorded: Sep 12 2019 54 mins
    Jeff Bean, Partner Solution Architect, Confluent
    Apache Kafka® is the technology behind event streaming which is fast becoming the central nervous system of flexible, scalable, modern data architectures. Customers want to connect their databases, data warehouses, applications, microservices and more, to power the event streaming platform. To connect to Apache Kafka, you need a connector!

    This online talk dives into the new Verified Integrations Program and the integration requirements, the Connect API and sources and sinks that use Kafka Connect. We cover the verification steps and provide code samples created by popular application and database companies. We will discuss the resources available to support you through the connector development process.

    This is Part 2 of 2 in Building Kafka Connectors - The Why and How
  • Why Build an Apache Kafka® Connector Recorded: Sep 10 2019 38 mins
    Sree Karuthody, Sr. Manager, Technology Partnerships, Confluent + Jeff Bean, Partner Solution Architect, Confluent
    Apache Kafka® is the technology behind event streaming which is fast becoming the central nervous system of flexible, scalable, modern data architectures. Customers want to connect their databases, data warehouses, applications, microservices and more, to power the event streaming platform. To connect to Apache Kafka, you need a connector!

    This online talk focuses on the key business drivers behind connecting to Kafka and introduces the new Confluent Verified Integrations Program. We’ll discuss what it takes to participate, the process and benefits of the program.
We provide a central nervous system for streaming real-time data.
Confluent, founded by the creators of open source Apache Kafka®, provides the leading streaming platform that enables enterprises to maximize the value of data. Confluent Platform empowers leaders in industries such as retail, logistics, manufacturing, financial services, technology and media, to move data from isolated systems into a real-time data pipeline where they can act on it immediately.

Backed by Benchmark, Index Ventures and Sequoia, Confluent is based in Palo Alto, California. To learn more, please visit www.confluent.io.

Embed in website or blog

Successfully added emails: 0
Remove all
  • Title: End-to-End Integration from the IoT Edge to Confluent Cloud
  • Live at: Jul 16 2019 7:35 pm
  • Presented by: Kai Waehner, Technology Evangelist, Confluent + Konstantin Karantasis, Software Engineer, Confluent
  • From:
Your email has been sent.
or close