Name: Part 2: Steps to Building a Streaming ETL Pipeline with Apache Kafka® and KSQL
Start: 2018-06-06T16:00:00Z
End: 2018-06-06T16:00:59.000Z
Location: BrightTALK
Rating: 0

Confluent is building the foundational platform for data in motion. Our cloud-native offering is designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organisation. With Confluent, organisations can create a central nervous system to innovate and win in a digital-first world.

2024 allows retailers to leverage new technologies to provide customers with a more personalised, efficient, and seamless shopping experience and optimise business operations to drive profit. Making the right technology investments and data strategy choices will enable success in a challenging financial environment. Deploying a data streaming platform throughout your business allows a wide range of use cases that address some of the most challenging business objectives of 2024. 


Join the Confluent team for a webinar that will delve into the most impactful data streaming use cases, leaving you clear on the value these can deliver to your business and how to get started with deploying a data streaming platform. 


By attending, you will learn how a data streaming platform can enable the following:



•Last Mile Delivery: Discover how real-time data can enable retailers to optimise their last-mile delivery operations, reducing delivery times, improving efficiency, and providing a more personalised and convenient delivery experience.


•Customer Personalisation: Real-time data can enable retailers to provide a more personalised and relevant shopping experience for customers, improving customer satisfaction and loyalty and ultimately driving sales.


•Supply Chain and Logistics: A data streaming platform can enable retailers to optimise their supply chain and logistics operations, reducing costs, improving efficiency, and ensuring that products are delivered to customers on time.


•Dynamic Pricing: Use real-time data to enable retailers to implement dynamic pricing strategies responsive to changes in customer demand, competition, inventory levels, and customer behaviour and preferences.


•AI and Analytics: Discover how to train ML models with real-time data that is constantly streaming so your applications can act on every event with context and intelligence.

The Top Data Streaming Use Cases for Retail

Data mesh is a new approach to designing data architectures by embracing organisational and data-centric constructs, such as data management and governance. The idea is that data should be easily accessible and interconnected across the entire business.

In this talk, will cover:
 
- The basics of building a streaming data mesh with Kafka & how Confluent enables this. 
- The four principles of the data mesh: domain-driven decentralisation, data as a product, self-service data platform, and federated governance.
- The differences between working with event streams versus centralised approaches.
- How to onboard data from existing systems into a mesh, modelling the communication within the mesh
- How to deal with changes to your domain’s “public” data, give examples of global standards for governance
- The importance of taking a product-centric view of data sources and the data sets they share.

Apache Kafka®, Confluent, and the Data Mesh

This webinar explores the state of data streaming for the Financial Services Industry in 2023. Join this webinar to learn more about:

- What disruptive trends in financial services are driving the pace of innovation
- How data streaming provides the ability to integrate and correlate data in real-time at any scale.
- Which enterprise architectures are trending in the FinServ industry globally

The State of Data Streaming for Financial Services in 2023

GenAI has the potential to revolutionize industries by improving efficiency, reducing costs, and providing better outcomes for individuals and businesses. This potential has come into being as a result of the availability of large amounts of data and the development of powerful computing systems that can process that data quickly. However, the current hype around AI is distracting businesses from its real value - the capacity to solve real-world problems and address practical challenges. As the AI hype dies down and productive tools begin to emerge, the integration between streaming data and AI will grow stronger.



Join Italo Nesi, Advisory Solutions Engineer at Confluent, for a webinar where he will discuss:



• Leading use cases for data streaming and GenAI
• How to tap into continuously enriched, trustworthy data streams to quickly scale and build real-time AI applications
• How to build governed, secured, and trusted AI
• How Confluent's depth of integration with Apache Flink makes it easier to build end-to-end AI pipelines that can handle data ingestion, processing, and analysis.

• The relationship between Machine Learning and Generative AI data products

Streaming & The Future: Artificial Intelligence as a Data Product

Data Streaming with Apache Kafka and Apache Flink is one of the world's most relevant and talked about paradigms in technology. With the buzz around this technology growing, join Kai Waehner, Global Field CTO at Confluent, to hear his predictions for the 'Top Five Trends for Data Streaming in 2024'. In this webinar, Kai will discuss:

• Data Sharing: Decentralized data mesh architecture with independent data products building or buying cloud-native software components. Collaborating within and outside the organization with data sharing using streaming protocols like Kafka, APIs like REST/HTTP, and open standards like AsyncAPI

• Data Contracts: Policy enforcement for structure, integrity constraints, metadata, rules, and other requirements to ensure data governance and compliance across the end-to-end data streaming pipelines

• Serverless Stream Processing: Focusing on business value leveraging a fully managed, integrated, and secure infrastructure for data correlation to implement streaming ETL and application development.

• Multi-Cloud: Data bridge, migrations, or disaster recovery across cloud providers with reliable uni- or bidirectional data synchronization

• AI: Data fabric for machine learning infrastructures to train models, real-time model scoring, and integrating 3rd party services like GenAI LLMs or SaaS offerings

Top Five Data Streaming Trends for 2024

Are you struggling to scale your data streaming use cases and break down organisational silos? Join Lyndon Hedderly and Burhan Nazir of Confluent as they share their expertise on deploying enterprise-wide data streaming and accelerating the speed to realising measurable business value derived from a data-streaming investment.

With years of experience enabling companies to deploy data streaming, Lyndon and Burhan have identified the missing factors in data streaming deployments, including a lack of general business visibility, an absence of allocated budget for Kafka or data streaming, and no senior-level or C-suite sponsorship.

During this webinar, you will learn five top tips on successfully implementing an enterprise-wide streaming platform. You will also discover how organisations are leveraging data streaming to scale their operations and break down silos.

Take advantage of this opportunity to learn from the experts in data streaming. Register now to join Lyndon and Burhan and take the first step towards realising the full potential of your data.

Five Tips for Enabling Enterprise-Wide Data Streaming

Data pipelines do much of the heavy lifting in organisations for integrating, transforming, and preparing the data for subsequent use in downstream systems for operational use cases. Despite being critical to the data value stream, data pipelines fundamentally haven’t evolved in the last few decades. These legacy pipelines are holding organisations back from getting value from their data as real-time streaming becomes essential.
 
This webinar will walk through the story of a bank that uses an Oracle database to store sensitive customer information and RabbitMQ as the message broker for credit card transaction events. Their goal is to perform real-time analysis on credit card transactions to flag fraudulent transactions and push suspicious activity flags to MongoDB Atlas, their modern cloud-native database that powers their in-app mobile notifications.
 
To illustrate this use case, you’ll see a live demo of:

• Confluent’s fully managed connectors for Oracle CDC Source and RabbitMQ Source to stream the data in real-time ksqlDB to merge the two data sources, generating a unified view of customers and their credit card activity and flagging fraudulent transactions

• The fully managed MongoDB Atlas sink connector to load the aggregated and transformed data into MongoDB Atlas

• Along with the live demo and customer use case, you’ll learn about the challenges with batch-based data pipelines and the benefits of streaming data pipelines to power modern data flows.

How to Build Streaming Pipelines for Cloud Databases

According to SAP, over seventy per cent of the world's business transactions touch an SAP system, cementing SAP's role in providing business-critical software. This business criticality requires faultless integration across cloud and on-premise software deployments to ensure a secure, reliable, uninterrupted service. Architecting a robust event-driven ecosystem is central to achieving this. Rojo constructed for a global manufacturer a Confluent solution for real-time distribution of data alongside a user-friendly interface.

Join Rojo and Confluent on October 26 for a 'how to' webinar in which  Rojo will outline how to optimise the use of Apache Kafka in your SAP integration initiatives. By attending, you will learn about the following:
 

- Traditional middleware (ETL/ESB).
- Web services (SOAP/REST)
- 3rd party turnkey solutions
- Kafka-native connectivity via Kafka Connect
- Self-written solutions using SAP SDKs
- A custom Confluent Kafka solution by Rojo - data streaming

How to Unleash the Power of Data with SAP® and Apache Kafka®

Data Mesh has been gaining traction over the past year or two - and with good reason. With organizations constantly exploring the potential of data, it’s becoming increasingly clear that it’s Data Mesh that may be the key to enabling businesses to truly realize their data-driven initiatives.

In this Confluent webinar, experts are coming together to discuss how Data Mesh has emerged as a paradigm shift in data architecture and to share how teams can be empowered to harness the power of their data at scale. Join Lyndon Hedderly - Director of Customer Solutions at Confluent and Piethein Strengholt - Chief Data Officer (The Netherlands) at Microsoft as they explore:

- The core principles and fundamentals of Data Mesh 
- Best practices for implementing a Data Mesh into your organization, including how businesses can foster a culture of data ownership, accountability, and autonomy 
- What Data Mesh challenges may arise and how to navigate them 
- The stakeholders involved in implementing and managing a Data Mesh
- The business value and new revenue streams a Data Mesh can help realize

Unlocking the Power of Data Mesh: Decentralizing and Democratizing Data

80% of consumer analytics happens on data that’s older than 1 day. Learn how to extract actionable real time insights for your shoppers and products through a streaming analytic application. 

With Confluent and Imply, you can build a streaming analytic application that provides a 360 view of your customer’s buying behavior in 1 hour.

1. Aggregate all your data to build timely context around each customer profile
2. Visualize buying patterns instantly
3. Identify top selling products and refill inventory
4. Identify potential revenue streams in real time
5. Connect teams for shared insights with a consistent view of both streaming and historical data

Join us for an exclusive opportunity to hear from the thought leaders of Kafka and Apache Druid on how Confluent Cloud and Imply Polaris enable customers to leverage the power of interactive streaming platforms to accelerate real time data analytics. Bring your laptop for this interactive workshop

Who should attend:

1. Application developers and engineers interested in real time analytics
2. Engineers, developers, architects, or anyone interested in understanding real time system design and use cases. 
3. Any skill or maturity level with event-drive systems, Apache Kafka, or messaging in general. 

Please note: 
Confluent Cloud account and Imply Polaris account are required for the lab session. 

Sign-up for Confluent Cloud. New signups receive $400 to spend on Confluent Cloud during their first 60 days.

Sign-up for Imply Polaris here. New signups receive free usage for the first 30 days valued at $1872.

Building a Streaming Analytics Application in 1 Hour

In this talk, we'll build a streaming data pipeline using nothing but our bare hands, the Kafka Connect API and KSQL. We'll stream data in from MySQL, transform it with KSQL and stream it out to Elasticsearch. Options for integrating databases with Kafka using CDC and Kafka Connect will be covered as well.

This is part 2 of 3 in Streaming ETL - The New Data Integration series.

Part 2: Steps to Building a Streaming ETL Pipeline with Apache Kafka® and KSQL

Streaming

Big Data

Apache

Apache Kafka

Data Integration

Data Infrastructure

Data Architecture

APIs

Data Platform

Cloud computing has exploded over the past few years, delivering a previously unimagined level of workplace mobility and flexibility. The cloud computing community on BrightTALK is made up of thousands of engaged professionals learning from the latest cloud computing research and resources. Join the community to expand your cloud computing knowledge and have your questions answered in live sessions with industry experts and vendor representatives.

Cloud Computing

Increasing expectations for good governance, effective risk management and complex demands for corporate compliance are presenting a growing challenge for organizations of all sizes. Join industry thought leaders as they provide you with practical advice on how to implement successful risk and compliance management strategies across your organization. Browse risk management resources in the form of interactive webinars and videos and ask questions of expert GRC professionals.

IT Governance, Risk and Compliance

The data center management community focuses on the holistic management and optimization of the data center. From technologies such as virtualization and cloud computing to data center design, colocation, energy efficiency and monitoring, the BrightTALK data center management community provides the most up-to-date and engaging content from industry experts to better your infrastructure and operations. Engage with a community of your peers and industry experts by asking questions, rating presentations and participating in polls during webinars, all while you gain insight that will help you transform your infrastructure into a next generation data center.

Data Center Management

Practicing business intelligence allows your company to transform raw data into sets of insights for targeted business growth. The business intelligence and analytics community on BrightTALK is made up of thousands of data scientists, database administrators, business analysts and other data professionals. Find relevant webinars and videos on business analytics, business intelligence, data analysis and more presented by recognized thought leaders. Join the conversation by participating in live webinars and round table discussions.

Business Intelligence and Analytics

Join thousands of engaged IT professionals in the application management community on BrightTALK. Interact with your peers in relevant webinars and videos on the latest trends and best practices for application lifecycle management, application performance management and application development.

Application Management

The IT project management community on BrightTALK includes thousands of IT project and portfolio management professionals. Find relevant webinars and videos on agile methodologies, scrum strategy, project management processes and more. Attend live webinars or view on demand content presented by recognized thought leaders in the IT project management industry.

IT Project Management

Welcome to the big data and data management community on BrightTALK. Join thousands of data quality engineers, data scientists, database administrators and other professionals to find more information about the hottest topics affecting your data. Subscribe now to learn about efficiently storing, optimizing a complex infrastructure, developing governing policies, ensuring data quality and analyzing data to make better informed decisions. Join the conversation by watching live and on-demand webinars and take the opportunity to interact with top experts and thought leaders in the field.

Big Data and Data Management

As an IT professional, many of the problems you face are multifaceted, complex and don’t lend themselves to simple solutions. The information technology community features useful and free information technology resources. Join to browse thousands of videos and webinars on ITIL best practices, IT security strategy and more presented by leading CTOs, CIOs and other technology experts.

Part 2: Steps to Building a Streaming ETL Pipeline with Apache Kafka® and KSQL

Presented by

About this talk

More from this channel