Databricks and StreamSets: Manage Big Data Pipelines in the Cloud

Presented by

Hiral Jasani, Senior Partner Marketing Manager (Databricks) Rupal Shah, Director of Cloud Services (StreamSets)

About this talk

Whether you are cloud-native or migrating to the cloud, enterprises are looking for speed and agility. Databricks and StreamSets have partnered to bring rapid data pipeline design and testing to critical cloud workloads. Together, they bring the power of Apache Spark™ to a broad audience with a logical and visual, UI-based pipeline development tool. This allows more users to leverage Apache Spark™ and Delta Lake with confidence, reliability and unmatched performance in the cloud. In this webinar, we will discuss: Using a drag-and-drop interface for pipeline development to continuously ingest and stream data into Delta Lake on Databricks, How Delta Lake helps make cloud data more reliable with features like ACID-compliant transactions, schema enforcement and scalable metadata handling, How to migrate on prem Data Lake workloads (e.g. Hadoop) to cloud services and easily manage compute resources using Databricks’ optimized auto-scaling for compute resources.

Related topics:

More from this channel

Upcoming talks (0)
On-demand talks (32)
Subscribers (3428)
The StreamSets DataOps platform enables companies to build, execute, operate and protect batch and streaming dataflows. It is powered by StreamSets Data Collector, award-winning open source software with approximately 2,000,000 downloads to date from thousands of companies. The commercial StreamSets Control Hub is the platform's cloud-native control plane through which enterprises design, monitor and manage complex data movement that is executed by multiple Data Collectors. Unique Intelligent Pipeline technology automatically inspects the data in motion, detecting unexpected changes, errors and sensitive data in-stream. Global 2000 customers use StreamSets for data lake ingestion, Apache Kafka enablement, cybersecurity, IoT, customer 360, GDPR compliance and more. In 2017, the company tripled its customer count and quadrupled revenues.