Name: Building a Real-time Streaming ETL Framework Using ksqlDB and NoSQL
Start: 2021-05-14T17:00:00Z
End: 2021-05-14T17:00:52.000Z
Location: BrightTALK
Rating: 0

The Scylla NoSQL database embraces a shared-nothing approach that increases throughput and storage capacity as much as 10X. It comes in open source, enterprise and database-as-a-service options. Comcast, Discord, Grab, Medium, Starbucks, Ola Cabs, Samsung, IBM, Investing.com, Zillow and many more leading companies have adopted Scylla to realize order-of-magnitude performance improvements and reduce hardware costs. For more information: ScyllaDB.com.

Disney+ Hotstar is the fastest growing branch of Disney+. Join Disney+ Hotstar Architect Vamsi Subhash and senior data engineer Balakrishnan Kaliyamoorthy to learn:

- How Disney+ Hotstar architected their systems to handle massive data loads
- Why they chose to replace both Redis and Elasticsearch
- Their requirements for massively scalable data infrastructure and evolving data models
- How they migrated their data to Scylla Cloud, ScyllaDB’s fully managed NoSQL database-as-a-service, without suffering downtime

Disney+ Hotstar: Scaling NoSQL for Millions of Video On-Demand Users

Traditional relational databases provide strong consistency, but they can be difficult to scale. Non-relational databases scale more easily, but generally offer less strict consistency guarantees. Now a new class of database, known as NewSQL, promises both strong consistency and the ability to scale.
 
In this webinar, ScyllaDB developer evangelist Avishai Ish Shalom will compare NoSQL and NewSQL databases. We will look at the significant architectural differences between the two; tradeoffs between availability, scalable performance and consistency; data models and other considerations. 
 
We will also share benchmark results to display the performance implications of NoSQL versus NewSQL. For this purpose we will compare Scylla with best-in-class NewSQL database CockroachDB.
 
By attending this webinar you’ll get: 
- An overview of NoSQL and NewSQL
- Highlight of architectural differences between NoSQL and NewSQL
- Tradeoffs between scalable performance and consistency in NoSQL and NewSQL
- YCSB Benchmark results for Scylla and CockroachDB

NoSQL and NewSQL: Tradeoffs Between Scalable Performance & Consistency

Learn the secrets behind how Russia’s #1 internet provider achieves high database performance at low cost.

What if you could store huge amounts of data using traditional low-cost Hard Disk Drives (HDDs) while still maintaining low single-digit millisecond latencies? That’s just what they did at Mail.Ru, the largest email service provider in Russia.

Join Kirill Alekseev, software engineering team lead at Mail.Ru, for an overview of the extremely efficient approach his team took when implementing a high-load NoSQL service for storing user actions. You’ll see how Mail.Ru uses Scylla to serve up to 240k writes per second with 95 percentile latencies of less than 2ms – using a Scylla cluster running on HDDs.

From attending this session you’ll learn:

- How to build a large time-series storage with inexpensive HDDs while maintaining consistent low latencies under heavy write load, including the hardware and cluster setup
- Problems they encountered with their HDD setup and how they solved them
- Overview of the service and the logic it implies: how they handled retries and sped up API response with concurrent reads
- Tips & tricks they picked up along the way

Fast NoSQL on HDDs?

Kyle Kingsbury (aka ‘Aphyr’) tortures databases for a living. He pokes and prods and stresses them with his Jepsen test to look for safety violations across distributed systems. Kyle worked with the ScyllaDB team to identify multiple issues in Scylla's lightweight transactions (LWT) implementation, including split-brain behavior and write loss in healthy clusters.

Tune in to learn about distributed database testing techniques, Scylla’s concurrency properties, Kyle’s deep dive into transaction analysis and how to use Scylla safely. Kyle will talk about the specific issues he found in Scylla, and how the ScyllaDB team addressed them.

Torture Testing NoSQL: Putting Scylla to the Jepsen Test

Consuming messages from multiple sources in a scalable service sharing the same Scylla cluster can be problematic, especially when those sources provide data from two queues each and cannot deliver guaranteed in-order messages. At Zillow, we provide the Scylla write timestamp and employ a couple other tricks at Zillow to provide correct and consistent data to our consuming services and avoid doing transactions. Additionally, the Zillow data model is a simple compound key that contains a large binary Avro document.

Zillow: Optimistic Concurrency Using Write-time Timestamps

Ticketmaster is part of Live Nation Entertainment, the world's leading live entertainment company. Learn why they went with Scylla after conducting performance testing between Scylla, Apache Cassandra and DataStax Enterprise.

Ticketmaster Performance Test: Scylla, DataStax DSE and Apache Cassandra

Scylla Monitoring has been enhanced to be a one stop shop to monitor Scylla. To achieve this, scylla-monitoring now includes a CQL interface and log collection analysis. The combination of matrices, alerts, logs & CQL allows not only a single point to view all the information but also to advise you what may be the source of a performance/latency degradation (malfunctioning node.)

The All-New Scylla Monitoring Stack

Change Data Capture (CDC) and the Kafka Scylla Connector now allow Scylla to act as a data producer for Kafka streams. Discover how combining Scylla with the Confluent platform allows to maximize the value of NoSQL data by introducing Scylla as a key component of event driven architecture and enables streaming database changes, enriching these updates via message transformations, and allowing users to efficiently run data pipelines.

Streaming Data from Scylla to Kafka

Cloud Native Databases are required to scale while serving the increase in online workload with a minimal disruption and complete it as fast as possible. In this session we will review the different components that are stressed in scaling scenarios and present work we have done over the year to improve Scylla’s elasticity as we enhance it to be a true Cloud Native Database.

Scylla’s Journey Towards Being an Elastic Cloud Native Database

The talk covers most of the performance enhancements introduced to Scylla over the past 12 months. As the throughput was very good before, we focused on Scylla’s behavior under all types of workloads and data models. Scylla improved its latency under all scenarios, improving the behaviors of data models such as large partitions and time series, improvement of the I/O scheduler and behavior of streaming and repair.

Scylla Performance Enhancements: Year in Review

Many NoSQL DBaaS vendors limit what cloud platform you can run on, the size of the data you can run and require you to over-provision cloud infrastructure resources while failing to deliver performance and low latency at scale.

In this session, we will compare the performance and Total Cost of Ownership (TCO) of competing NoSQL DBaaS offerings. We will also review how to migrate to Scylla Cloud, our fully managed database service.

You will learn:

- The true cost of ownership for selected NoSQL DBaaS offerings
- The 8 essentials for selecting a NoSQL DBaaS
- Migration options from Apache Cassandra, DynamoDB and other databases

The True Cost of NoSQL DBaaS Options

Scylla Kubernetes support is long anticipated. Powered by Kubernetes, Scylla becomes easy to manage. Scale out, rolling upgrade, automatic configuration change in canary deployment style and so forth. The talk will demonstrate Scylla Operator 1.0 and go over its supported feature set and describe the roadmap ahead.

The Magic of Scylla Operator 1.0 Powered by Kubernetes

Join us for a session showcasing Scylla Cloud and how easy it is to integrate and connect with your applications and forget about all aspects of managing, securing, scaling and maintaining a distributed, always-on database. We will demo Scylla Cloud and present deployment and account options over various cloud providers.

Scylla Cloud on Display

The Jepsen framework is a tool for testing databases, combining fault injection, generative testing, and sophisticated safety checkers to find concurrency bugs and availability issues. We worked with the ScyllaDB team to identify multiple safety issues in Scylla's lightweight transactions (LWT) layer, including split-brain behavior and write loss in healthy clusters. In this talk, we'll discuss testing techniques more generally, and what we found (and what was fixed) in ScyllaDB.

Scylla and Jepsen Tests

Gumgum is a global media company specializing in contextual intelligence. Find out how and why they moved to Scylla Cloud to meet their growing customer demands and deliver on their promise of future proofed solutions.

Scylla @ GumGum: Contextual Intelligence Ads

Disney+ Hotstar provides on-demand streaming to more than 18.5 million paid subscribers and 300 million monthly active users. Hotstar's India Premier League (IPL) coverage in 2020 made it the most widely streamed sports events in the world to date. Learn why they chose Scylla Cloud to replace both Redis and Elasticsearch, and how Hotstar data usage was migrated with zero downtime.

Scylla @ Disney+ Hotstar: On-Demand Streaming Service

Agreement in a distributed system is complicated but required. Scylla gained lightweight transactions through Paxos but the latter has a cost of 3X roundtrips. Raft can allow consistent transactions without the performance penalty. Beyond LWT, we plan to integrate Raft with most aspects of Scylla making a leap forward in manageability and consistency.

Raft in Scylla

The ScyllaDB team has accomplished a lot since our last Scylla Summit. For example, our DynamoDB API is now GA, running on any cloud or on-prem, and certified as AWS Outposts ready. Our team is busy working on even bigger advancements to widen the gap between Scylla and other NoSQL databases. Join ScyllaDB CEO and co-founder Dor Laor to hear about these new initiatives and to see what the future of Scylla holds.

Prepare to Love Scylla Even More

Event streaming applications unlock new benefits by combining various data feeds. However, getting actionable insights in a timely fashion has remained a challenge, as the data has been siloed in disparate systems. ksqlDB solves this by providing an interactive SQL interface that can seamlessly combine and transform data from various sources.

In this webinar, we will show how streaming queries of high throughput NoSQL systems can derive insights from various push/pull queries via ksqlDB's User-Defined Functions, Aggregate Functions and Table Functions.

Watch this to learn:
- Real-world examples of the benefits of using a streaming database like ksqlDB and seamlessly combining data from Kafka & Cassandra/Scylla (NoSQL).
- The functionality of ksqlDB via push/pull queries and UDFs/UDAFs/UDTFs.
- The ease with which data stored in a NoSQL database can be transformed using ksqlDB and then persisted back for long-term storage.

Building a Real-time Streaming ETL Framework Using ksqlDB and NoSQL

NoSQL

ksqldb

streaming database

Kafka

event streaming

data feeds

Cloud computing has exploded over the past few years, delivering a previously unimagined level of workplace mobility and flexibility. The cloud computing community on BrightTALK is made up of thousands of engaged professionals learning from the latest cloud computing research and resources. Join the community to expand your cloud computing knowledge and have your questions answered in live sessions with industry experts and vendor representatives.

Cloud Computing

Enterprise applications have become a crucial piece of infrastructure for many businesses. The enterprise applications community on BrightTALK features the latest insights in enterprise application integration, enterprise application architecture and EAS software. Join the community to gain access to thousands of videos and webinars presented by recognized enterprise information systems experts and arm yourself with the knowledge you need to succeed.

Enterprise Applications

Join thousands of engaged IT professionals in the application management community on BrightTALK. Interact with your peers in relevant webinars and videos on the latest trends and best practices for application lifecycle management, application performance management and application development.

Application Management

The storage community on BrightTALK is made up of thousands of storage and IT professionals. Find relevant webinars and videos on storage architecture, cloud storage, storage virtualization and more presented by recognized thought leaders. Join the conversation by participating in live webinars and round table discussions.

Storage

Welcome to the big data and data management community on BrightTALK. Join thousands of data quality engineers, data scientists, database administrators and other professionals to find more information about the hottest topics affecting your data. Subscribe now to learn about efficiently storing, optimizing a complex infrastructure, developing governing policies, ensuring data quality and analyzing data to make better informed decisions. Join the conversation by watching live and on-demand webinars and take the opportunity to interact with top experts and thought leaders in the field.

Big Data and Data Management

As an IT professional, many of the problems you face are multifaceted, complex and don’t lend themselves to simple solutions. The information technology community features useful and free information technology resources. Join to browse thousands of videos and webinars on ITIL best practices, IT security strategy and more presented by leading CTOs, CIOs and other technology experts.

Building a Real-time Streaming ETL Framework Using ksqlDB and NoSQL

Presented by

About this talk

More from this channel