Apache® Spark™ MLlib 2.x: Migrating ML Workloads to DataFrames

See how Apple, Finra, FIS, Overstock.com, Hewlett-Packard, Shell, Hotels.com and many others overcame these challenges to connect data science and data engineering using Databricks, the company founded by the original creators of Apache Spark™. The results? Faster performance, scaled data processes, simplified infrastructure, streamlined workflows, and greater collaboration.

Netflix is bringing engaging, culturally diverse stories to people all across the globe. With each original movie or TV show, we learn more about what our members want – and rest assured, they want an increasingly broad, deep, global, dynamic library!

Whether it’s planning how to satisfy these global tastes with the right content portfolio or personalizing recommendations to each member, Netflix relies heavily on data science techniques. This talk will highlight some of our core data science strategies and applications involving predictive models & algorithms, experimentation, and analytics.

In the last year Hotels.com has begun it’s journey to becoming an algorithmic business. Matt will talk about their experiences of exponential growth in Data Science Algorithms whilst at the same time the team have migrated to using Spark as their core underlying architecture from SAS / SQL, migrated to the cloud from on-premise are transforming the capability of the data science function. He will also highlight the key enablers that have made this successful including CEO support, the internal concepts of organic intelligence and how Databricks has helped make this happen. He will also highlight the pitfalls on the journey.

Comcast is the largest cable and internet provider in the US, reaching more than 30 million customers. Over the last couple years, Comcast has transformed the customer experience using machine learning. For example, Comcast uses machine learning to power the X1 voice remote, which was used over 8B times in 2018 by our customers to find something they love to watch, get the latest sports statistics, control their home, or check their bill and troubleshoot their service using natural language.

What all these different applications have in common is that to create and operate the machine learning models powering these applications we need to ingest many TBs of data on daily basis in an efficient and resilient manner, and need a machine learning platform that allows for fast exploration of new ideas while at the same time automatic deployment of the resulting machine learning models into a production environment that can handle Comcast scale.

In this talk we describe our data and machine learning infrastructure built on Databricks Unified Analytics Platform including how Databricks Delta is used for the ingest and initial processing of the raw telemetry from our video and voice applications and devices. We then explain how this data can be used by both the product organizations to gain deeper insights into how our products are being used, as well as by our research and engineering teams to train and fuel the machine learning models at the heart of of these products. This keynote will also include an end-to-end demonstration of our machine learning platform that is centered around Databricks and MLFlow and how it integrates with other open source machine learning frameworks such as Tensorflow, PyTorch, Sklearn, H20 and Kubeflow to name a few.

In this webinar, we will cover some of the latest innovations brought into the Databricks Unified Analytics Platform for Machine Learning, including:

Get started quickly using the Databricks Runtime 5.0 for Machine Learning, that provides a pre-configured Databricks clusters including the most popular ML frameworks and libraries, Conda support, performance optimizations, and more.

Track, tune, and manage models, from experimentation to production, with MLflow, an open-source framework for the end-to-end Machine Learning lifecycle that allows data scientists to track experiments, share and reuse projects, and deploy models quickly, locally or in the cloud.

Scale up deep learning training workloads from a single machine to large clusters for the most demanding applications using the new HorovodRunner.