Name: Monitoring Models at Scale with Domino
Start: 2019-12-11T18:00:00Z
End: 2019-12-11T18:00:32.000Z
Location: BrightTALK
Rating: 3.3

Today, the best-run companies run their business on models, and those that don’t face existential threat. Welcome to "The Model Driven Business" - a channel where we will share use cases and best practices for organizations striving to make data science an organizational capability that drives business impact.

Building data science models requires specialized hardware which can be a pain to provision and procure. IT teams often struggle to wrap their arms around the rapidly evolving needs of Data Science teams.


Dell Technologies is recognized globally for its expertise in computing hardware and infrastructure. The company’s portfolio of Ready Solutions for AI bridges the gap between IT and Data Science by bundling validated designs and software reference architectures with its hardware. A new addition to this portfolio is Dell’s validated design for the Domino data science platform.

The result? A refreshingly simple, building-block-based approach to help companies get their data science teams -- and the technologies they need -- up and running faster, with an easy path to scale.

Dell Chief Technology Strategist Natalie Kortum and Domino Chief Data Scientist Josh Poduska will delve into the challenges this solution solves, how it’s bringing IT and Data Science Leaders together, what’s in the design under the hood, and the benefits it’s delivering.

(Dell Technologies and Domino Data Lab are joint hosts for this webinar. By registering for the webinar, you opt in to share your contact information with both Dell Technologies and Domino Data Lab.)

How Dell & Domino Are Helping IT Govern the AI/ML Ecosystem

Are you a data scientist interested in training a classifier but do not have enough data? Are you interested in generating data to potentially improve the accuracy of your models? Have you heard of  GANs yet are hesitant to dive in because of potential training instability or failure to converge issues? If so, attend this upcoming webinar, "Generative Adversarial Networks (GAN): A Distilled Hands-On Tutorial", by Nikolay Manchev, Domino Principal Data Scientist for EMEA.

In this hands-on code webinar, Manchev covers:

-the GAN framework
-how to implement a basic GAN model
-how adversarial networks are used to generate training samples
-training difficulties, including mode collapse
-recent research to improve upon GANs' training, including Wasserstein GAN (WGAN)

Generative Adversarial Networks (GAN): A Distilled Hands-On Tutorial

Models are at the heart of data science. Data scientists, machine learning (ML) researchers, and business stakeholders have a high-stakes investment in the predictive accuracy of models. Data scientists and researchers ascertain predictive accuracy using different techniques, methodologies, and settings, including hyperparameters. Regularized hyperparameters provide data scientists and researchers with the ability to control the flexibility of the model. This control prevents overfitting and reduction in predictive accuracy on new test data. When hyperparameters are also used for optimization and tuned for a specific dataset, they also impact predictive accuracy. 

In this hands-on code webinar, "Towards Predictive Accuracy: Tuning Hyperparameters and Pipelines", data scientist and book author, Dr. Mark Fenner covers:

- building a pipeline for automating ML workflow.
- core concepts, including cross-validation.
- how to compare, select, and tune hyperparameters.

*this webinar is pre-recorded, but the content will be exclusively released to all registrants on December 4, 2019 at 10 am PT

Towards Predictive Accuracy: Tuning Hyperparameters and Pipelines

Many data scientists and their organizations may have hundreds of models running in production, interacting with the real world, and are not keeping track of how their models are performing on live data. Bias and variance can creep into models over time, and we should know when that happens. The world changes, often slowly, and most models perform worse as time goes on. Ensuring everything is working well is a huge undertaking, and unfortunately, many organizations are simply ignoring the problem. Donald Miner, drawing upon his prior experience as a data scientist, engineer, and CTO, details the tracking of machine learning models in production to ensure model reliability, consistency, and performance into the future. 

In this webinar Miner covers:
 
-why you should invest time in monitoring your machine learning models.  
-real-world anecdotes about some of the dangers of not paying attention to how a model’s performance can change over time. 
-metrics you should be gathering for each model and what they tell you with a list of “vitals,” what value they provide, and how to measure them. 
-vitals that include classification label distribution over time, distribution of regression results, measurement of bias, measurement of variance, change in output from previous models, and changes in accuracy over time. 
-implementation strategies to keep watch on model drift over time.

Machine Learning Vital Signs: Metrics and Monitoring Models in Production

Although there are an increasing number of commercial AutoML products, the open-source ecosystem has been innovating here as well. In the early days of the AutoML movement, the focus was on those looking to leverage the power of ML models without a background in data science - citizen data scientists. Today, however, AutoML tools have a lot to offer experts too. 

In this webinar, we will dive into popular open source AutoML tools such as auto-sklearn, TPOT, MLBox, and AutoKeras. We will also walk through hands-on examples of how to install and use these tools, and highlight special features of each while providing Jupyter notebooks so you can start using these technologies in your work right away. Those who wish to follow along interactively during the webinar and download the notebooks and slides can do so by signing into Domino’s trial version at https://dominodatalab.com/try. Instructions on accessing the materials in Domino trial will be given during the webinar.

Turbo-Charging Data Science with AutoML

Data science is a team sport. Sharing your work is critical for getting early feedback from stakeholders and, ultimately, for having your work impact the business. There’s no better way to share your work than with a web app. Creating interactive visualizations lets data scientists build engaging tools for end consumers or prototype something before handing it off to an engineer.

In this webinar we’ll review the pros and cons of the top data science web app frameworks (Shiny, Dash, Flask, etc.), discuss best practices for building and debugging web apps, and provide a step-by-step walk through on how to quickly deploy web apps in a secure environment while maintaining a system of record for your work.

This will be a hands-on, technical webinar, aimed at helping data scientists do their job better. We look forward to having you join us.

Best Practices for Getting Data Science Web Apps in Production

Navigating Predictive and Interpretable Models

Model ethics, interpretability, and trust will be seminal issues in data science in the coming decade. This technical webinar discusses traditional and modern approaches for interpreting black box models. Additionally, we will review cutting edge research coming out of UCSF, CMU, and industry. This new research reveals holes in traditional approaches like SHAP and LIME when applied to some deep net architectures and introduces a new approach to xML/xAI where interpretability is a hyperparameter in the model building phase rather than a post-modeling exercise. We will provide step-by-step guides that practitioners can use in their work to navigate this interesting space. 

We will review code examples of interpretability techniques. You can follow along with the presentation by running your own notebook hosted in Domino's trial environment. Create a free trial account at: http://dominodatalab.com/try?utm_source=brighttalk

A Data Science Playbook for Explainable ML/AI

Over the last two decades business and IT leaders chased the “big data” dream. They were sold on a data-driven future, complete with competitive advantages automagically gained collecting and storing “big data”. In the end, these enterprises needed more than analyst dashboards. They need to make data science an organizational capability to become a model-driven organization. 

You can have all the data you want, do all the machine learning you want, but if you aren’t running your business on models, you’ll soon be left behind. In this webinar, we will demystify the model-driven business:

-Urgency: why companies running on models are automating decisions and winning
-Experts: how deep expertise is required to make this change
-Purpose Built: what are the platforms incorporating data science as a core capability

Finally, we will highlight the major Domino platform investments that enable a model-driven business: 

-Open, Flexible Infrastructure: agile scalable compute with accelerated experiments
-Collaborative R&D: an experimental, iterative, and exploratory environment for teams
-Governance of Data Science: explore new ways to review, quality control, and monitor

Data-driven to Model-driven

While working with Fortune 500 companies who have sophisticated data science organizations, we realized that existing tools don't meet collaboration, visibility, and governance needs. 

Domino 3.5 answers the needs of data science leaders. It allows data science managers to define their own data science project life cycle, and easily track and manage projects with a holistic understanding of the latest developments. It also surfaces projects that need immediate attention in real time by showing the projects that are blocked. Watch this short webinar to learn how Domino 3.5 equips data science leaders and IT admins with:

-Project Portfolio Dashboard
-Project Stage Configuration 
-License Usage Reporting

Providing unprecedented transparency into the health of data science teams

Recorded at Rev 2 | May 23-24, 2019 | New York

Day 2 - Case Studies - Climate Corporation, AWS

The Climate Corporation provides a platform for farmers around the world to use best-in-class analytic capabilities to digitize their operations and optimize their profits. Come learn about Climate’s journey from a scrappy startup to a mature company and hear how we grew our data science capabilities 10x, from supporting 20 data scientists to supporting 200 data scientists, over 2 years.

As a new startup, data scientists were working on customized servers with a complex set of unstandardized libraries. Time and effort were lost to maintenance and overhead, with researchers spending 50% of their time maintaining and customizing research environments. Additionally, sharing work between groups was a significant challenge and versioning of models/data was done manually, with a high risk for error. We identified a need to standardize our environments to minimize time spent configuring research environments as well as simplifying collaboration across data scientists on an ongoing basis.

To meet this need, we developed a process and infrastructure whereby hardware and software are tailored to a researcher’s needs based on their domain, and we built out automation to enable this process by default. By automating the configuration of Yarn, Spark, Docker, AWS and Domino we drove standardized infrastructure for research and discovery within Climate. This enabled the data science team to deliver models to production faster and at less than half the previous cost, enabling farmers around the world to increase crop yields and grow more food for all of us!

How Climate Grew Its Data Science Capabilities 10x in 2 Years

Recorded at Rev 2 | May 23-24, 2019 | New York

Day 2 - Case Studies - Walmart

Digital Transformation is not necessarily about digital technology, but about the fact that technology, which is digital, allows people/companies to solve their traditional problems - and they prefer this digital solution to the old solution. Digital transformation is more about the integration of digital technology into all areas of a business, fundamentally changing how you operate and deliver value to customers. Learn about how Walmart is managing digital transformation as a cultural change that requires organizations at the behemoth size to startup-scale to continually challenge the status quo, experiment, and get comfortable with failure. The evolution of the role of data scientist will be explored at length along with what the required skill sets of the future will entail.

Data Science as the Enabler of Digital Transformation

Recorded at Rev 2 | May 23-24, 2019 | New York

Day 2 - Case Studies - Trupanion

Trupanion is one of the fastest growing players in the pet medical insurance space. At Trupanion we aim to revolutionize the way pet owners are able to approach the costs of veterinary care by completely restructuring the claim reimbursement model. Leveraging NLP and machine learning we are able to auto-adjudicate claims in a matter of seconds compared to the industry standard of one week. Eliminating the laborious claims process not only allows our customers and partnering veterinary clinics to focus on providing the best care for their pets but also allows Trupanion to operate at scale as our business grows. In this talk, we will discuss how Domino has empowered the data science and engineering teams at Trupanion to create a multi-layer framework for auto-adjudicating veterinary claims in real time.

Auto-Adjudicating Veterinary Claims at Trupanion

Recorded at Rev 2 | May 23-24, 2019 | New York

Day 2 - Case Studies - The New York Times

The Data Science group at The New York Times develops and deploys machine learning solutions to newsroom and business problems. Re-framing real-world questions as machine learning tasks require not only adapting and extending models and algorithms to new or special cases but also sufficient breadth to know the right method for the right challenge. I’ll first outline how unsupervised, supervised, and reinforcement learning methods are increasingly used in human applications for description, prediction, and prescription, respectively. I’ll then focus on the ‘prescriptive’ cases, showing how methods from the reinforcement learning and causal inference literature can be of direct impact in engineering, business, and decision-making more generally.

Data Science at The New York Times

Recorded at Rev 2 | May 23-24, 2019 | New York

Day 2 - Leadership - Microsoft

With his background spanning work in government, the commercial sector, and academia alike, Marck brings a unique and comprehensive perspective to this talk, where he’ll delve into the common themes and lessons learned across all kinds of organizations aiming to improve outcomes through data science.

Topics that will be addressed include:

- The importance and key considerations of building a data science community comprised of diverse backgrounds and skill sets
- How to recognize and communicate the impact of models and data science, and how to articulate that value to business stakeholders
- How to tackle hard problems in human ways by upskilling domain experts
- Cautions and mitigation strategies around giving non-technical users access to data science technologies
- Analogies between the federal and commercial sectors, and how they can learn from each other

Best Practices for Advancing Human Progress with Models

Recorded at Rev 2 | May 23-24, 2019 | New York

Day 2 - Leadership - BNP Paribas Cardif

Issuing an internal model risk governance is often not prioritized in the early stages of the transformation journey towards a model-driven company. Indeed, designing and validating such governances can be seen as a complex and heavy task, cannibalizing resources. In addition, some stakeholders of the transformation see governances as innovation killers. In this talk, Sebastien will explain all the benefits he experimented after releasing an internal analytics model risk governance at BNP Paribas Cardif, how it boosted the transformation pace, and why he thinks that is an important deliverable to be prioritized as soon as possible on the transformation path.

Catalyzing Your Transformation, Thanks to Early Model Risk Governance

Recorded at Rev 2 | May 23-24, 2019 | New York

Day 2 - Leadership - Workday

Companies that understand how to apply AI will scale and win their respective markets over the next decade. That said, delivering on this promise and managing machine learning projects is much harder than most people anticipate. Many organizations hire teams of PhDs and data scientists, then fail to ship products that move business metrics. The root cause is often a lack of product strategy for AI, or the failure to adapt their product development processes to the needs of machine learning systems. This talk will cover some of the common ways machine learning fails in practice, the tactical responsibilities of AI product managers, and how to approach product strategy for AI.

Peter Skomoroch, former Head of Data Products at Workday and LinkedIn, will describe how you can navigate these challenges to ship metric moving AI products that matter to your business.

Peter will provide practical advice on:

- The role of an AI Product Manager
- How to evaluate and prioritize your AI projects
- The ways AI product management differs from traditional product management
- Bridging the worlds of design and machine learning
- Making trade offs between data quality and other business metrics

Product Management for AI

Recorded at Rev 2 | May 23-24, 2019 | New York

Day 2 - Practitioner - Bloomberg

Quantitative finance is a rich field in finance where advanced mathematical and statistical techniques are employed by both the sell-side and buy-side institutions. Techniques like time-series analysis, stochastic calculus, multivariate statistics, and numerical optimization are often used by “quants” for modeling asset prices, portfolio construction/optimization, building automated trading strategies. My talk will focus on how machine learning and deep learning techniques are being used in this field.

In the first part of the talk, we will look at use cases involving both structured and unstructured data sets in finance, where machine learning techniques can be applied. Then we will pick a few case studies and examine in detail how machine learning models can be applied for predictive analytics.

We’ll look at interactive plots running in Jupyter notebooks. The main focus of the talk will be on reproducible research and model interpretability.

Applied Machine Learning in Finance

As more models go to production, managing and monitoring them becomes more onerous. Without proactive monitoring of production models, organizations are exposed to the risk of poor predictions on evolving data affecting their business outcomes.

Domino enables you to standardize model monitoring across all your data science teams. Key stakeholders get continuous visibility, and teams can initiate proactive actions before your business is negatively impacted due to model decay. It frees your data scientists to work on building newer models, and facilitates your IT/Ops teams to take charge of monitoring in-production models.

Join this webinar to learn how Domino can help you monitor your models with: 

-Drift Checks: Detecting changes in pattern of real-world data your models are seeing in production.
-Quality Checks: Tracking how model accuracy and other quality metrics are changing over time.
-Health Alerts: Getting alerted when health checks fail so that resolution workflows can be triggered.

Monitoring Models at Scale with Domino

domino data lab

Data Science

Analytics

Machine Learning

Deep Learning

Artificial Intelligence

Predictive Analytics

Practicing business intelligence allows your company to transform raw data into sets of insights for targeted business growth. The business intelligence and analytics community on BrightTALK is made up of thousands of data scientists, database administrators, business analysts and other data professionals. Find relevant webinars and videos on business analytics, business intelligence, data analysis and more presented by recognized thought leaders. Join the conversation by participating in live webinars and round table discussions.

Business Intelligence and Analytics

Welcome to the big data and data management community on BrightTALK. Join thousands of data quality engineers, data scientists, database administrators and other professionals to find more information about the hottest topics affecting your data. Subscribe now to learn about efficiently storing, optimizing a complex infrastructure, developing governing policies, ensuring data quality and analyzing data to make better informed decisions. Join the conversation by watching live and on-demand webinars and take the opportunity to interact with top experts and thought leaders in the field.

Big Data and Data Management

As an IT professional, many of the problems you face are multifaceted, complex and don’t lend themselves to simple solutions. The information technology community features useful and free information technology resources. Join to browse thousands of videos and webinars on ITIL best practices, IT security strategy and more presented by leading CTOs, CIOs and other technology experts.

Monitoring Models at Scale with Domino

Presented by

About this talk

More from this channel