Strategies for exploring and optimizing multiple metrics

Logo
Presented by

Harvey Cheng, SigOpt

About this talk

During model development we often study metrics with the name “training loss,” “validation accuracy” or “F1 score”.  Ultimately, the reason we use these metrics when building models is because they are available, computable, theoretically supported and, hopefully, correlated to the success of the model.  The actual success of the model, in production, will be measured through a number of business metrics such as “engagement,” “consistency” or “profit”. In recognition of this dilemma, we have seen the community embrace multiple metrics of success during the deployment of models.  In this talk, we discuss incorporating more metrics into the model development process -- this can allow us to better guide our model development towards success in production (not just during training).  At first, these can be incorporated as simply as passively recording metrics which point to business criteria (such as model complexity) and then reviewing them immediately before deployment.  Eventually, we can embed these metrics in the development process to better align the resulting models with business success.  We discuss our strategies for managing (and optimizing) multiple metrics with the goal of empowering users to make better decisions in these complicated circumstances.

Related topics:

More from this channel

Upcoming talks (0)
On-demand talks (42)
Subscribers (4292)
With Intel® Data Center Manager (Intel® DCM), IT and facility managers have the tools to improve manageability, increase availability, and reduce costs across key parameters, such as energy usage, monitoring, security, automation, and cloud. In an era of rapid data center evolution, Intel DCM helps you control and optimize your infrastructure, business models, and operational expenditures.