Strategies for exploring and optimizing multiple metrics

During model development we often study metrics with the name “training loss,” “validation accuracy” or “F1 score”.  Ultimately, the reason we use these metrics when building models is because they are available, computable, theoretically supported and, hopefully, correlated to the success of the model.  The actual success of the model, in production, will be measured through a number of business metrics such as “engagement,” “consistency” or “profit”. In recognition of this dilemma, we have seen the community embrace multiple metrics of success during the deployment of models.  In this talk, we discuss incorporating more metrics into the model development process -- this can allow us to better guide our model development towards success in production (not just during training).  At first, these can be incorporated as simply as passively recording metrics which point to business criteria (such as model complexity) and then reviewing them immediately before deployment.  Eventually, we can embed these metrics in the development process to better align the resulting models with business success.  We discuss our strategies for managing (and optimizing) multiple metrics with the goal of empowering users to make better decisions in these complicated circumstances.