DataOps Unleashed is the first-ever event for the global DataOps community. We came together on Wednesday, March 17th, 2021 for DataOps Unleashed – a gathering of DataOps, CloudOps, AIOps, MLOps, and data-oriented DevOps professionals, including all data team members and their management, up to the CDO level. We shared the latest trends and best practices for running, managing, and monitoring data pipelines and data-intensive analytics workloads.
James Fielder, Senior Data Engineer at Cox Automotive, shows how a small data team manages DataOps for his organization’s global footprint, highlighting their use of Databricks on Microsoft Azure. Designing a data platform is no easy task, particularly when there are new technologies, techniques, and approaches appearing every week. At Cox Auto UK we have been on a journey from manually deployed Hadoop clusters to a full platform as a service setup using Azure Databricks. This journey hasn’t always been smooth however and we’ve learned some things along the way! In this talk, we examine how we have made design choices while evolving our platform, our decision to open source some of our work, and what our past, present, and future look like.