Data Science, Past & Future

Presented by

Paco Nathan - Managing Partner, Derwen, Inc.

About this talk

Recorded at Rev 2 | May 23-24, 2019 | New York Day 2 - General Session - Derwen, Inc. This talk explores big themes — the challenges in industry, the advances in technology — which brought us to this point. Origins of our field followed a simple formula at nearly every step along the way. We can use that formula as a lens to understand changes that are emerging. Looking through six decades since Tukey first described “data analytics”, challenges and advances have often upset the status quo. Hardware capabilities evolved in dramatic leaps. Software layers provided new kinds of control systems and conceptual abstractions in response. These manifested as surges in data rates and compute resources. Then industry teams applied increasingly advanced mathematics to solve for novel business cases. That’s the formula. For example, as spinny disks gave way to SSDs and commodity CPUs became multicore, Hadoop use cases gave way to Spark which fit the hardware better. Cluster computing workloads that had been ETL or clickstream, gave way to more complex math used for recommender systems, anti-fraud, anti-churn, and other advanced predictive analytics. We’re at a point now where more of the predictive analytics are moving to Python or R in-memory processing (Arrow), while more advanced workloads such as deep learning take over the clusters. On the horizon, even more complex use cases such as knowledge graph work will be consuming what new hardware provides. Let’s examine this “lens” into how our field evolves. Also keep in mind that growing security threats and increasingly complex regulatory requirements drive from the top, placing even more premium on novel business cases. We’ll look through the trends in history, leading up to now, consider examples from this conference, then look at what’s on the horizon.

Related topics:

More from this channel

Upcoming talks (0)
On-demand talks (50)
Subscribers (3748)
Today, the best-run companies run their business on models, and those that don’t face existential threat. Welcome to "The Model Driven Business" - a channel where we will share use cases and best practices for organizations striving to make data science an organizational capability that drives business impact.