Name: Putting AI to Work on Apache Spark
Start: 2019-06-13T13:00:00Z
End: 2019-06-13T13:00:40.000Z
Location: BrightTALK
Rating: 0

At Unravel, we see an urgent need to help every business understand and optimize the performance of their applications, while managing data operations with greater insight, intelligence, and automation. 
 
For these businesses, Unravel is the AI-powered data operations company. We offer novel solutions that leverage AI, machine learning, and advanced analytics to help you fully operationalize the way you drive predictable performance in your modern data applications and pipelines.

Are you looking to optimize costs and resource usage for your Spark jobs on Amazon EMR? Then this is the webinar for you. Overallocating resources, such as memory, is a common fault when setting up Spark jobs. And for Spark jobs running on EMR, adding resources is a click away - but it’s an expensive click, so cost management is critical. 

Unravel Data is our AI-enabled observability platform for Spark jobs on Amazon EMR and other Big Data technologies. Unravel helps you right-size memory allocations, choose the right number of workers, and map your cluster needs to available instance types. Unravel’s troubleshooting capabilities mean you can fix problems the right way. You may never have to overallocate memory and other resources again! 

Join Mike Wong, Solutions Engineer at Unravel Data, as he offers tricks and tips to help you get the most from your EMR environment, while taking advantage of auto-scaling, different instance types, while reducing cost. You’ll learn:

-How Unravel cuts costs by an average of 30-40%. 
-How Unravel cuts time to solve problems (MTTR) by an average of 50%. 
-How to auto-tune and fix jobs to speed them up, eliminate errors, and meet SLAs. 
-How to screen jobs with Unravel before they go into production, ensuring a smooth launch and happy users. 
-How Unravel’s AI-powered recommendations, AutoActions, and TopX reports  save you time, money, and stress.

Unravel "Optimize" Webinar Series | Managing Costs for Spark on Amazon EMR

Are you looking to optimize costs and resource usage for your Spark jobs on Databricks? Then this is the webinar for you. Overallocating resources, such as memory, is a common fault when setting up Spark jobs. And for Spark jobs running on Databricks, adding resources is a click away - but it’s an expensive click, so cost management is critical. 

Unravel Data is our AI-enabled observability platform for Spark jobs on Databricks and other Big Data technologies. Unravel helps you right-size memory allocations, choose the right number of workers, and map your cluster needs to available instance types. Unravel’s troubleshooting capabilities mean you can fix problems the right way. You may never have to overallocate memory and other resources again! 

On November 4th at 10 AM PT, join Patrick Mawyer, Senior Solutions Engineer at Unravel Data, as he offers tricks and tips to help you get the most from your Databricks environment, while taking advantage of auto-scaling, interactive clusters vs. job clusters, and reducing cost. You’ll learn:

-How Unravel cuts costs by an average of 30-40%. 
-How Unravel cuts time to solve problems (MTTR) by an average of 50%. 
-How to auto-tune and fix jobs to speed them up, eliminate errors, and meet SLAs. 
-How to screen jobs with Unravel before they go into production, ensuring a smooth launch and happy users. 
-How Unravel’s AI-powered recommendations, AutoActions, and TopX reports save you time, money, and stress.

Unravel "Optimize" Webinar Series | Managing Costs for Spark on Databricks

Spark jobs require resources - and those resources? They can be pricey. If you're looking to speed up completion times, optimize costs, and reduce resource usage for your Spark jobs, this is the webinar for you. 

For Spark jobs running on-premises, optimizing resource usage is key. For Spark jobs running in the cloud, for example on Amazon EMR or Databricks, adding resources is a click away - but it’s an expensive click, so cost management is critical. 

On October 21st at 10 AM PT, join Chris Santiago, Director of Solutions Engineering at Unravel Data, as he offers tricks and tips to help you the most from your Spark environment, on-premises or in the cloud, while reducing resource requirements and cost. You’ll learn:
-How Unravel cuts resource requirements and costs for Spark jobs by an average of 30-40%.
-How Unravel cuts time to solve problems (MTTR) for Spark jobs by an average of 50%.
-How to auto-tune and fix jobs to speed them up, eliminate errors, and meet SLAs.
-How Unravel’s TopX reports, AutoActions, and AI-powered recommendations save you time, money, and stress.

Register now to learn how Unravel Data can help you meet SLAs for reliability and performance, cost-effectively.

Unravel "Optimize" Webinar Series | Managing Resource Use and Costs for Spark

Amazon EMR is growing in popularity, and is emerging as the leading platform for big data processing on AWS. EMR is the preferred platform for “lift and shift” migration of existing Hadoop and Spark workloads to the cloud, with minimal refactoring. You get better control, enhanced flexibility, and greater responsiveness. 
 
However, as the importance of EMR grows, so does the importance of reliability for EMR jobs - especially big data jobs such as Spark workloads. Information you need for troubleshooting is scattered across multiple, voluminous log files. The right log files can be hard to find, and even harder to understand. There are other tools, each providing part of the picture, leaving it to you to try to assemble the jigsaw puzzle yourself. 
 
Would your organization benefit from rapid troubleshooting for your EMR workloads? If you’re running significant workloads on EMR, then you may be looking for ways to find and fix problems faster and better - and to find new approaches that steadily reduce your problems over time. You will want to find equivalents to the approaches you used on-premises, plus cloud-specific ways to fix jobs, faster. 
 
Join Mike Wong, Solutions Engineer at Unravel Data, on Tuesday, October 5th at 10AM PT. See how Unravel can deliver:
- Enhanced observability through the use of additional sensors, placed in the JVM, plus intelligent curation and presentation of existing log and other data
- End-to-end monitoring, measurement, and troubleshooting of apps using Spark and related technologies.
 - AI-powered recommendations and automated actions to enable pre-emptive fixes of problems with your big data pipelines and applications.
- Detailed insights, plain language recommendations, and auto-tuning of apps to make the most of your Spark environment.
 
Don’t wait. Register today for this informative and actionable webinar.

Unravel "Optimize" Webinar Series | Troubleshooting Amazon EMR

The popularity of Databricks is rocketing skyward, and it is now the leading multi-cloud platform for Spark and analytics workloads, offering fully managed Spark clusters in the cloud. Databricks is fast and organizations generally refactor their applications when moving them to Databricks. The result is strong performance. However, as usage of Databricks grows, so does the importance of reliability for Databricks jobs - especially big data jobs such as Spark workloads. But information you need for troubleshooting is scattered across multiple, voluminous log files. The right log files can be hard to find, and even harder to understand. There are other tools, each providing part of the picture, leaving it to you to try to assemble the jigsaw puzzle yourself. 
 
Join Patrick Mawyer, Senior Solutions Engineer at Unravel Data, on September 16th @ 10:00am PT. See how Unravel can deliver:

- Enhanced observability through the use of additional sensors, placed in the JVM, plus intelligent curation and presentation of existing log and other data
- End-to-end monitoring, measurement, and troubleshooting of apps using Spark and related technologies.
- AI-powered recommendations and automated actions to enable pre-emptive fixes of problems with your Big Data pipelines and applications.
- Detailed insights; clear, AI-powered recommendations; and user-specified AutoActions to help you make the most of your Spark environment. 
 
Don’t wait. Register today for this informative and actionable webinar.

Unravel Optimize Webinar Series | Troubleshooting Databricks

Apache Spark is the leading technology for big data processing, on-premises and in the cloud. Spark powers advanced analytics, AI, machine learning, and more. Spark provides a unified infrastructure for all kinds of professionals to work together to achieve outstanding results. Technologies such as Cloudera’s offerings, Amazon EMR, and Databricks are largely used to run Spark jobs. However, as Spark’s importance grows, so does the importance of Spark reliability - and troubleshooting Spark problems is hard. Information you need for troubleshooting is scattered across multiple, voluminous log files. The right log files can be hard to find, and even harder to understand. There are other tools, each providing part of the picture, leaving it to you to try to assemble the jigsaw puzzle yourself. 
 
Would your organization benefit from rapid troubleshooting for your Spark workloads? If you’re running significant workloads on Spark, then you may be looking for ways to find and fix problems faster and better - and to find new approaches that steadily reduce your problems over time. See how Unravel can deliver:
 
- Enhanced observability through the use of additional sensors, placed in the JVM, plus intelligent curation and presentation of existing log and other data
- End-to-end monitoring, measurement, and troubleshooting of apps using Spark, Hadoop, Kafka, and related technologies.
- AI-powered recommendations and automated actions to enable pre-emptive fixes of problems with your big data pipelines and applications.
- Detailed insights, plain language recommendations, and auto-tuning of apps to make the most of your Spark environment.
 
Don’t wait. Register today for this informative and actionable webinar.

Unravel "Optimize" Webinar Series | Troubleshooting Apache Spark

Amazon EMR is growing in popularity, and is emerging as the leading platform for big data processing on AWS. EMR is the preferred platform for “lift and shift” migration of existing Hadoop and Spark workloads to the cloud, with minimal refactoring. You get better control, enhanced flexibility, and greater responsiveness. 
 
Would your organization benefit from rapid troubleshooting and performance optimization for your Amazon EMR workloads? If you’re running significant workloads on Amazon EMR then you may be looking for ways to get faster performance, and meet SLAs, without excessive resource use and cost. You will want to find the equivalents to the approaches you used on-premises, plus cloud-specific ways to get the job(s) done, faster. 
 
Join Chris Santiago, Director of Solutions Engineering at Unravel Data, on August 19th to see how Unravel can deliver:
 
- AI-powered recommendations and automated actions to enable intelligent optimization of your big data pipelines and applications.
- End-to-end monitoring, measurement, and troubleshooting of apps using Spark, Hadoop, Kafka, and related technologies.
- Detailed insights, plain language recommendations, and auto-tuning of apps to make the most of your Amazon EMR environment.
 
Don’t wait. Register today for this informative and actionable webinar.

Unravel Optimize Webinar Series | Accelerate Amazon EMR for Spark & More!

Databricks is the leading multi-cloud platform for Spark and analytics workloads with fully managed Spark clusters in the cloud. Would your organization benefit from rapid troubleshooting and performance optimization for your Databricks workloads? If you’re running significant workloads on Databricks, you’ve undoubtedly looked for ways to free up more DBUs to address more use cases and business challenges.  This can result in identifying ways to run jobs faster while maintaining or decreasing cloud spend.  Join Patrick Mawyer, Solutions Engineer at Unravel Data, on July 15th. See how Unravel can deliver:

- AI-powered recommendations and automated actions to enable intelligent optimization of your big data pipelines and applications.
- End-to-end monitoring, measurement, and troubleshooting of apps using Spark and related technologies.
- Detailed insights, plain language recommendations, and auto-tuning of apps to make the most of your Databricks environment.

Don’t wait. Register today for this informative and actionable webinar.

Unravel "Optimize" Webinar Series | Accelerate Performance for Databricks

Are you looking for a radically simple way to monitor, troubleshoot, and optimize Spark performance and reliability for Apache Spark? If so, then this webinar is for you. Join Mike Wong, Solutions Engineer at Unravel Data as he offers tricks and tips invaluable to getting the most from your Sparks environment – on-premises or in the cloud. Learn how Unravel Data’s built-in AI engine provides insights, recommendations, and auto-tuning for Spark applications and pipelines. You’ll benefit from:

- Automated root cause analysis (RCA) for failures and delays, with detailed explanations telling you what happened and why.
- Recommendations and tweaks to get your Spark jobs running at optimal levels.
- Auto-tuning and fixes to speed up jobs, get rid of errors, and guarantee SLAs are met.
- Using Unravel to screen jobs before putting them into production, so your job is optimized, efficient, and reliable from Day 1.
- Insights into misconfiguration, parallelism, partitioning, garbage collection, RDD caching, resource contention, container resource utilization, and more.

Register today for this informative and actionable webinar, the first of the series.

Unravel "Optimize" Webinar Series | Accelerate Performance for Spark

Amazon EMR is a go-to platform for those who want all the power of Hadoop and Spark in the cloud. However, cost and performance trade-offs can reduce the advantages of EMR over alternatives. Lack of visibility into the root cause of problems, right-sizing options, and cost allocation can add confusion and frustration for EMR users. Unravel Data gives you visibility into the minute-to-minute operations of your workloads on EMR. Get root cause analysis (RCA) of workload breakdowns and slowdowns; AI-powered recommendations; and proactive fixes for many problems.  With Unravel Data, you can meet and beat your SLAs, saving thousands - even millions - of dollars per year in the process. 

Chris Santiago is Director of Solutions Engineering at Unravel Data, and is a true master of cost and performance issues in the cloud. Join him as he demonstrates best practices for running Hadoop and Spark on Amazon EMR, including:

●	Tracking, managing, and allocating costs, minute by minute
●	Optimizing performance and costs while meeting SLAs
●	Using the free Unravel Data two-week trial to get a flying start on optimizing your Amazon EMR environment

There's a reason AWS partners provides many of its customers free access to Unravel Data for moving workloads to the cloud, and why many of those customers stay with Unravel Data for cloud cost and performance management. Learn more in this free webinar.

Effective Cost and Performance Management for Amazon EMR

In this webinar, Unravel CDO and VP Engineering Sandeep Uttamchandani describes the fourth and final step for any large, data-driven project: the Operationalize phase. You've found your data (Discover phase), readied it for processing (Prep phase), and built out your processing logic and machine learning model(s) (Build phase). Now you need to Operationalize all your work to data as a live project, in production. 

Sandeep Uttamchandani is a leader in the fields of data, AI, and machine learning. This webinar is the third talk from his new O'Reilly animal book, The Self-Service Data Roadmap. The book shows how to start, implement, and complete large data science projects, up to and including the creation of a complete, self-service data science platform for your organization.

Operationalize Your Insights - The Self-Service Data Roadmap, Session 4 of 4

Join Unravel’s CDO & VP of Engineering, Sandeep Uttamchandani and Matteo Pelati, Executive Director, Head of Technology - Data Platform at DBS Bank as they discuss:

How their tactical/strategic focus areas are evolving in these challenging times
Cloud big data migration strategy, do's and don'ts
Practical advice they can share for other leaders in the big data community
How Unravel has helped DBS optimize their big data

CDO Sessions: Transforming DataOps in Banking

Organizations are moving big data from on-premises to the cloud, using best-of-breed technologies like Databricks, Amazon EMR, Azure HDI, and Cloudera, to name a few. However, many cloud migrations fail. Why? And, how can you overcome the barriers and succeed? Join Chris Santiago, Director of Solution Engineering, as he describes the biggest pain points and how you can avoid them, and make your move to the cloud a success. He will cover:

The elements you must include in a successful cloud migration plan
How to find the right strategy for your cloud migration
Successful models for big data deployments in the cloud
How Unravel customers are making solid plans, meeting their goals, and saving time and money

Reasons Why Big Data Cloud Migrations Fail - and Ways to Succeed

Behind every successful insight (BI analytics or ML model) is a reliable data pipeline! These pipelines are planned, implemented, deployed, and monitored in an ongoing fashion referred to as the DataOps infinity loop (similar to CI/CD for traditional software). This talk covers battle scars in managing DataOps at scale, and how building checkpoints in the DataOps loop can reduce missed SLAs, cost outages, escalation from data users, and most importantly avoid data pipeline surprises!

DataOps Unleashed -  Building Checkpoints in Your DataOps

DataOps Unleashed is the first-ever event for the global DataOps community. We came together on Wednesday, March 17th, 2021 for DataOps Unleashed – a gathering of DataOps, CloudOps, AIOps, MLOps, and data-oriented DevOps professionals, including all data team members and their management, up to the CDO level. We shared the latest trends and best practices for running, managing, and monitoring data pipelines and data-intensive analytics workloads.

This talk demystifies the new data stack that thousands of companies are deploying to convert data into insights continuously and with high agility. This stack continues to evolve with the emergence of new data roles like analytics engineers and ML engineers as well as new data technologies like lake houses and data validation.
A new wave of operational challenges has emerged with this stack that, unless addressed from day one, will derail its success. Session presented by Shivnath Babu, Co-Founder and CTO, at Unravel Data discusses these DataOps challenges and the best practices to address them. The talk is accompanied by a brief demonstration.

DataOps Unleashed - DataOps for the New Data Stack

DataOps Unleashed is the first-ever event for the global DataOps community. We came together on Wednesday, March 17th, 2021 for DataOps Unleashed – a gathering of DataOps, CloudOps, AIOps, MLOps, and data-oriented DevOps professionals, including all data team members and their management, up to the CDO level. We shared the latest trends and best practices for running, managing, and monitoring data pipelines and data-intensive analytics workloads. Sessions included talks by DataOps professionals at leading organizations, detailing how they’re establishing data predictability, increasing reliability, and reducing costs. 

Hear from Jeff Lambert, Vice President of Data Solutions at Kroger/84.51˚ and Suresh Devarakonda, Lead Database Engineer at Kroger/84.51˚ as they give a 30,000 ft view into their management of Yarn and Impala. They share how they solved challenges associated with small files and used a centralized DataOps approach to troubleshoot issues with their big data pipelines. 84.51° also take from their executive dashboards and share key learnings in helping your business improve efficiency and reduce operational costs.

DataOps Unleashed - How 84.51 Slashed Costs & Improved DataOps Efficiency

DataOps Unleashed is the first-ever event for the global DataOps community. We came together on Wednesday, March 17th, 2021 for DataOps Unleashed – a gathering of DataOps, CloudOps, AIOps, MLOps, and data-oriented DevOps professionals, including all data team members and their management, up to the CDO level. We shared the latest trends and best practices for running, managing, and monitoring data pipelines and data-intensive analytics workloads. Sessions included talks by DataOps professionals at leading organizations, detailing how they’re establishing data predictability, increasing reliability, and reducing costs. 

Adobe has just embarked on a multi-year journey to transition their on-premise Hadoop data platform to the cloud. With thousands of users, petabytes of data, and millions of monthly job executions, transitioning to the cloud will be a tremendously challenging task. Join Kevin Davis as he shares the catalysts that started Adobe on this journey, the processes being employed to ensure key customer challenges are addressed in the new environment, and other tools and strategies that are helping along the way.

If your organization is contemplating a move to the cloud, this session will provide key insights into the early stages of Adobe’s transition that will help you plan your initiative.

DataOps Unleashed -  A Journey to the Cloud for Adobe’s Corporate Data Platform

DataOps Unleashed is the first-ever event for the global DataOps community. We came together on Wednesday, March 17th, 2021 for DataOps Unleashed – a gathering of DataOps, CloudOps, AIOps, MLOps, and data-oriented DevOps professionals, including all data team members and their management, up to the CDO level. We shared the latest trends and best practices for running, managing, and monitoring data pipelines and data-intensive analytics workloads. 

 James Fielder, Senior Data Engineer at Cox Automotive, shows how a small data team manages DataOps for his organization’s global footprint, highlighting their use of Databricks on Microsoft Azure. Designing a data platform is no easy task, particularly when there are new technologies, techniques, and approaches appearing every week. At Cox Auto UK we have been on a journey from manually deployed Hadoop clusters to a full platform as a service setup using Azure Databricks. This journey hasn’t always been smooth however and we’ve learned some things along the way! In this talk, we examine how we have made design choices while evolving our platform, our decision to open source some of our work, and what our past, present, and future look like.

DataOps Unleashed - The Evolution of a Data Platform

Apache Spark simplifies AI, but why not use AI to simplify Spark performance and operations management? An AI-driven approach can drastically reduce the time Spark application developers and operations teams spend troubleshooting problems.

This talk will discuss algorithms that run real-time streaming pipelines as well as build ML models in batch to enable Spark users to automatically solve problems like: (i) fixing a failed Spark application, (ii) auto tuning SLA-bound Spark streaming pipelines, (iii) identifying the best broadcast joins and caching for SparkSQL queries and tables, (iv) picking cost-effective machine types and container sizes to run Spark workloads on the AWS, Azure, and Google cloud; and more.

Putting AI to Work on Apache Spark

SPARK

spark ai

unravel data

Big Data

Data Analysis

data operations

big data operations

Cloud computing has exploded over the past few years, delivering a previously unimagined level of workplace mobility and flexibility. The cloud computing community on BrightTALK is made up of thousands of engaged professionals learning from the latest cloud computing research and resources. Join the community to expand your cloud computing knowledge and have your questions answered in live sessions with industry experts and vendor representatives.

Cloud Computing

The application development community features top thought leadership focusing on optimal practices in software development, SDLC methodology, mobile app development and application development platforms and tools. Join top software engineers and coders as they cover emerging trends in everything from enterprise app development to developing for mobile platforms such as Android and iOS.

Application Development

Practicing business intelligence allows your company to transform raw data into sets of insights for targeted business growth. The business intelligence and analytics community on BrightTALK is made up of thousands of data scientists, database administrators, business analysts and other data professionals. Find relevant webinars and videos on business analytics, business intelligence, data analysis and more presented by recognized thought leaders. Join the conversation by participating in live webinars and round table discussions.

Business Intelligence and Analytics

Welcome to the big data and data management community on BrightTALK. Join thousands of data quality engineers, data scientists, database administrators and other professionals to find more information about the hottest topics affecting your data. Subscribe now to learn about efficiently storing, optimizing a complex infrastructure, developing governing policies, ensuring data quality and analyzing data to make better informed decisions. Join the conversation by watching live and on-demand webinars and take the opportunity to interact with top experts and thought leaders in the field.

Big Data and Data Management

As an IT professional, many of the problems you face are multifaceted, complex and don’t lend themselves to simple solutions. The information technology community features useful and free information technology resources. Join to browse thousands of videos and webinars on ITIL best practices, IT security strategy and more presented by leading CTOs, CIOs and other technology experts.

Putting AI to Work on Apache Spark

Presented by

About this talk

More from this channel