How to Overcome the Five Most Common Spark Challenges

Running Spark on Kubernetes is growing in popularity. Reasons for the growth are improved isolation, better resource sharing, and the ability to leverage homogeneous and cloud-native infrastructure for the entire stack.

But running Spark on Kubernetes in a stable, performant, cost-efficient, and secure manner also presents specific challenges. In this webinar, Alex Pierce will talk about best practices for optimizing Spark performance on Kubernetes. Topics include:

– Making Spark-on-k8s reliable at scale
– Core concepts and setup of Spark on Kubernetes
– Configuration tips for performance and efficient resource sharing
– Spark-app level dynamic allocation and cluster level autoscaling
– Monitoring and security best practices

Complex applications running on Kubernetes scale super fast, but this can create visibility gaps that can make detecting and troubleshooting Kubernetes issues as difficult as finding a needle in a haystack. Although Docker and Kubernetes are now becoming standard components when building and orchestrating applications, you’re still responsible for managing the performance of applications built atop this new stack.

With many companies prioritizing containers for more applications and more uses, an increasing area of concern for everyone in IT is finding a way to monitor, manage, and optimize performance across these sprawling environments. Join this webinar to learn:

– A brief history of current trends in computing, cloud, containerization, and Kubernetes
– Challenges: virtualization, distributed applications, and multi-cloud
– How to meet the demands of new microservices apps while maintaining legacy apps
– How to deploy, manage, monitor, and simplify: big data analytics monitoring, platform monitoring, and dynamic optimization
– Ways to reduce the complexity of monitoring and managing Kubernetes with automated full-stack observability
– What Kubernetes performance management success looks like

Apache Hive is a powerful tool frequently used to analyze data while handling ad-hoc queries and regular ETL workloads. Despite being one of the more mature solutions in the Hadoop ecosystem, developers, data scientists and IT operators are still unable to avoid common inefficiencies when running Hive at scale. Inefficient queries can mean missed SLAs, negative impact on other users, and slow database resources. Poorly tuned platforms or poorly sized queues can cause even efficient queries to suffer.

This webinar discusses proven approaches to Hive query tuning that improve query speed and reduce cost. Learn how to understand the detailed performance characteristics of query workloads and the infrastructure-wide issues that impact these workloads.

Pepperdata Field Engineer, Kirk Lewis will discuss:

- Finding problem queries - Pinpointing delayed queries, expensive queries, and queries that waste CPU and memory
- Improving query utilization and performance with database and infrastructure metrics
- Ensuring your infrastructure is not adversely impacting query performance

Big data with cloud computing is a powerful combination that can transform your organization, process and analyze your big data faster, and improve your products and business with actionable insights. Bringing your big data cluster to the cloud presents huge opportunities, but there are some challenges that need to be overcome. Is your organization really ready for the complexity of managing big data analytics in the cloud?

Most big data enterprises have either adopted cloud computing to improve IT operations and develop better software, faster, or they have an initiative to get there. Preparing for a successful move to the cloud is the difference between realizing the ROI the cloud promises, managing impatient stakeholders and SLAs, and possibly moving back to an on-premises solution.

Creating an accurate cloud footprint requires good planning, a deep understanding of resource utilization, and granular data. In this webinar, we’ll discuss how to prepare and ensure that your organization has a solid plan to manage big data analytics in the cloud.

Topics include:

– Primary characteristics of big data and putting your data in the cloud
– The challenges of managing big data performance in the cloud
– FinOps (chargeback, analyzing wasted spend)
– Planning for day 2
– Achieving cloud performance
– Observability, continuous tuning, and managed autoscaling