Capacity Planning for Big Data Hadoop Environments
Learn about Hadoop capacity planning at the cluster, queue, and application levels with Pepperdata
As the data analytics field matures, the amount of data generated is growing rapidly and so is its use by enterprise organizations. This increase in data improves data analytics and the result is a continuous circle of data and information generation. To manage these new volumes of data, IT organizations and DevOps teams must understand resource usage and right-size their Hadoop clusters to balance the OPEX and CAPEX.
This presentation discusses capacity planning for big data Hadoop environments. Pepperdata field engineer Kirk Lewis explores big data Hadoop capacity planning at the cluster level, the queue level, and the application level via the Pepperdata big data performance management UI.
RecordedJul 29 201919 mins
Your place is confirmed, we'll send you email reminders
Hitachi Vantara Sr Director of Product Marketing, Chuck Yarbrough and Pepperdata Field Engineer, Alex Pierce
In 2019, the CFO of a large global bank realized a problem: Their data continued to grow, costs for their Hadoop cluster rapidly escalated, and these costs started eating into their annual IT budget. Moving off of Hadoop, or “lift-and-shift” was out of the question. They needed a way to cap their cost and growth without impacting their ability to remain market competitive.
Learn how you can expose, simplify, and solve the problems created by large big data clusters. Save time and money, and ensure compliance.
Cloud providers make managing big data look easy, but autoscaling is wasteful and inefficient. Qubole takes advantage of the separation between compute and storage to help their customers reduce their spend in the cloud. However, Qubole customers can use cloud computing resources even more efficiently, only pay for what they use, and avoid over-provisioning servers and virtual machines with managed autoscaling.
In this webinar, presenter Alex Pierce will use customer examples to demonstrate how Qubole customers can automatically improve infrastructure utilization and gain more throughput with Pepperdata big data performance management solutions.
Ahmed Kamran Imadi, Fortune 100 Finserv, Mark Kidwell, Autodesk, Satish Nekkalapudi, Magnite, Joel Stewart, Pepperdata
Have you changed the way you use big data in your business? Understanding the rapid pace of data usage across your organization and planning for the future of big data is a key skill. Sometimes we all need a little insight.
During this webinar, hear from industry leaders, Ahmed Kamran Imadi, Big Data Solutions Engineering at Fortune 100 Financial Institution, Mark Kidwell, Chief Data Architect at Autodesk, Satish Nekkalapudi, Sr. Manager at Magnite, and VP of Customer Success Joel Stewart at Pepperdata about what role big data is playing in their business today and how they are adapting their IT ops and development teams to keep pace with change.
What will be big data’s role in the future for business and how will IT adapt and grow?
How will the growth in big data affect IT ops and developer processes today?
Will this change skill sets for these roles?
What skills will be needed in IT as the need for big data increases?
The ability to scale the number of nodes in your cluster up and down on the fly is among the major features that make cloud deployments attractive. Estimating the right number of cluster nodes for a workload is difficult; user-initiated cluster scaling requires manual intervention, and mistakes are often costly and disruptive.
Autoscaling enables applications to perform their best when demand changes. But the definition of performance varies, depending on the app. Some are CPU-bound, others memory-bound. Some are “spiky” in nature, while others are constant and predictable. Autoscaling automatically addresses these variables to ensure optimal application performance. Amazon EMR, Azure HDInsight, and Google Cloud Dataproc all provide autoscaling for big data and Hadoop, but each takes a different approach.
Pepperdata field engineer, Kirk Lewis will discuss the operational challenges associated with maintaining optimal big data performance, what milestones to set, and offer recommendations on how to create a successful cloud migration framework. Topics include:
– Types of scaling
– What does autoscaling do well? When should you use it?
– Does traditional autoscaling limit your success?
– What is optimized cloud autoscaling?
This discussion explores the results of analyzing thousands of Spark jobs on many multi-tenant production clusters. We will discuss common issues we have seen, the symptoms of those issues, and how you can address and overcome them without thinking too hard.
Pepperdata big data performance management gathers trillions of performance data points on hundreds of production clusters running Spark, covering a variety of industries, applications, and workload types.
Based on analyzing the behavior and performance of thousands of Spark applications and use case data from the Pepperdata Big Data Performance report, Heidi and Alex will discuss key performance insights. Topics include best and worst practices, gotchas, machine learning, and tuning recommendations.
Kafka performance relies on implementing continuous intelligence and real-time analytics. It is important to be able to ingest, check the data, and make timely business decisions.
Stream processing systems provide a unified, high-performance architecture. This architecture processes real-time data feeds and guarantees system health. But, performance and reliability are challenging. IT managers, system architects, and data engineers must address challenges with Kafka capacity planning to ensure the successful deployment, adoption, and performance of a real-time streaming platform. When something breaks, it can be difficult to restore service, or even know where to begin.
This webinar discusses best practices to overcome critical performance challenges for Kafka data streaming that can negatively impact the usability, operation, and maintenance of the platform, as well as the data and devices connected to it. Topics include: Kafka data streaming architecture, key monitoring metrics, offline partitioning, broker, topics, consumer groups, and topic lag.
Growing adoption of Hadoop and Spark has increased demand for Big Data and Performance Management solutions that operate at scale. However, enterprise organizations quickly realize that scaling from pilot projects to large-scale production clusters involves a steep learning curve. Despite progress, DevOps teams still struggle with multi-tenancy, cluster performance, and workflow monitoring. This webinar discusses the top considerations when choosing a big data performance management solution.
In this webinar, field engineer Alex Pierce discusses the key things to consider when choosing a big data performance management solution. Learn how to:
– Maximize your infrastructure investment
– Achieve up to 50 percent increase in throughput, and run more jobs on existing infrastructure
– Ensure cluster stability and efficiency
– Avoid overspending on unnecessary hardware
– Spend less time in backlog queues
Learn how to automatically tune and optimize your cluster resources, and recapture wasted capacity. Alex will walk through use case examples to demonstrate the types of results you can expect to achieve in your own big data environment.
Pepperdata Product Manager Heidi Carson, Pepperdata Field Engineer, Alex Pierce
Does your big data analytics platform provide you with the Spark recommendations you need to optimize your application performance and improve your own skillset? Explore how you can use Spark recommendations to untangle the complexity of your Spark applications, reduce waste and cost, and enhance your own knowledge of Spark best practices.
- Avoiding contention by ensuring your Spark applications are requesting
the appropriate amount of resources,
- Preventing memory errors,
- Configuring Spark applications for optimal performance,
- Real-world examples of impactful recommendations,
- and More!
Join Product Manager Heidi Carson and Field Engineer Alex Pierce from Pepperdata to gain real-world experience with a variety of Spark recommendations, and participate in the Q and A that follows.
Autoscaling is the process of automatically increasing or decreasing the computational resources delivered to a cloud workload based on need. This typically means adding or reducing active servers (instances) that are leveraged against your workload within an infrastructure. The promise of autoscaling is that workloads should get exactly the cloud computational resources they require at any given time, and you only pay for the server resources you need, when you need them. Autoscaling provides the elasticity that customers require for their big data workloads, but it can also lead to exorbitant runaway waste and cost.
Pepperdata provides automated deployment options that can be seamlessly added to your Amazon EMR, Google Dataproc, and Qubole environments to recapture waste and reduce cost. Join us for this webinar where we will discuss how DevOps can use managed autoscaling to be even more efficient in the cloud. Topics include:
– Types of scaling
– What does autoscaling do well? When should you be using it?
– Is traditional autoscaling limiting your big data success?
– What is missing? Why is this problem important?
– Managed cloud autoscaling with Pepperdata Capacity Optimizer
Big data analytics performance management is a competitive differentiator and a priority for data-driven companies. However, optimizing IT costs while guaranteeing performance and reliability in distributed systems is difficult. The complexity of distributed systems makes it critically important to have unified visibility into the entire stack. This webinar discusses how to maximize the business value of your big data analytics stack investment and achieve ROI while reducing expenses. Learn how to:
- Correlate visibility across big data applications and infrastructure for a complete and transparent view of performance and cost.
- Continuously tune your platform, and run up to 50% more jobs on Hadoop clusters.
- Optimally utilize resources, and ensure customer satisfaction.
- Simplify troubleshooting and problem resolution while resolving issues to meet SLAs.
In this webinar, learn specific ways to automatically tune and optimize big data cluster resources, recapture wasted capacity, and improve ROI for your big data analytics stack.
Observability is an extremely popular topic these days. What's driving this interest? Why is observability needed? What is the difference between observability and monitoring?
When IT Ops knows there is a problem, but they can't pinpoint or quickly get to the root cause, traditional monitoring approaches are not enough anymore. Achieving observability requires carefully correlating many different sources from logs, metrics, and traces. And this can present additional challenges in distributed environments that use containers and micro-services.
In this webinar, you’ll get the answers to these questions:
- Why is observability essential in distributed big data environments?
- What are the critical challenges of the multi-cloud and containerized world?
- How can analytics stack performance solutions help you move from monitoring to observability?
Gain the knowledge of Spark veteran, Alex Pierce on how to manage the challenges of maintaining the performance and usability of your Spark jobs
Apache Spark provides sophisticated ways for enterprises to leverage Big Data compared to Hadoop. However, the increasing amounts of data being analyzed and processed through the framework is massive and continues to push the boundaries of the engine.
This webinar draws on experiences across dozens of production deployments and explores the best practices for managing Apache Spark performance. Learn how to avoid common mistakes, improve the usability, supportability and performance of Spark.
Apache Hive is a powerful tool frequently used to analyze data while handling ad-hoc queries and regular ETL workloads. Despite being one of the more mature solutions in the Hadoop ecosystem, developers, data scientists and IT operators are still unable to avoid common inefficiencies when running Hive at scale. Inefficient queries can mean missed SLAs, negative impact on other users, and slow database resources. Poorly tuned platforms or poorly sized queues can cause even efficient queries to suffer.
This webinar discusses proven approaches to Hive query tuning that improve query speed and reduce cost. Learn how to understand the detailed performance characteristics of query workloads and the infrastructure-wide issues that impact these workloads.
Pepperdata Field Engineer, Kirk Lewis will discuss:
- Finding problem queries - Pinpointing delayed queries, expensive queries, and queries that waste CPU and memory
- Improving query utilization and performance with database and infrastructure metrics
- Ensuring your infrastructure is not adversely impacting query performance
Learn five ways to improve your Kafka operations’ readiness and platform performance through proven Kafka best practices.
The influx of data from a wide variety of sources is already straining your big data IT infrastructure. On top of that, data must be ingested, processed, and made available in near real-time to support business-critical use cases. Kafka data streaming is used today by 30% of Fortune 500 companies because of its ability to feed data in real-time into a predictive analytics engine in support of these use cases. However, there are critical challenges and limitations.
By following the latest Kafka best practices, you can more easily and effectively manage Kafka. Join us for a webinar where we will discuss five specific ways to help keep your Kafka deployment optimized and more easily managed.
Best practices covered:
-Monitoring key component states to understand Kafka cluster health
-Measuring crucial metrics to understand Kafka cluster performance
-Observing critical building blocks in the Kafka hardware stack
-Tracking important metrics for Kafka capacity planning
-Knowing what to alert on and what can be monitored passively
Supply chain and logistic challenges due to the global COVID-19 outbreak are making it difficult for companies to address their growing big data capacity needs and purchase and provision more servers as needed.
Many organizations are addressing these issues by expediting the use of cloud services, but this can get costly if the infrastructure is not optimized. A better solution is to improve performance and get more out of your existing infrastructure.
Even the most experienced IT operations teams and capacity planners can’t manually tune every application and workflow.
The scale—thousands of applications per day and a growth rate of dozens of nodes per year—is too large for manual efforts.
There’s a better way: Automatic capacity optimization eliminates manual tuning and allows you to run 30-50% more jobs on your existing Hadoop or Spark clusters.
This webinar discusses four specific ways to automatically tune and optimize cluster resources, recapture wasted capacity, and improve your big data analytics stack ROI—on-premises or in the cloud.
451 Research Data Analyst, James Curtis, Pepperdata Customer VP, Joel Stewart
You and your organization just survived migrating to the cloud. A successful Day 1 is accomplished. But what about cloud migration Day 2? Are you prepared for life in the cloud? Your stakeholders and SLAs aren’t going to wait until things settle down. Planning for success after a cloud migration can mean the difference between seeing the ROI that cloud promises or having to consider moving back to an on-premises solution.
Creating an accurate cloud footprint requires good planning, a deep understanding of resource utilization, and granular data. Register for this webinar and learn how you can avoid the pitfalls of your cloud migration Day 2 and continue to make data a driving force in your business.
Learn how to create a durable, low latency message system by getting improved clarity into your Kafka data streaming pipeline.
In this webinar, learn how to:
- forecast Kafka data streaming capacity needs to protect throughput performance
- correlate infrastructure and application metrics across Kafka, Spark, Hive, Impala, HBase, and more
- automatically detect and alert on atypical Kafka scenarios to prevent data loss
- ensure preservation of SLAs for real-time stream processing applications
Kirk Lewis will cover the challenges around monitoring Kafka data streaming analytics and how Pepperdata can help. Pepperdata enables customers to integrate Kafka metrics into a big data analytics dashboard and get detailed visibility into Kafka cluster metrics, broker health, topics, partitions, and the rate of data coming in and going out.
Get Deep Insight into Query Execution and Database Performance by Using Pepperdata Query Spotlight
Enterprise customers report that Hive queries are a significant portion of their analytics workloads, and the performance of these workloads is critical to their big data success. Inefficient queries can mean missed SLAs, negative impact on other users, and slow database resources.
In this webinar, Field Engineer Alex Pierce divulges how to get the 360° query view that you need as well as how to overcome the key issues customers face with queries in their deployments. Topics include:
- Simplifying root cause analysis with visibility into delayed queries, most expensive queries, and queries that are wasting CPU and memory.
- Improving query utilization and performance with database and infrastructure metrics.
- Resolving problems faster through improved visibility and immediate feedback through real-time metrics.
Alex will demonstrate the new Pepperdata query performance management solution: Pepperdata Query Spotlight. Query Spotlight makes it easy to understand the detailed performance characteristics of query workloads, together with infrastructure-wide issues that impact these workloads. With this new functionality, operators and developers can tune query workloads, debug, and optimize for better performance and reduced costs, both in the cloud and on-premises.
Learn How You Can Migrate to the Cloud and Reduce the Management Costs of a Hybrid Data Center
In the early days of cloud migration, it was all upside: Operating a data center in the cloud was always cheaper than dedicated on-premises servers. Fast-forward a few years, IT Operations is in a visibility crisis and many big data teams cannot understand what they are spending or why.
Ultimately, in the quest to control and understand cloud spend, analytics are critically important. Without powerful, in-depth insights, big data teams simply don’t have the information they need to do their job.
Please join Pepperdata CEO, Ash Munshi; Peter Cnudde, former VP of Engineering of Yahoo's Big Data and Machine Learning platforms; Pepperdata Field Engineer, Alex Pierce, for a roundtable Q and A discussion on how to take the guesswork out of migrating to the cloud, and reduce the runaway management costs of a hybrid data center.
Would your big data organization benefit from automatic capacity optimization that eliminates manual tuning and enables you to run 30-50% more jobs on your Hadoop clusters?
As analytics platforms grow in scale and complexity on-prem and in the cloud, managing and maintaining efficiency is a critical challenge, and money is wasted.
In this webinar, Pepperdata Field Engineer Eric Lotter discusses how your organization can:
– Maximize your infrastructure investment
– Achieve up to 50 percent increase in throughput and run more jobs on existing infrastructure
– Ensure cluster stability and efficiency
– Avoid overspending on unnecessary hardware
– Spend less time in backlog queues
On a typical cluster, hundreds and even thousands of decisions are made per second, increasing typical enterprise cluster throughput up to 50 percent. Even the most experienced operator dedicated to resource management can’t make manual configuration changes with the required precision and speed. Learn how to automatically tune and optimize your cluster resources, and recapture wasted capacity. Eric will provide relevant use case examples and the results achieved to show you how to get more out of your infrastructure investment.
Pepperdata is the Big Data performance company. Fortune 1000 enterprises depend on Pepperdata to manage and optimize the performance of Hadoop and Spark applications and infrastructure. Developers and IT Operations use Pepperdata soluions to diagnose and solve performance problems in production, increase infrastructure efficiencies, and maintain critical SLAs. Pepperdata automatically correlates performance issues between applications and operations, accelerates time to production, and increases infrastructure ROI. Pepperdata works with customer Big Data systems on-premises and in the cloud.