Name: Storing and Using Metrics for 3000 nodes - How Instaclustr use a Time Series Cas
Start: 2019-10-03T17:00:00Z
End: 2019-10-03T17:43:41.000Z
Location: BrightTALK
Rating: 5

Instaclustr delivers reliability at scale through our integrated data platform of open source technologies such as Apache Cassandra, Apache Kafka, Apache Spark and Elasticsearch. Our expertize stems from delivering more than 25+ million node hours under management, allowing us to run the world’s most powerful data technologies effortlessly.

We provide a range of managed, consulting and support services to help our customers develop and deploy solutions around open source technologies. Our integrated data platform, built on open source technologies, powers mission critical, highly available applications for our customers and help them achieve scalability, reliability and performance for their applications.

Can you survive losing a Kafka cluster? Do you need to have your Kafka cluster replicated in multiple geographies? Do you have a business that needs to distribute events to multiple regions?

If your Kafka cluster fails for any reason you will need to quickly and reliably switch over applications to another cluster. 


In this webinar, we will cover various approaches and strategies for this situation.
- Active/Passive Strategy: We will provide implementation and planning for a switchover of Kafka applications if one cluster is currently not available.
- Active/Active Strategy: We will provide implementation and planning for serving the same Kafka content from multiple geographies using smart partitioning strategy


Finally, we will cover various technologies such as MirrorMaker2 and the whole framework approach for implementing the solutions outlined above.

Disaster Recovery for Your Kafka Clusters

Ensuring reliable data storage and delivery with Apache Kafka can prove to be a concern. Those trying to implement Apache Kafka are forced to deal with critical questions such as: how data order can be at risk? How data could potentially be lost? How records could be accidentally duplicated?


Join this webinar as we explore the essential components of building an effective strategy for data integrity with Apache Kafka. This webinar will also dive into how to resolve the above issues using a combination of:
- Effective Topic and Partitioning strategies
- Effective Data Keying Strategies
- Exactly once Semantics with Producers and Consumers

Strategies to Ensure Data Integrity with Apache Kafka

Join our Technical Evangelist, Paul Brebner, in a journey of exploration upriver with "Kongo", a scalable streaming IoT logistics demonstration application using Apache Kafka, the popular open-source distributed streaming platform. Along the way you'll discover: an example logistics IoT problem domain (involving the rapid movement of thousands of goods by trucks between warehouses, with real-time checking of complex business and safety rules from sensor data); an overview of the Apache Kafka architecture and components; lessons learned from making critical Kaka application design decisions; an example of Kafka Streams for checking truck load limits; and finish the journey by overcoming final performance challenges and shooting the rapids to scale Kongo on a production Kafka cluster.

Kongo: Building a Scalable Streaming IoT Application using Apache Kafka

This webinar will overview the key trends that Instaclustr is seeing in the adoption of open source solutions for large scale, high-reliability applications. What are the predominant use cases and technologies and what approaches are driving the most successful adoption? Looking forward, we will survey the trends shaping the development and proliferation of open source technology and how enterprises can shape their technology strategy to take advantage of these developments.

The Future of Open Source 2020 + Beyond

Instaclustr is back with part 2 of Apache Kafka 101 webinar series for an audience coming from traditional ESB (Enterprise Service Bus), Middleware and ETL based systems who want to learn about a modern-day approach to message processing, stream processing, real-time analytics, event sourcing, or acting as an Enterprise-wide central data bus.

In part-1 of this series, we looked at the basics of Apache Kafka, Kafka ecosystem, an overview of its architecture and explored concepts like brokers, topics, partitions, logs, producers, consumers, consumer groups, etc. 

Instaclustr is now organising part 2 of this webinar series. In this part, we will talk about topic design and partitioning. We will at good practices to keep in mind when designing for the required throughput of your applications. We will also look at the message key role with regards to guaranteeing message order. This will move you forward another step in your journey of learning and making the best use of Apache Kafka.

Apache Kafka Guide 101 Part 2

This session walks devs through Instaclustr's Cassandra tools and how they add key functionality and ease-of-use to their deployments. 

-An LDAP authenticator plug-in for Cassandra: The open-source LDAP authenticator plug-in works closely with the existing Cassandra Authorizer implementation. The plug-in enables developers to quickly reap the benefits of secure LDAP authentication without the need to write their own solutions, and to transition to using the authenticator with zero downtime. 

-A Kerberos authenticator plug-in for Cassandra: The open-source Kerberos authenticator plug-in enables Cassandra users to leverage Kerberos’ industry-leading secure authentication and true single sign-on capabilities. The open-source project also includes a Kerberos authenticator plugin for the Cassandra Java driver. 

-Cassandra Prometheus Exporter: The Cassandra-exporter is a high-performance metrics collection agent that allows for easy integration with the Prometheus monitoring solution. It has been designed to collect detailed metrics on production-sized clusters with complex schemata with minimal performance impact while at the same time following Prometheus's best practices for exporting metrics.

- Additional utilities and debugging tools including a utility to assist with backup and restore to various cloud providers, and tools to provide debug-level information about SSTables.

- A Cassandra operator for running and operating Cassandra within Kubernetes: The open-source Cassandra operator functions as a Cassandra-as-a-Service on Kubernetes, fully handling deployment and operations duties so that developers don’t have to. It also offers a consistent environment and set of operations founded on best practices, which is reproducible across production clusters and development, staging, and QA environments. 

The audience for this presentation will learn the specifics of how to implement – and get the most out of – these open source solutions.

Instaclustr's Open Source Tools For Cassandra-LDAP/Kerberos, Prometheus Exporter

Are you responsible for building Data pipelines for your organization or for your customers? Are you coming from a background where you worked on traditional ESB (Enterprise Service Bus), Middleware and ETL based systems but are no longer comfortable with modern trends in these domains? Or are you just interested in learning a technology that provides framework for message processing, stream processing, real time analytics, event sourcing, or acting as a central data bus for your organization. 

Apache Kafka is the leading distributed streaming and queuing technology for large-scale, always-on applications. Kafka has built-in features of horizontal scalability, high-throughput and low-latency. It is highly reliable, has high-availability, and allows geographically distributed data streams and stream processing applications. It’s used by pretty much all the big tech giants; LinkedIn, Netflix, Uber, Twitter, AirBnB and traditional names like Adidas, Goldman Sachs; the list goes on. 

Instaclustr is organizing a multi-part webinar series on Apache Kafka with an aim to help you in your journey of learning and making the best use of this technology.

In part-1 of this series, we will start with the basics of Apache Kafka, Kafka ecosystem, overview of its architecture and explore concepts like brokers, topics, partitions, logs, producers, consumers, consumer groups, etc.

Apache Kafka 101

Instaclustr is back with part 2 of Apache Kafka 101 webinar series for an audience coming from traditional ESB (Enterprise Service Bus), Middleware and ETL based systems who want to learn about a modern-day approach to message processing, stream processing, real-time analytics, event sourcing, or acting as an Enterprise-wide central data bus.
 
In part-1 of this series, we looked at the basics of Apache Kafka, Kafka ecosystem, an overview of its architecture and explored concepts like brokers, topics, partitions, logs, producers, consumers, consumer groups, etc.      

Instaclustr is now organising part 2 of this webinar series. In this part, we will talk about topic design and partitioning. We will at good practices to keep in mind when designing for the required throughput of your applications. We will also look at the message key role with regards to guaranteeing message order. This will move you forward another step in your journey of learning and making the best use of Apache Kafka.

Instaclustr is back with part 2 of Apache Cassandra 101  webinar series with an aim to help Software Engineers,  Database developers, DBA, DevOps, Enterprise or Solution Architects, take the next step in their journey of learning and making the best use of Apache Cassandra.

In part-1 of this series, we looked at Cassandra basics, compared it with RDBMS for good and bad, and then moved on to explore concepts like replication, partitioning, token ranges, consistency.

In part 2, we will explore in detail how Cassandra stores data i.e. write path and how data is retrieved i.e. read path. In this process, you will learn about some key concepts like memtable, SSTables, Co-ordinator, Compaction, Bloom filter, Row and key caches etc.

Apache Cassandra Guide 101 Part 2

If you are a Software Engineer, Database developer, DBA, DevOps, Enterprise or Solution Architect who has heard of Big Data and NoSQL but doesn't know exactly what the fuss is all about and want to learn more then this webinar is for you.

Apache Cassandra is a highly available, linearly scalable, fault-tolerant database that can offer extremely high throughput and very low latency. It's one of the most popular and most successful NoSQL databases. It has been widely adopted by biggest tech names, major world-renowned brands and research institutes alike. This list includes companies like Apple, Twitter, Facebook, Netflix, Rackspace, Reddit and also research institutes like CERN. 

Instaclustr is organizing a multi-part webinar series on Apache Cassandra with an aim to help you in your journey of learning and making the best use of Apache Cassandra.

In part-1 of this series, we will start with demystifying NoSQL, look at Cassandra basics, compare it with RDBMS for good and bad, and then move on to explore concepts like replication, partitioning, token ranges, consistency etc.

Apache Cassandra 101

Are you responsible for building Data pipelines for your organization or for your customers? Are you coming from a background where you worked on traditional ESB (Enterprise Service Bus), Middleware and ETL based systems but are no longer comfortable with modern trends in these domains? Or are you just interested in learning a technology that provides framework for message processing, stream processing, real time analytics, event sourcing, or acting as a central data bus for your organization. 

Apache Kafka is the leading distributed streaming and queuing technology for large-scale, always-on applications. Kafka has built-in features of horizontal scalability, high-throughput and low-latency. It is highly reliable, has high-availability, and allows geographically distributed data streams and stream processing applications. It’s used by pretty much all the big tech giants; LinkedIn, Netflix, Uber, Twitter, AirBnB and traditional names like Adidas, Goldman Sachs; the list goes on. 
 
Instaclustr is organising a multi-part webinar series on Apache Kafka with an aim to help you in your journey of learning and making the best use of this technology.

In part-1 of this series, we will start with the basics of Apache Kafka, Kafka ecosystem, overview of its architecture and explore concepts like brokers, topics, partitions, logs, producers, consumers, consumer groups, etc.

Apache Kafka 101 Guide Part 1

If you are a Software Engineer,  Database developer, DBA, DevOps, Enterprise or Solution Architect who has heard of Big Data and NoSQL but doesn't know exactly what the fuss is all about and want to learn more then this webinar is for you.

Apache Cassandra is a highly available, linearly scalable, fault-tolerant database that can offer extremely high throughput and very low latency. It's one of the most popular and most successful NoSQL databases. It has been widely adopted by biggest tech names, major world-renowned brands and research institutes alike.  This list includes companies like Apple, Twitter, Facebook, Netflix, Rackspace, Reddit and also research institutes like CERN. 

Instaclustr is organising a multi-part webinar series on Apache Cassandra with an aim to help you in your journey of learning and making the best use of Apache Cassandra.

In part-1 of this series, we will start with demystifying NoSQL, look at Cassandra basics, compare it with RDBMS for good and bad, and then move on to explore concepts like replication, partitioning, token ranges, consistency etc.

Apache Cassandra 101 Guide Part 1

Apache Kafka, Apache Cassandra and Kubernetes are open source big data technologies enabling applications and business operations to scale massively and rapidly. While Kafka and Cassandra underpins the data layer of the stack providing capability to stream, disseminate, store and retrieve data at very low latency, Kubernetes is a container orchestration technology that helps in automated application deployment and scaling of application clusters. In this webinar, we will discuss how we architected a massive scale deployment of a streaming data pipeline with Kafka and Cassandra to cater to an example Anomaly detection application running on a Kubernetes cluster and generating and processing massive amount of events. Anomaly detection is a method used to detect unusual events in an event stream. It is widely used in a range of applications such as financial fraud detection, security, threat detection, website user analytics, sensors, IoT, system health monitoring, etc. When such applications operate at massive scale generating millions or billions of events, they impose significant computational, performance and scalability challenges to anomaly detection algorithms and data layer technologies. We will demonstrate the scalability, performance and cost effectiveness of Kafka and Cassandra, with results from our experiments allowing the Anomaly detection application to scale to 19 Billion anomaly checks per day.

Kafka, Cassandra and Kubernetes: Real-time Anomaly Detection at Scale

Gain confidence to build on open source Apache Cassandra

Introducing the Instaclustr Open Source Certification Framework and Certified Apache Cassandra. Learn about Instaclustr's new Open Source Certification Framework and first certified product, Apache Cassandra. The certification framework aims to build on Instaclustr’s extensive open source experience to provide additional assurance to companies that the open source software they are building on is robust and well supported. This webinar will explain the details of the framework and the certification process for Apache Cassandra.

Bio: Ben Slater, Chief Product Officer at Instaclustr. As Chief Product Officer, Ben is charged with steering Instaclustr’s development roadmap and overseeing the product engineering, production support, open source and consulting teams. Ben has over 20 years experience in systems development including previously as lead architect for the product that is now Oracle Policy Automation and over 10 years as a solution architect and project manager for Accenture. He has extensive experience in managing development teams and implementing quality controlled engineering practices.

Introducing Certified Apache Cassandra

At Instaclustr, we use one of our own Cassandra clusters and a custom monitoring application to store, process, and retrieve metrics on every single node in our fleet. The talk will introduce how we collect, process, store, and rollup all the metrics which pass through our monitoring system every second. We will discuss why using Cassandra, time buckets, and Spark is really suited to efficiently and quickly store, and query all that monitoring data.

Storing and Using Metrics for 3000 nodes - How Instaclustr use a Time Series Cas

Cassandra

Database Management

Open Source

SPARK

Apache Spark

NoSQL

Get powerful e-commerce insights to grow your online transaction volume. Connect with thought leaders and colleagues to get the most up-to-date knowledge on the e-commerce strategies and techniques that drive growth.

E-commerce

The marketing community on BrightTALK is made up of thousands of engaged marketing professionals. Find relevant webinars and videos on marketing strategy, branding and more presented by recognized thought leaders. Join the conversation by participating in live webinars and round table discussions.

Marketing

Cloud computing has exploded over the past few years, delivering a previously unimagined level of workplace mobility and flexibility. The cloud computing community on BrightTALK is made up of thousands of engaged professionals learning from the latest cloud computing research and resources. Join the community to expand your cloud computing knowledge and have your questions answered in live sessions with industry experts and vendor representatives.

Cloud Computing

Practicing business intelligence allows your company to transform raw data into sets of insights for targeted business growth. The business intelligence and analytics community on BrightTALK is made up of thousands of data scientists, database administrators, business analysts and other data professionals. Find relevant webinars and videos on business analytics, business intelligence, data analysis and more presented by recognized thought leaders. Join the conversation by participating in live webinars and round table discussions.

Business Intelligence and Analytics

Welcome to the big data and data management community on BrightTALK. Join thousands of data quality engineers, data scientists, database administrators and other professionals to find more information about the hottest topics affecting your data. Subscribe now to learn about efficiently storing, optimizing a complex infrastructure, developing governing policies, ensuring data quality and analyzing data to make better informed decisions. Join the conversation by watching live and on-demand webinars and take the opportunity to interact with top experts and thought leaders in the field.

Big Data and Data Management

Get powerful marketing data and analytics insights from influential experts. Connect with thought leaders and colleagues to get the most up-to-date knowledge on the analytics that quantify your investments.

Data and Analytics

As an IT professional, many of the problems you face are multifaceted, complex and don’t lend themselves to simple solutions. The information technology community features useful and free information technology resources. Join to browse thousands of videos and webinars on ITIL best practices, IT security strategy and more presented by leading CTOs, CIOs and other technology experts.

Storing and Using Metrics for 3000 nodes - How Instaclustr use a Time Series Cas

Presented by

About this talk

More from this channel