Name: Build your open data lakehouse on Apache Iceberg
Start: 2022-08-19T19:21:00Z
End: 2022-08-19T20:16:56.000Z
Location: BrightTALK
Rating: 0

Dremio is the easy and open data lakehouse, providing self-service analytics with data warehouse functionality and data lake flexibility across all of your data. Dremio increases agility with a revolutionary data-as-code approach that enables Git-like data experimentation, version control, and governance.

Most companies use Hadoop for big data analytical workloads. The problem is, on-premises Hadoop deployments have failed to deliver business value after it is implemented. Over time, the high cost of operations and poor performance places a limitation on an organization’s ability to be agile.  As a result, data platform teams are looking to modernize their Hadoop workloads to the data lakehouse.

In this video, you'll learn about:
- Use cases for modernizing Hadoop workloads
- How the data lakehouse solves the inefficiencies of on-premises Hadoop
- Success stories from organizations that have modernized Hadoop with the data lakehouse on Dremio

Hadoop to Data Lakehouse Migration: Use Cases, Benefits, & Success Stories

Enterprise data platform teams are often frustrated with inheriting infrastructures that impact their ability to access reliable and high-quality data. Economic factors are forcing  organizations to become more capital-efficient, and teams are now building their applications on the data lakehouse with Dremio.

In this video, you will learn about:
- The importance of a simplified, open data architecture
- Simplifying your on-prem and cloud analytical workloads with Dremio
- Use cases on the data lakehouse

Getting Started with Dremio’s Data Lakehouse

Most users of the Hadoop platform are fed up with its high cost of operational overhead and poor performance. With innovations around open source standards, like Apache Iceberg and Arrow, the data lakehouse has emerged as the destination for companies migrating off Hadoop.

In this video of Gnarly Data Waves, you will learn about: 
- 5 key factors to consider as you migrate off Hadoop to the Data Lakehouse
- Why Apache Iceberg replaced Hive metastore
- Creating a unified access layer on your data lakehouse with Dremio

Five Factors to Consider When Migrating from Hadoop to the Data Lakehouse

As organizations strive to provide value faster to end users, data silos makes it difficult to provide insights on time.  Learn how Dremio’s data lakehouse accelerates data delivery and discovery, without copies.

Dremio’s easy and open data lakehouse helps organizations overcome these common challenges.

In this video of Gnarly Data Waves, you will learn about:
- Dremio’s Data Lakehouse with live demos of both Dremio Sonar and Dremio Arctic
- Apache Iceberg and Dremio’s lakehouse capabilities with Iceberg tables
- Customer data lakehouse use cases, including stories from Shell, TransUnion, and NCR.

Getting Started with Dremio: Sonar | Arctic Live Demo and Customer Use Cases

So you want to base your data lakehouse around Apache Iceberg to take advantage of its features, performance and vast ecosystem of platform/tool compatibility. You’ll need to take your current Hive tables and convert them to Iceberg. Iceberg offers several tools to do so depending on your needs and in this video we’ll explore that migration process.

- How to do an in-place migration and avoid rewriting your data
- How to do a shadow migration where you can update your data’s schema and partitioning
- How to move Apache Iceberg tables from one catalog to another

How to Modernize Hive to the Data Lakehouse with Dremio and Apache Iceberg

Watch Dremio developer advocacy and engineering teams for an installment of Apache Iceberg Office Hours. During this time we’ll have a brief Iceberg presentation on Hidden Partitioning and Partitioning transforms in Iceberg and then lots of dedicated time for Q&A on the presented topic or any other questions or guidance you’re looking for help on in learning about Apache Iceberg or architecting your data lakehouse around Apache Iceberg.

Examples of questions you can come to ask:
- How can I optimize my Iceberg tables for my different use cases?
- What tools will best handle my ETL job to write to Iceberg?
- How can I control access to my Iceberg tables?
- How can I convert data from X into an Iceberg table?
- How can I get started with Iceberg in Databricks?

Apache Iceberg Office Hours April Edition

Querying 100s of petabytes of data demands optimized query speed specifically when data accumulates over time. We have to ensure that the queries remain efficient because over time you may end up with a lot of small files and your data might not be optimally organized.

In this talk, we will cover:
- Apache Iceberg table format
- Problems in the data lake: small files, unorganized files
- Techniques such as: partitioning, compaction, metrics filtering
- Overlapping metrics problem
- Solving it using sorting, Z-order clustering

Optimizing Data Files in Apache Iceberg: Performance strategies

The data lakehouse is quickly emerging as the ideal data architecture because it combines the flexibility and scalability of data lakes with the data management, data governance, and data analytics capabilities of data warehouses. Table formats bring many of the “house” features to the data lakehouse. Apache Iceberg is a truly open table format that is built for easy management and high performance analytics on the largest data volumes in the world.  

In this video, we’ll discuss:
- Why open table formats are fundamental to building a data lakehouse
- How Fivetran automates data movement and helps organizations easily move data from various sources to their Amazon S3 data lake in Apache Iceberg tables.
- How Dremio & Fivetran simplify your data lakehouse architecture while providing high performance and ease of use.

Build your open data lakehouse on Iceberg with Fivetran and Dremio

As data lakes become the primary destination for growing volumes of customer and operational data, data teams need tools and processes that ensure data quality and consistency across data consumers and use cases. Join Dremio’s Jeremiah Morrow and Alex Merced as they discuss the emergence of data as code for data management, its benefits for data teams, and how Dremio customers are using it to deliver access to a consistent and accurate view of data in their data lakes. 

In this video on Gnarly Data Waves - Managing your data as code with Dremio Arctic, you will learn about:
- Why data as code is necessary for ensuring consistency and data quality for large data lakes.
- How Dremio Arctic uses Git-like concepts such as branches, tags, and commits to make data management easy.
- Some high value use cases for data as code.

Managing your data as code with Dremio Arctic

As enterprise data platforms look to operate at a more efficient level, they face the pressure to pivot their data management strategies. The increasing volume of data, demand for self-service analytics that meets compliance requirements, and complexity of data distribution channels are all factors to consider when making a business case.

Watch Dremio and Forrester, as we talk about the three-year Total Economic Impact™ of the data lakehouse and quantifiable benefits to productivity across all teams. You will learn about:

- Key challenges organizations face with explosive data growth and data silos
- Increasing team productivity and focusing more on high-value projects
- Reducing data storage costs and retiring complicated ETL processes

Total Economic Impact of the Data Lakehouse

Watch the Dremio developer advocacy and engineering teams for an installment of Apache Iceberg Office Hours. During this time we’ll have a brief Iceberg presentation on Hidden Partitioning and Partitioning transforms in Iceberg and then lots of dedicated time for Q&A on the presented topic or any other questions or guidance you’re looking for help on in learning about Apache Iceberg or architecting your data lakehouse around Apache Iceberg.

Examples of questions:
- How can I optimize my Iceberg tables for my different use cases?
- What tools will best handle my ETL job to write to Iceberg?
- How can I control access to my Iceberg tables?
- How can I convert data from X into an Iceberg table?
- How can I get started with Iceberg in Databricks?

Apache Iceberg Office Hours

Tableau is a visual analytics platform that helps more people in organizations see and understand their data. Dremio helps Tableau users accelerate access to data, including cloud data lakes, and it can dramatically improve query performance, delivering analytics for every data consumer at interactive speed. 

In this video, you will learn:
- How the Dremio open data lakehouse connects Tableau users directly to data lake storage and other data repositories
- How reflections accelerate query performance for ad hoc analysis and interactive dashboards
- How the Dremio semantic layer extends self-service capabilities beyond the visualization layer, so anyone can join and query data easily.

Best Practices for Optimizing Tableau Dashboards with Dremio

Every organization considers dashboards a key asset to support their decision-making process. BI dashboards have been crucial in democratizing data but the underlying data architecture that powers them has some serious downsides.

In this video, we’ll explore:
- The traditional approach to building dashboards
- The problems with this approach
- How Dremio & Lakehouse architecture provides a better way
- Demo of an Apache Superset dashboard migration (5 easy steps)
- Benefits of this new approach

Migrating a BI Dashboard to your Data Lakehouse with Apache Superset and Dremio

Iceberg has been gaining wide adoption in the industry as the defacto open standard for data lakehouse table formats. This video will help you learn the options and strategies you can employ when migrating tables from Delta Lake to Apache Iceberg.  

We’ll cover:
- Why migrate to Apache Iceberg
- How to do an In-place migration and avoid rewriting files
- How to do a shadow migration
- Best practices

Migrating from Delta Lake to Iceberg

With the explosive growth in data volumes, data access requests, and interest in data democratization, data security and governance has become more important than ever. To ensure secure, reliable, and proper access to data, organizations need to implement a centralized data security governance strategy. Dremio & Privacera have partnered together to provide a solution that allows organizations to easily implement fine-grained access controls on their open data lakehouse.  

This video will detail how Dremio & Privacera can provide organizations with the tools to build their unified data security governance platform.   

You'll learn:
- The importance of implementing unified security data governance for the open data lakehouse
- How to use Privacera to implement fine-grained access controls for queries on Dremio’s open data lakehouse
- How the integrated Dremio & Privacera solution provides column and row access controls across federated datasets
- Best practices for deploying and using the Dremio & Privacera solution

Centralize Data Security Governance with Dremio & Privacera

Deriving value from data is critical for businesses to not just thrive, but even survive, in the world today. However, oftentimes costs can run sky-high in trying to achieve more and more of this value. 

In this panel, we’ll discuss real-world strategies for optimizing data analytics costs, including topics such as:
- How pressure to be product-led affects the balance between centralized and decentralized control
- Misalignment between cost-control objectives and organizational structure
- Chargeback in a data mesh environment
- The relationship between cost controls and vendor lock-in

Real World Strategies for Optimizing Data Analytics Costs

Data lakes will be the primary destination for growing volumes and variety of data, and more technical and non-technical data consumers will require access to that data for Business Intelligence and reporting workloads. In this session, learn how the Dremio Open Data Lakehouse and dbt work together to give data teams all of the data management, data governance, and analytics capabilities of the data warehouse directly on the data lake. 

You'll learn:
- How anyone who knows SQL can transform, join, and query data.
- How the open data lakehouse centralizes data governance and security while enabling broad access for data consumers
- How easy it is to get started with the open data lakehouse with a brief demonstration of dbt & Dremio

Build Your Open Data Lakehouse with Dremio and dbt Labs

A modern open data lakehouse, with data stored in vendor-agnostic formats, is the architecture that best enables data democratization, both today and for years to come. Learn why and how to build a modern open data lakehouse on Apache Iceberg.  

We’ll be discussing:
- the most common problems in data architectures
- how adopting a lakehouse architecture solves these problems
- the advantages of the Iceberg table format over other formats

And we’ll show a demo of an open lakehouse in action!

Build your open data lakehouse on Apache Iceberg

#datalakehouses

#Apache Iceberg

#Analytics

#OpenLakehouse

#DataIntegration

#data democratization

#dremio

#AWS

Welcome to the virtualization community on BrightTALK! Whether it affects servers, storage, networks, desktops or other parts of the data center, virtualization provides real benefits by reducing the resources needed for your
infrastructure and creating software-defined data center components. However, it can also complicate your infrastructure. Join this active community to learn best
practices for avoiding virtual machine sprawl and other common virtualization pitfalls as well as how you can make the most of your virtualization environment.

Virtualization

Are you an IT service management professional interested in developing your knowledge and improving your job performance? Join the IT service management community to access the latest updates from industry experts. Learn and share insights related to IT service management (ITSM) including topics such as the service desk, service catalog, problem and incident management, ITIL v4 and more. Engage with industry experts on current best practices and participate in active discussions that address the needs and challenges of the ITSM community.

IT Service Management

Cloud computing has exploded over the past few years, delivering a previously unimagined level of workplace mobility and flexibility. The cloud computing community on BrightTALK is made up of thousands of engaged professionals learning from the latest cloud computing research and resources. Join the community to expand your cloud computing knowledge and have your questions answered in live sessions with industry experts and vendor representatives.

Cloud Computing

Increasing expectations for good governance, effective risk management and complex demands for corporate compliance are presenting a growing challenge for organizations of all sizes. Join industry thought leaders as they provide you with practical advice on how to implement successful risk and compliance management strategies across your organization. Browse risk management resources in the form of interactive webinars and videos and ask questions of expert GRC professionals.

IT Governance, Risk and Compliance

The data center management community focuses on the holistic management and optimization of the data center. From technologies such as virtualization and cloud computing to data center design, colocation, energy efficiency and monitoring, the BrightTALK data center management community provides the most up-to-date and engaging content from industry experts to better your infrastructure and operations. Engage with a community of your peers and industry experts by asking questions, rating presentations and participating in polls during webinars, all while you gain insight that will help you transform your infrastructure into a next generation data center.

Data Center Management

The application development community features top thought leadership focusing on optimal practices in software development, SDLC methodology, mobile app development and application development platforms and tools. Join top software engineers and coders as they cover emerging trends in everything from enterprise app development to developing for mobile platforms such as Android and iOS.

Application Development

Practicing business intelligence allows your company to transform raw data into sets of insights for targeted business growth. The business intelligence and analytics community on BrightTALK is made up of thousands of data scientists, database administrators, business analysts and other data professionals. Find relevant webinars and videos on business analytics, business intelligence, data analysis and more presented by recognized thought leaders. Join the conversation by participating in live webinars and round table discussions.

Business Intelligence and Analytics

The IT project management community on BrightTALK includes thousands of IT project and portfolio management professionals. Find relevant webinars and videos on agile methodologies, scrum strategy, project management processes and more. Attend live webinars or view on demand content presented by recognized thought leaders in the IT project management industry.

IT Project Management

Network infrastructure professionals understand that a reliable and secure infrastructure is crucial to enabling business execution. Join the network infrastructure community to interact with thousands of IT professionals. Browse hundreds of on-demand and live webinars and videos to learn about the latest trends in network computing, SDN, WAN optimization and more.

Network Infrastructure

Welcome to the big data and data management community on BrightTALK. Join thousands of data quality engineers, data scientists, database administrators and other professionals to find more information about the hottest topics affecting your data. Subscribe now to learn about efficiently storing, optimizing a complex infrastructure, developing governing policies, ensuring data quality and analyzing data to make better informed decisions. Join the conversation by watching live and on-demand webinars and take the opportunity to interact with top experts and thought leaders in the field.

Big Data and Data Management

As an IT professional, many of the problems you face are multifaceted, complex and don’t lend themselves to simple solutions. The information technology community features useful and free information technology resources. Join to browse thousands of videos and webinars on ITIL best practices, IT security strategy and more presented by leading CTOs, CIOs and other technology experts.

Build your open data lakehouse on Apache Iceberg

Presented by

About this talk

More from this channel