Name: Automating the Open Data Lakehouse Using an Open Intelligent Metastore
Start: 2022-11-22T17:00:00Z
End: 2022-11-22T18:02:11.000Z
Location: BrightTALK
Rating: 0

Dremio is the easy and open data lakehouse platform, providing an intuitive UI that enables data teams to get started with analytics in minutes, as well as the flexibility to use Dremio’s lightning-fast SQL query service and any other data processing engine.

With the explosive growth in data volumes, data access requests, and interest in data democratization, data security and governance has become more important than ever. To ensure secure, reliable, and proper access to data, organizations need to implement a centralized data security governance strategy. Dremio & Privacera have partnered together to provide a solution that allows organizations to easily implement fine-grained access controls on their open data lakehouse.  

This video will detail how Dremio & Privacera can provide organizations with the tools to build their unified data security governance platform.   

You'll learn:
- The importance of implementing unified security data governance for the open data lakehouse
- How to use Privacera to implement fine-grained access controls for queries on Dremio’s open data lakehouse
- How the integrated Dremio & Privacera solution provides column and row access controls across federated datasets
- Best practices for deploying and using the Dremio & Privacera solution

Centralize Data Security Governance with Dremio & Privacera

Deriving value from data is critical for businesses to not just thrive, but even survive, in the world today. However, oftentimes costs can run sky-high in trying to achieve more and more of this value. 

In this panel, we’ll discuss real-world strategies for optimizing data analytics costs, including topics such as:
- How pressure to be product-led affects the balance between centralized and decentralized control
- Misalignment between cost-control objectives and organizational structure
- Chargeback in a data mesh environment
- The relationship between cost controls and vendor lock-in

Real World Strategies for Optimizing Data Analytics Costs

Data lakes will be the primary destination for growing volumes and variety of data, and more technical and non-technical data consumers will require access to that data for Business Intelligence and reporting workloads. In this session, learn how the Dremio Open Data Lakehouse and dbt work together to give data teams all of the data management, data governance, and analytics capabilities of the data warehouse directly on the data lake. 

You'll learn:
- How anyone who knows SQL can transform, join, and query data.
- How the open data lakehouse centralizes data governance and security while enabling broad access for data consumers
- How easy it is to get started with the open data lakehouse with a brief demonstration of dbt & Dremio

Build Your Open Data Lakehouse with Dremio and dbt Labs

Join the Dremio developer advocacy and engineering teams for the first installment of Apache Iceberg office hours. During this time we’ll have a brief Iceberg presentation on copy-on-write vs merge-on-read in Iceberg and then lots of dedicated time for Q&A on the presented topic or any other questions or guidance you’re looking for help on in learning about Apache Iceberg or architecting your data lakehouse around Apache Iceberg.

Examples of questions you can come to ask:
- How can I optimize my Iceberg tables for my different use cases?
- What tools will best handle my ETL job to write to Iceberg?
- How can I control access to my Iceberg tables?
- How can I convert data from X into an Iceberg table?
- How can I get started with Iceberg in Databricks?

Join us for the first installment of our regular Apache Iceberg office hours to help build your low-cost high performance data lakehouse!

Apache Iceberg Office Hours

Organizations with mission-critical applications need highly available deployments with high SLAs, as well as the ability to understand what's happening throughout the system. Delivering this level of availability and visibility requires good monitoring. 
 
In this webinar, experts with first-hand technical experience planning, deploying, monitoring, and supporting Dremio Software will discuss:
- How to monitor each component in Dremio Software
- What metrics to pay close attention to of each component
- High availability best practices

Best Practices for Monitoring Dremio Software

Self-service analytics is the goal for organizations that want to empower end users to become data-driven decision makers. The semantic layer has been a key pillar in making this a reality.
 
The problem with traditional semantic layer solutions is that they required additional data copying from the data warehouse into data marts and cubes. Data engineers, architects, and business intelligence specialists know the limitations all too well.
 
While the concept of using a semantic layer to query data isn’t new, simplified data architectures like the open data lakehouse have expanded the semantic layer’s potential, enabling faster, more efficient reporting and analytics across user groups.
 
Join Dremio’s solutions experts and learn
Common challenges with semantic layers and how to overcome them
Where the unified semantic layer fits in the open data lakehouse architecture
How a unified semantic layer simplifies ETL for data engineers without data sprawls

Enable No-Copy Data Marts with a Unified Semantic Layer

As a result of the accelerated growth of data lakes, data teams have been forced to either build and maintain expensive and complex processes to make new sources of data available for use in proprietary data warehouses, or hinder access to analytics for all data consumers. In this webinar, learn how the Alteryx Analytic Platform and Dremio Open Lakehouse combine to simplify data operations and enable broad access to the data lake for exploration, discovery, and insights.

Unlocking Analytics from your Data Lake with Alteryx and Dremio

Apache Iceberg is an open table format for large-scale analytics on the data lake. In this webinar, we’ll deep dive into the internals of an Apache Iceberg table from a practical point of view using Dremio Sonar as the compute engine.

In this webinar we will learn:
The various components of an Apache Iceberg table
Perform CREATE, INSERT, UPDATE & analyze what happens underneath with the structure
Learn about Iceberg metadata tables
Example use cases for leveraging metadata tables

Register for this webinar to gain a practical understanding of what happens under the hood of an Apache Iceberg table & how CRUD works.

A Hands-on Look at the Structure of an Apache Iceberg Table

According to the latest DZone Data Pipelines Trend Report, data professionals spend over half of their time working on data extraction, loading, and transformation, and the most prevalent methods of ingestion and transformation are manual and ad hoc ETL processes. As data sources proliferate, the challenge facing data teams will be to scale ingestion and transformation processes and respond quickly to a growing number of data access requests from technical and non-technical data consumers across their businesses.
 
In this on-demand session, we'll explore the results of the Trend Report, and show how an open data lakehouse can simplify your data architecture and accelerate time to value for the newest and fastest growing data sources.

DZone Data Pipeline Trends: Simplify Data Architectures With an Open Lakehouse

Although banking and finance companies have more customer data than ever before, many firms struggle to get timely insights into the hands of business leaders, and struggle to deliver a consistent customer experience across channels and lines of business. In this session, learn how direct access to analytics on Amazon S3 with Dremio can help banking and finance firms achieve a consistent and data-driven customer experience.

The Digital Bank: Build a data-driven customer experience on the open lakehouse

A modern open data lakehouse, with data stored in vendor-agnostic formats, is the architecture that best enables data democratization, both today and for years to come. Learn why and how to build a modern open data lakehouse on Apache Iceberg.  

We’ll be discussing:
- the most common problems in data architectures
- how adopting a lakehouse architecture solves these problems
- the advantages of the Iceberg table format over other formats

And we’ll show a demo of an open lakehouse in action!

Build your open data lakehouse on Apache Iceberg

Becoming a data-driven organization is a top priority for most executives. The newest and fastest-growing data sources across organizations are already being stored in cloud object storage.
One of the biggest data-driven business problems is data lake management. As the demand for data requests increases, it becomes more difficult to deliver data products to end users. 

Simplify data lake management at scale with DataOps -- a new paradigm taking software engineering principles of source code repositories and treating your data as code. 

In this webinar, learn new capabilities for data lake management as we discuss DataOps best practices on:
- Continuous integration and deployment (CI/CD)
- Data pipeline testing for quality
- Data lake governance

DataOps: A New Methodology for Data Lakehouse Management

Apache Iceberg offers the tools for query engines to make fast and efficient query plans on your data lakehouse. In this webinar, we’ll learn how Iceberg queries play out through planning and execution.

In this webinar we will learn:
- The step-by-step process of an INSERT query
- The step-by-step process of a DELETE query
- The step-by-step process of an UPSERT/MERGE query
- The step-by-step process of a SELECT query
- Understanding of Iceberg delete files

The Life of an Apache Iceberg Query

Data teams in large enterprise organizations are facing greater demand for data to satisfy a wide range of analytic use cases, yet they are challenged with providing access to all of their data across business units, regions, and cloud environments. In this webinar, learn how Enel Group worked with Agile Lab to implement Dremio as a data mesh solution for providing broad access to a unified view of their data, and how they use that architecture to enable a multitude of use cases.

In this session, you will learn:
- How the silos development led to challenges with data growth, data quality, data sharing, and data governance. An example of datamesh paradigm adoption.
- How Agile Lab and Enel Group used Dremio to connect their disparate organizations across geographies and business units.
- Leveraging Dremio for data governance and multi-cloud with Arrow Flight.

How Enel Group built a data mesh architecture with Dremio and Agile Lab

In this webinar, a panel of developers and system integrators discuss experiences with clients who have transitioned from traditional data warehousing to an open data lakehouse approach. TDWI analyst David Loshin will review core aspects of an open data lakehouse architecture and then moderate a conversation among data lakehouse practitioners.

Panel Discussion on Building Cloud Analytics Using an Open Lakehouse

The innovative idea of a data lake holding a collection of data assets in their original or raw forms has influenced cloud-based data storage tactics. This is especially true in organizations wanting to empower a variety of data consumers of different levels of technical expertise. However, it is apparent that the absence of governance coupled with limited data awareness has diminished the utility of the data lake.

Automating the Open Data Lakehouse Using an Open Intelligent Metastore

Dremio

Knowledge Integrity

Data Lake

Data Analytics

Cloud

Data Integration

Data Transformation

Data Warehouse

Open Lakehouse

Data Governance

Welcome to the big data and data management community on BrightTALK. Join thousands of data quality engineers, data scientists, database administrators and other professionals to find more information about the hottest topics affecting your data. Subscribe now to learn about efficiently storing, optimizing a complex infrastructure, developing governing policies, ensuring data quality and analyzing data to make better informed decisions. Join the conversation by watching live and on-demand webinars and take the opportunity to interact with top experts and thought leaders in the field.

Big Data and Data Management

As an IT professional, many of the problems you face are multifaceted, complex and don’t lend themselves to simple solutions. The information technology community features useful and free information technology resources. Join to browse thousands of videos and webinars on ITIL best practices, IT security strategy and more presented by leading CTOs, CIOs and other technology experts.

Automating the Open Data Lakehouse Using an Open Intelligent Metastore

Presented by

About this talk

More from this channel