Name: Getting Started with Dremio’s Data Lakehouse
Start: 2023-06-06T15:00:00Z
End: 2023-06-06T15:00:57.000Z
Location: BrightTALK
Rating: 5

Dremio is the easy and open data lakehouse, providing self-service analytics with data warehouse functionality and data lake flexibility across all of your data. Dremio increases agility with a revolutionary data-as-code approach that enables Git-like data experimentation, version control, and governance.

Watch this an insightful webinar video titled "ELT, ETL, and the Dremio Data Lakehouse," where we explore the cutting-edge capabilities of Dremio in revolutionizing data engineering and analytics workflows. This webinar delves into the strategic use of Dremio's innovative technologies to optimize Extract, Load, Transform (ETL) and Extract, Load, Transform (ELT) patterns for enhanced efficiency and cost-effectiveness.

The session will commence with an in-depth exploration of traditional ETL and ELT methodologies, highlighting the challenges faced by organizations in managing large-scale data transformations. We will analyze the critical role of ELT patterns in the modern data landscape and the growing significance of data lakes for storage and processing.

Subsequently, we will introduce Dremio, a powerful and flexible data lakehouse platform, as a game-changer for executing ETL and ELT operations. Dremio's unique architecture empowers users to directly query data residing in the data lake, eliminating the need for unnecessary data copies and reducing data movement overhead significantly.

During the webinar video, attendees will gain valuable insights into how Dremio's no-copy architecture minimizes data redundancy, accelerates data processing, and drastically reduces the associated costs. By harnessing the full potential of data lake storage, organizations can simplify their data engineering workflows, enhance data availability, and achieve unparalleled performance for analytical workloads.

Key webinar takeaways:
- A comprehensive overview of ETL and ELT patterns and their relevance in modern data environments.
- The rise of data lakes and the pivotal role of Dremio's data lakehouse platform in transforming data management paradigms.
- Understanding the benefits of Dremio's no-copy architecture in optimizing data processing and analytics.
- Best practices and practical implementation tips for leveraging Dremio effectively in ETL and ELT workflows.

ELT, ETL and the Dremio Data Lakehouse

In modern data architectures, stream processing engines such as Apache Flink are used to ingest continuous streams of data into data lakes such as Apache Iceberg. Streaming ingestion to Iceberg tables can suffer from two problems: the small files problem that can hurt read performance, and poor data clustering that can make file pruning less effective.

In this video, we will discuss how data teams can address those problems by adding a shuffling stage to the Flink Iceberg streaming writer to intelligently group data via bin packaging or range partition, reduce the number of concurrent files that every task writes, and improve data clustering. We will explain the motivations in detail and dive into the design of the shuffling stage. We will also share the evaluation results that demonstrate the effectiveness of smart shuffling.

Tame Small Files and Optimize Data Layout for Streaming Ingestion to Iceberg

Discover the power of Dremio, a robust Data Lakehouse platform and query engine, combined with DuckDB, an in-process database, to revolutionize your data analytics experience. By seamlessly integrating these cutting-edge technologies, you can unlock cost savings, scalability, and accessibility, making analytics a breeze for end users.

During this informative video, you will:
- Explore the capabilities of Dremio and understand how it enables running big data workloads affordably and at scale on your data lakehouse.
- Learn about DuckDB and its ability to vastly expand the possibilities of what can be achieved on your local hardware, enhancing performance and efficiency.
- Discover how the synergy between Dremio and DuckDB can deliver unparalleled benefits to your data analytics workflows.
- Gain insights into how these technologies complement each other, amplifying their individual strengths and providing a comprehensive solution for your analytical needs.
- Dive into practical examples and use cases to see how Dremio and DuckDB can be easily leveraged in Python, empowering you to extract valuable insights from your data effortlessly.

You will acquire a deeper understanding of Dremio and DuckDB, their unique features, and how they work together harmoniously. You'll be equipped with the knowledge to drive cost-effective analytics, optimize performance, and make data-driven decisions with confidence.

Unlock the Potential of Data Analytics with Dremio and DuckDB

With close to exabyte-scale data at Pinterest and evolving business needs, the ability to perform row-level data deletions efficiently on petabytes of data is important. This talk shares how we utilize Apache Iceberg to achieve this goal at Pinterest. We will discuss challenges specific to row-level deletion, solutions we considered, and their trade-offs. Furthermore, we will share some bottlenecks that row-level data deletions run into and the optimizations we added to resolve them. Given how important data deletion requirements are in the current world, we hope that the learnings and solutions shared in this session will help you save money for your respective businesses while improving reliability.

Scaling Row Level Deletions at Pinterest

As data volumes grow - and more users across your organization want access to data to accelerate business decision-making - managing data governance is more important than ever.  What this video in how to simplify data governance for analytics, and deliver data governance at scale with Dremio.

You will learn:
- Data governance on the data lakehouse
- How to balance data access and control to accelerate analytics
- What does good data governance look like?
- How Dremio supports simplified data governance

Simplify data governance at scale across all your data

More organizations than ever before are adopting data lakes to drive business outcomes and innovate faster. As data lakes grow in size and scope, data lake architectures have evolved to balance agility, innovation and governance. Amazon Web Services (AWS) is a pioneer in cloud data lakes, to the point where Amazon S3 is now the de facto storage for data lakes. In this session, Rajesh Sampath, General Manager for Amazon S3 API Experience, and Rahim Bhojani, Dremio’s SVP and Head of Engineering, discuss the evolution of data lakes, capabilities required to build a modern data lake architecture, emerging trends, and how organizations turn data into strategic assets.

Lakes and Lakehouses: The Evolution of Analytics in the Cloud with AWS

The new lakehouse architectural design pattern provides many technical benefits like ACID support, time travel for machine learning, and better query performance. Apache Iceberg implements this pattern and provides the flexibility to further enhance based on real-world needs. Using Apache Iceberg table format requires special vacuuming like snapshot expire and orphan removal for data governance and  metadata and data compaction for efficient and fast access of data. 

This video will discuss how to handle these table operations at very large scale by keeping cost in mind without compromising on data engineering, ML, analytical, and BI use cases. Automating these operations makes life easier for engineers leveraging the platforms without having to worry about how Iceberg internals work. We will share lessons learned for optimizing the streaming and batch data sets in a cost-effective and efficient way.

Lakehouse: Smart Iceberg Table Optimizer

In this video, we will present how Dremio Data Lakehouse came to the rescue when a use case from the electricity retail domain with a large scale machine learning problem around power consumption forecasting imposed some significant challenges on the data engineering and data science teams in Shell.

We will describe how we:  
1.) addressed the large variety of data sources that needed to be ingested, processed and served to the Data Science team 
2.) streamlined some of the Data Science workflows around data exploration, feature engineering and model testing 
3.) operationalized and scaled ML training and inferencing

Modern Data Lakehouse at Shell

This panel will discuss the practical considerations of data mesh, tips for addressing the people and cultural aspects of data mesh, and real-world lessons learned when implementing a data mesh or data mesh-like approach. We will hear from experts and people with real-world experience implementing data mesh, including the creator of Data Mesh herself, Zhamak Dehghani. We will discuss topics such as the challenges faced during the implementation of data mesh, balancing data mesh with existing enterprise architectures and solutions, success metrics when implementing data mesh, and strategies for overcoming the organizational resistance to change. Join us to learn more about data mesh and how to implement it successfully in your organization.

Data Mesh in Practice Panel

Everything was going great: your data was in your data lake, queries were fast, and the SREs were happy. But then things started to slow down. Queries took longer, even specific queries which used to be fast now take a long time. The culprit? Small and unorganized files. The solution? Apache Iceberg’s RewriteDatafile action. This talk will dive into how RewriteDataFiles can 1) right-size your files, merging small files and splitting large ones, ensuring that no time is wasted in query planning or in opening files; and 2) reorganize the data within your files, supporting hierarchal sort and multidimensional z-ordering algorithms, enabling you to make sure your data is optimally set out for your queries. With these two capabilities, any table can be kept at peak performance regardless of ingestion patterns and table size.

Managing Data Files In Apache Iceberg

Scaling data governance practices while continuing to provide efficient access to the data lake is a challenge all businesses face. No one knows this better than Microsoft, whose exabyte-scale data lake platform supports analytics that empower multiple billion-dollar lines of business. Join this session to hear Mike Flasko, VP & GM, Data Governance & Privacy at Microsoft, chat with Read Maloney, CMO of Dremio, about their journey and lessons learned, as they built one of the largest governed data lake platforms in the world.

Data Governance at Scale with Microsoft

Data and Analytics organizations have worked to balance improving access and self-service for the business with achieving security and governance at scale. Data lakehouses are ideally suited to help organizations provide both agility and governance. Deepika Duggirala, SVP Global Technology Platforms, at TransUnion and Tamas Kerekjarto, Head of Engineering, Renewables and Energy Solutions at Shell, will share their journeys to deliver governed self-service.

Apache Iceberg development and adoption accelerated significantly this year, enabling modern data lakes to deliver data warehouse functionality and become data lakehouses. With Apache Iceberg, the industry has consolidated around a vendor-agnostic table format, and innovation from tech companies (Apple, Netflix, etc.) and service providers (AWS, Dremio, GCP, Snowflake, etc.) is creating a world in which data and compute are independent. In this new world, companies can enjoy advancements in data processing, thanks to engine freedom, and data management, thanks to new paradigms such as Data-as-Code. Tomer Shiran, CPO and Founder of Dremio, will deliver Subsurface’s keynote address.

The Year of the Data Lakehouse

Watch the Dremio developer advocacy and engineering teams for an installment of Apache Iceberg Office Hours. During this time we’ll have a brief Iceberg presentation on table format interoperability, going over the table format migration options, converters, and newer interoperability solutions like onetable and uniform. We’ll go through the capabilities, limitations, and considerations and then have lots of dedicated time for Q&A on the presented topic or any other questions or guidance you’re looking for help on in learning about Apache Iceberg or architecting your data lakehouse around Apache Iceberg.

We will cover topics:
- Format Interop
- Using Lakehouse Engines to Unite Table Formats
- Using Onehouse's Onetable Technology
- Using Delta Lake 3.0 UniFormat
- Consideration - Consistency
- Consideration - Vendor Agnosticism
- Consideration - Flexibility

Apache Iceberg Office Hours

Lakehouse, Data Mesh und Cloud - Erfahrungen aus vier verschiedenen Richtungen,
wo laufen die Meinungen zusammen und wo auseinander?
Lakehouse, Data Mesh und Cloud - sind diese Themen nur Trends oder ist es der logische nächste Schritt?
Vier Spezialisten diskutieren über Erfahrungen und Projekte aus dem Data-Umfeld.
Sind die Konzepte bereits in der Praxis einsetzbar und verdrängen bestehende Installation? Wo sind Mehrwerte für Kunden? Sind diese Konzepte bereit vernünftig tragfähig?
Wenn Kunden auf neue Techniken wie Data Mesh und Lakehouse setzen, wird es ein Erfolg und woran wird dieser dann gemessen?

Eine zweite Meinung inklusive!

The Data Revolution: Lakehouse, Data Mesh and Cloud – eine Meinung

Das Data Lakehouse ist noch jung, aber es gibt bereits die Version 2 mit
Dremio Artic und Dremio Sonar. Ein Ausblick auf die nächste Generation.
Dremio Artic und Dremio Sonar liefern in der Cloud den Komfort eines
Data Lakehouses mit dem Luxus eines Datacatalog. Offen und ohne
Vendor Login auf dem Lake der Wahl. Die Lücken, die ein Data
Lakehouse gegenüber einem EDW Enterprise Data Warehouse hatte,
werden mit Artic und Sonar geschlossen.
Die Geschwindigkeit ist erheblich schneller und die Kosten für
Anschaffung und Betrieb sind nur ein Bruchteil eines EDW.
Mit Data-as-Code wird ein Git-like-Feeling im Bereich Data neue
Maßstäbe setzen. Von Daten bis zum neuen Produkt wird nicht mehr
Monate dauern, sondern nur noch Tage.

Diese Webinar-Serie richtet sich an alle, die Antworten auf folgende brennende Fragen suchen:
- Data-as-code
- Data Catalog on top a Datalake
- Git for Data - like

Next Generation Data Lakehouse 2.0 mit Arctic und Sonar

Maersk is a global leader in container shipping, logistics, and energy. With an extensive network of offices in 116 countries, over 900 vessels, hundreds of warehouses, and a modern fleet of aircraft. Maersk provides comprehensive shipping services across the globe with commitments to achieve decarbonization and reach net-zero emissions.

Watch the fireside chat with Mark Sear, Director of Data Analytics and AI/ML at Maersk, and Tomer Shiran, founder and chief product officer at Dremio, as they talk about Maersk’s journey in building a next-generation data platform for solution development using Dremio’s open data lakehouse and GenerativeAI. 

In this video, you will learn:
- Common data platform challenges in the shipping and logistics industry
- How Maersk uses Dremio’s open data lakehouse to empower their developers and end users to deliver agile and cost-effective solutions
- A live demo of GenerativeAI

How Maersk is Building A Next Gen Data Lakehouse with Dremio & Gen AI

Das Data Lakehouse ist die Zukunft. Das Enterprise Data Warehouse
stirbt mit dem Datalake und der Cloud. Rein aus Kostengründen hat es keine Zukunft mehr.
Das Lakehouse wird in kurzer Zeit das EDW verdrängen. Mit dremio sin ddie Möglichkeiten so einfach und schnell wie nie zuvor.
Im Video wird gezeigt, wo die Probleme mit Data Lake Produktensind und welche Vorteile von Lakehouse Produkte haben.
Verpasse nicht, wie dremio den Data Lake verändert und in ein Lakehouse verwandelt.

Diese Webinar-Serie richtet sich an alle, die Antworten auf folgende brennende Fragen suchen:
- Lakehouse vs. Data lake
- Speed does matter
- Was ist einfache und schneller - Lakehouse oder Data Lake

Lakehouse vs. Warehouse

Aufbereiten der Daten für das Training und die Entwicklung der Modelle.  
Um so einfacher und schneller diese Vorbereitung erledigt werden kann, umso schneller sind die Ergebnisse verfügbar.  RDMS System können oft die Datenmengen nicht mehr bewältigen und das Umformen oder Ändern der Daten, dauert entsprechend lange oder ist nicht mehr realisierbar.  Ein Semantischer Layer braucht diese Arbeitsschritte nicht und auf einen Data Lake gibt es keine Mengenbegrenzung mehr.  ML-Plattformen mit Dremio als Datenlieferant ist die Antwort für diese Aufgabe.

Diese Webinar-Serie richtet sich an alle, die Antworten auf folgende brennende Fragen suchen:
- ML ist überall im Einsatz. Warum ist es mit einem Data Lake am besten?
- Welche Vorteile bringt es dem Anwender
- Woher kommt dieses riesige Einsparpotenzial

So verbessern Sie Ihre ML-Plattform mit Dremio

Enterprise data platform teams are often frustrated with inheriting infrastructures that impact their ability to access reliable and high-quality data. Economic factors are forcing  organizations to become more capital-efficient, and teams are now building their applications on the data lakehouse with Dremio.

In this video, you will learn about:
- The importance of a simplified, open data architecture
- Simplifying your on-prem and cloud analytical workloads with Dremio
- Use cases on the data lakehouse

Getting Started with Dremio’s Data Lakehouse

Data Lake

Data Warehousing

Data Analytics

Data Architecture

Open Source

Data Governance

Advanced Analytics

Big Data Analytics

Cloud Analytics

Data Management

Welcome to the virtualization community on BrightTALK! Whether it affects servers, storage, networks, desktops or other parts of the data center, virtualization provides real benefits by reducing the resources needed for your
infrastructure and creating software-defined data center components. However, it can also complicate your infrastructure. Join this active community to learn best
practices for avoiding virtual machine sprawl and other common virtualization pitfalls as well as how you can make the most of your virtualization environment.

Virtualization

Are you an IT service management professional interested in developing your knowledge and improving your job performance? Join the IT service management community to access the latest updates from industry experts. Learn and share insights related to IT service management (ITSM) including topics such as the service desk, service catalog, problem and incident management, ITIL v4 and more. Engage with industry experts on current best practices and participate in active discussions that address the needs and challenges of the ITSM community.

IT Service Management

Cloud computing has exploded over the past few years, delivering a previously unimagined level of workplace mobility and flexibility. The cloud computing community on BrightTALK is made up of thousands of engaged professionals learning from the latest cloud computing research and resources. Join the community to expand your cloud computing knowledge and have your questions answered in live sessions with industry experts and vendor representatives.

Cloud Computing

Increasing expectations for good governance, effective risk management and complex demands for corporate compliance are presenting a growing challenge for organizations of all sizes. Join industry thought leaders as they provide you with practical advice on how to implement successful risk and compliance management strategies across your organization. Browse risk management resources in the form of interactive webinars and videos and ask questions of expert GRC professionals.

IT Governance, Risk and Compliance

The data center management community focuses on the holistic management and optimization of the data center. From technologies such as virtualization and cloud computing to data center design, colocation, energy efficiency and monitoring, the BrightTALK data center management community provides the most up-to-date and engaging content from industry experts to better your infrastructure and operations. Engage with a community of your peers and industry experts by asking questions, rating presentations and participating in polls during webinars, all while you gain insight that will help you transform your infrastructure into a next generation data center.

Data Center Management

The application development community features top thought leadership focusing on optimal practices in software development, SDLC methodology, mobile app development and application development platforms and tools. Join top software engineers and coders as they cover emerging trends in everything from enterprise app development to developing for mobile platforms such as Android and iOS.

Application Development

Practicing business intelligence allows your company to transform raw data into sets of insights for targeted business growth. The business intelligence and analytics community on BrightTALK is made up of thousands of data scientists, database administrators, business analysts and other data professionals. Find relevant webinars and videos on business analytics, business intelligence, data analysis and more presented by recognized thought leaders. Join the conversation by participating in live webinars and round table discussions.

Business Intelligence and Analytics

The IT project management community on BrightTALK includes thousands of IT project and portfolio management professionals. Find relevant webinars and videos on agile methodologies, scrum strategy, project management processes and more. Attend live webinars or view on demand content presented by recognized thought leaders in the IT project management industry.

IT Project Management

Network infrastructure professionals understand that a reliable and secure infrastructure is crucial to enabling business execution. Join the network infrastructure community to interact with thousands of IT professionals. Browse hundreds of on-demand and live webinars and videos to learn about the latest trends in network computing, SDN, WAN optimization and more.

Network Infrastructure

Welcome to the big data and data management community on BrightTALK. Join thousands of data quality engineers, data scientists, database administrators and other professionals to find more information about the hottest topics affecting your data. Subscribe now to learn about efficiently storing, optimizing a complex infrastructure, developing governing policies, ensuring data quality and analyzing data to make better informed decisions. Join the conversation by watching live and on-demand webinars and take the opportunity to interact with top experts and thought leaders in the field.

Big Data and Data Management

As an IT professional, many of the problems you face are multifaceted, complex and don’t lend themselves to simple solutions. The information technology community features useful and free information technology resources. Join to browse thousands of videos and webinars on ITIL best practices, IT security strategy and more presented by leading CTOs, CIOs and other technology experts.

Getting Started with Dremio’s Data Lakehouse

Presented by

About this talk

More from this channel