Source data that is already relational may go directly into the data warehouse, using an ETL process, skipping the data lake.ĭata lake stores are often used in event streaming or IoT scenarios, because they can persist large amounts of relational and nonrelational data without transformation or schema definition. Typically this transformation uses an ELT (extract-load-transform) pipeline, where the data is ingested and transformed in place. With this approach, the raw data is ingested into the data lake and then transformed into a structured queryable format. Typical uses for a data lake include data exploration, data analytics, and machine learning.Ī data lake can also act as the data source for a data warehouse. Data lake processing involves one or more processing engines built with these goals in mind, and can operate on data stored in a data lake at scale. Data lake storage is designed for fault-tolerance, infinite scalability, and high-throughput ingestion of data with varying shapes and sizes.
This approach differs from a traditional data warehouse, which transforms and processes the data at the time of ingestion. The idea with a data lake is to store everything in its original, untransformed state. The data typically comes from multiple heterogeneous sources, and may be structured, semi-structured, or unstructured. Data lake stores are optimized for scaling to terabytes and petabytes of data. And I’ll discuss Microsoft version of the data mesh.A data lake is a storage repository that holds a large amount of data in its native, raw format. I’ll also include use cases so you can see what approach will work best for your big data needs. They all may sound great in theory, but I’ll dig into the concerns you need to be aware of before taking the plunge. What do all these terms mean and how do they compare to a modern data warehouse? In this session I’ll cover all of them in detail and compare the pros and cons of each. So many buzzwords of late: Data Lakehouse, Data Mesh, and Data Fabric. Presentation abstract: Data Lakehouse, Data Mesh, and Data Fabric (the alphabet soup of data architectures) Look for a blog post of mine in a couple months that will cover Microsoft’s vision and technology solution of a data mesh.
#Data lakehouse vs data lake series
Check out the SaxonGlobal Data Story Podcast Series that covers all the architectures as well as common data models in four episodes.A 30-minute video at the Hevo Cloud Data Warehousing Summit: Why Modern Enterprises Need a Cloud Data Warehouse.These are my blog posts on the subject matter: Data Lakehouse defined, Data Fabric defined, Data Mesh defined, Data Mesh: Centralized vs decentralized data architecture, Data Mesh: Centralized ownership vs decentralized ownership.I did a 20-minute video explaining the Modern Data Warehouse that you can view here.The Data Lakehouse, Data Mesh, and Data Fabric presentation slides can be found here.I will also be presenting it at: SQLBits on 3/10/22 ( info) and Data Summit 2022 on 5/17/22 ( info). Videos of me presenting on “Data Lakehouse, Data Mesh, and Data Fabric (the alphabet soup of data architectures)” can be found in three different lengths: DataMinutes ( recording – 10 minutes), Data Agility Day ( recording – 30 minutes), and India Azure Community Conference 2021 ( recording – 1 hour).Hopefully these presentations, blog posts, and videos can help clarify all these data architectures for you: There is no clear definition of all these data architectures, and I have created a presentation using my own take that I have been presenting frequently internally at Microsoft and externally to customers and at conferences. There certainly has been a lot of discussion lately on the topic of Data Lakehouse, Data Mesh, and Data Fabric, and how they compare to the Modern Data Warehouse. That being said: the views and opinions in this blog are mine and not that of Microsoft). Contact your Microsoft account executive for more info. (NOTE: I have returned to Microsoft and am working as a Solution Architect in Microsoft Industry Solutions, formally known as Microsoft Consulting Services (MCS), where I help customers build solutions on Azure.
0 kommentar(er)
