An Introduction to Delta Lake and The Lakehouse

Once upon a time, there was a data warehouse, and it lived happily as a set of tables within our relational database management system (RDMS) called Microsoft SQL Server. The data warehouse had three children known as extract, transform, and load. One day a blue/azure coloured cloud appeared overhead, and it started to rain. The data warehouse got wet and was never the same again! Or was it? Spoiler alert, the data warehouse is the same, still happy, and well, it just evolved and moved from its RDMS home to a new home in the cloud. The end!

In this session, we'll look at the evolution of the data warehouse and understand how we can now deliver the same data engineering concepts for our solutions on the Microsoft cloud platform using the open-source Delta.io standard, in Azure and Fabric. We'll introduce the standard (originally developed by Databricks) and then explore the implications it has for our next-generation cloud data warehouse.

The original data warehouse set of tables remain, but now they are delivered using the cloud-native Delta Lake technology with distributed storage/compute as standard. Delta.io gives us those much-needed ACID properties over our data lakes meaning our data warehouse understanding can move to the cloud and is made easier within Azure. The data warehouse just grew up and became a Delta Lake-House.

• Understand the evolution and transition of traditional data warehouses to cloud-based solutions using Delta.io on the Microsoft cloud platform. • Explore the open-source Delta.io standard, initially developed by Databricks, and its implications for next-generation cloud data warehouses. • Learn how Delta Lake technology in Azure and Fabric provides ACID properties and distributed storage/compute to enhance cloud-native data warehousing.