Demystifying Spark Profile Optimizations in Microsoft Fabric

Optimizing Spark workloads in Microsoft Fabric can be a complex endeavor, given the array of available configurations and techniques. Terms like V-Order, Z-Order, and considerations for read-heavy versus write-heavy profiles often add to the confusion. In this session, we'll bring clarity to these concepts, offering a structured overview of each optimization strategy and how it impacts performance on the table level of a Delta Lake table

Through practical examples and performance considerations, attendees will gain insights into how to fit these optimizations into their data architecture, whether dealing with read-intensive analytics or write-heavy data ingestion processes. By the end of this talk, you'll be equipped with the knowledge to make informed decisions, turning the complexity of Spark optimizations in Microsoft Fabric into a strategic advantage

- What Spark resource profiles are available in Microsoft Fabric? - Which optimizations are behind these resource profiles? - Which profile should you use for which part of your data architecture (medallion architecture example)?