DevOps for Databricks

2022

TL; DR

Applying DevOps to Databricks can be a daunting task. In this talk this will be broken down into bite size chunks. Common DevOps subject areas will be covered, including CI/CD (Continuous Integration/Continuous Deployment) and IAC (Infrastructure as Code). We will explore different tooling available to achieve our goals and how they compare

Session Details

Applying DevOps to Databricks can be a daunting task. In this talk this will be broken down into bite size chunks. Common DevOps subject areas will be covered, including CI/CD (Continuous Integration/Continuous Deployment), IAC (Infrastructure as Code) and Build Agents. We will explore how to apply DevOps to Databricks, primarily using Azure DevOps tooling, but we will also explore the alternatives, including Github Actions. As a lot of Spark/Databricks users are Python users, we will look at the Databricks Rest API (using Python) to perform CI/CD tasks. For IAC we will primarily look at Terraform, but also explore other options, including ARM templates, Azure Bicep and Pulumi As data professionals come with a variety of different backgrounds and skill sets, this talk will focus on providing options, and live demos, that demonstrate ways of achieving a DevOps solution that can be understood and maintained by everyone

3 things you'll get out of this session

Anna-Maria Wykes's other proposed sessions for 2026

What we Learned Migrating a Financial Giant from Hudi to Delta (and Why Iceberg was in the Mix) - 2026

From “Who Wrote This ETL?” to Databricks, Claude Saves the Day via Microsoft Foundry - 2026

Getting Started with Claude in Microsoft Foundry - 2026

MCP Unleashed From “Huh?” to “Heck Yeah!” Building Smarter AI Knowledge Bases - 2026

Migrating the Mammoth - 2026

Anna-Maria Wykes's previous sessions

How to Run Code Clubs for Neurodiverse Children

Code Clubs offer an amazing opportunity to introduce our next generation to coding, with simple brightly colored drag-and-drop tooling to get them started, we are successfully inspiring many to join the tech industry. In this session I want to talk you through my journey setting up a Code Club for neurodiverse children, what I found worked, and what doesn’t. I hope that from this session you will be inspired to follow the same path I have, using your amazing tech experience to empower some of the most vulnerable children, enabling them to become inspired not just by coding, but the tech industry itself.

Introduction to the wonders of Azure DevOps

Azure DevOps is the leading deployment tool for build and release solutions end to end. It helps you plan your Agile project, manages Git code, and deploys solutions using Continuous Integration (CI) and Continuous Deployment (CD) pipelines. In this session we will cover some of the core components of Azure DevOps and show you how to implement a secure deployment pipeline, using unit tests and gating with your CI builds and CD releases.

Automate the deployment of Databricks components using Terraform

Introduction into Terraform, Databricks provider and steps required to build an automated solution to provision Databricks workspace and resources into Azure cloud platform using Terraform.

So you want to be a Data Engineer?

An introduction to becoming a Data Engineer, Anna, Mikey and Ust will introduce the technology stack, tools and development skills needed for data engineering and show you how and where to go to learn them. We'll also show you how the skills you already have can kickstart your journey to becoming a Data Engineer.

Scala for Big Data the Big Picture

An opportunity to explore Scala, and why it is truly a “Data Engineers language”. Using Azure Functions, Data Factory, Azure Data Lake Gen2 and Databricks the basics will be explored, followed by real world examples