Introduction
This blog post explains how to configure and build end to end CI/CD pipeline solutions on Databricks using Azure devops and best practices to deploy libraries in workspace using azure service principal in CI/CD pipeline for security aspects.
A Typical Azure Databricks pipeline includes following steps.
Continuous integration
- Develop code using databricks notebook or external IDE.
- Build libraries.
- Release — generate a release artifact.
Continuous deployment
- Deploy libraries or notebooks.
- Run automated tests.
- Programmatically schedule data engineering and analytics workflows.
Suppose you have developed your code using IDE or notebooks and committed to the Azure Git repository for which you would like to build a library whl or JAR file using DevOps principles.
Consider the following screenshot as your committed code for which you would like to build a library and pipeline.py as main python notebook which you would like to schedule for running analytics workflows.