CI/CD on Databricks using Azure Devops

<h1>Introduction</h1> <p>This blog post explains how to configure and build end to end CI/CD pipeline solutions on Databricks using Azure devops and best practices to deploy libraries in workspace using azure service principal in CI/CD pipeline for security aspects.</p> <p>A Typical Azure Databricks pipeline includes following steps.</p> <p>Continuous integration</p> <ol> <li>Develop code using databricks notebook or external IDE.</li> <li>Build libraries.</li> <li>Release &mdash; generate a release artifact.</li> </ol> <p>Continuous deployment</p> <ol> <li>Deploy libraries or notebooks.</li> <li>Run automated tests.</li> <li>Programmatically schedule data engineering and analytics workflows.</li> </ol> <p>Suppose you have developed your code using IDE or notebooks and committed to the Azure Git repository for which you would like to build a library whl or JAR file using DevOps principles.</p> <p>Consider the following screenshot as your committed code for which you would like to build a library and pipeline.py as main python notebook which you would like to schedule for running analytics workflows.</p> <p><a href="https://medium.com/@yatin.kumar/ci-cd-on-databricks-using-azure-devops-3f8a4aabebaa"><strong>Learn More</strong></a></p>
Tags: DevOps Azure