CI/CD on Databricks using Azure Devops
<h1>Introduction</h1>
<p>This blog post explains how to configure and build end to end CI/CD pipeline solutions on Databricks using Azure devops and best practices to deploy libraries in workspace using azure service principal in CI/CD pipeline for security aspects.</p>
<p>A Typical Azure Databricks pipeline includes following steps.</p>
<p>Continuous integration</p>
<ol>
<li>Develop code using databricks notebook or external IDE.</li>
<li>Build libraries.</li>
<li>Release — generate a release artifact.</li>
</ol>
<p>Continuous deployment</p>
<ol>
<li>Deploy libraries or notebooks.</li>
<li>Run automated tests.</li>
<li>Programmatically schedule data engineering and analytics workflows.</li>
</ol>
<p>Suppose you have developed your code using IDE or notebooks and committed to the Azure Git repository for which you would like to build a library whl or JAR file using DevOps principles.</p>
<p>Consider the following screenshot as your committed code for which you would like to build a library and pipeline.py as main python notebook which you would like to schedule for running analytics workflows.</p>
<p><a href="https://medium.com/@yatin.kumar/ci-cd-on-databricks-using-azure-devops-3f8a4aabebaa"><strong>Learn More</strong></a></p>