Managing Databricks: Asset Bundles
<h1>Introduction</h1>
<p>Remember when I mentioned that Databricks was cooking up a process for better <a href="https://medium.com/@matt_weingarten/databricks-q3-roadmap-w2w4-8e6be36fb5cb" rel="noopener">workflow management</a> in their quarterly roadmap call? Well, it appears to finally be here (or in Public Preview at least).</p>
<p><a href="https://www.databricks.com/blog/announcing-public-preview-databricks-assets-bundles-apply-software-development-best-practices" rel="noopener ugc nofollow" target="_blank">Asset bundles</a> are a way to follow proper software development practices when it comes to handling all your Databricks resources. I was naturally ecstatic to see this available and had to give it a spin.</p>
<h1>Getting Started</h1>
<p>I decided to take a crack at creating a job via asset bundles by following the corresponding <a href="https://docs.databricks.com/en/workflows/jobs/how-to/use-bundles-with-jobs.html" rel="noopener ugc nofollow" target="_blank">documentation</a>. I took an existing notebook, put it into a new directory, and then defined the job configuration in the databricks.yml file (interesting that Databricks decided to go with YAML here when their jobs are represented as JSON in the UI). For the job configuration details, I was able to just convert my already-existing job details from JSON to YAML, and then created the asset bundle with one simple command. Sure enough, both my notebook and job were created in Databricks very quickly. Easy!</p>
<p>As I’ve mentioned before, we already have a <a href="https://medium.com/@matt_weingarten/databricks-ci-cd-66b7341345ec" rel="noopener">similar process</a> in place for creating Databricks jobs. In fact, the job configuration is essentially the same since they’re both YAML-based and leverage the API to actually instantiate the jobs. Now let’s get into comparing them.</p>
<p><a href="https://medium.com/@matt_weingarten/managing-databricks-asset-bundles-ee1677413c2f"><strong>Read More</strong></a></p>