Databricks: Monitor the UC+DLT pipelines
<p>In this article, let's quickly discuss (hack) how to monitor UC+DLT pipelines owned by the <strong>Service Principals</strong>.</p>
<p>As per Databricks <a href="https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/observability" rel="noopener ugc nofollow" target="_blank">Documentation</a>, it can be done using a Table Valued Function, <strong>event_log</strong>.</p>
<p>Start a Shared Compute Cluster or DB SQL Warehouse and execute the following query. Pass the pipeline ID as a parameter to the event_log function.</p>
<pre>
%sql
SELECT * FROM event_log("41c73963-6d78-451a-94bc-f247e36a12fd") </pre>
<p>At a higher level, this sounds simple & straightforward. In reality, there is an underlying problem that Databricks is currently working on.</p>
<blockquote>
<p>The problem is the select statement can be executed <strong>only by the Pipeline Owner</strong>. No one else can access the pipeline event log.</p>
</blockquote>
<p>In a typical production environment, the Service Principal will own all the pipelines. Because of this, no one, including the Admins, can query the <strong>event_logs</strong> TVF.</p>
<p><a href="https://ganeshchandrasekaran.com/databricks-monitor-the-uc-dlt-pipelines-e1511d733af5"><strong>Read More</strong></a></p>