Databricks: Monitor the UC+DLT pipelines

<p>In this article, let&#39;s quickly discuss (hack) how to monitor UC+DLT pipelines owned by the&nbsp;<strong>Service Principals</strong>.</p> <p>As per Databricks&nbsp;<a href="https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/observability" rel="noopener ugc nofollow" target="_blank">Documentation</a>, it can be done using a Table Valued Function,&nbsp;<strong>event_log</strong>.</p> <p>Start a Shared Compute Cluster or DB SQL Warehouse and execute the following query. Pass the pipeline ID as a parameter to the event_log function.</p> <pre> %sql SELECT * FROM event_log(&quot;41c73963-6d78-451a-94bc-f247e36a12fd&quot;) </pre> <p>At a higher level, this sounds simple &amp; straightforward. In reality, there is an underlying problem that Databricks is currently working on.</p> <blockquote> <p>The problem is the select statement can be executed&nbsp;<strong>only by the Pipeline Owner</strong>. No one else can access the pipeline event log.</p> </blockquote> <p>In a typical production environment, the Service Principal will own all the pipelines. Because of this, no one, including the Admins, can query the&nbsp;<strong>event_logs</strong>&nbsp;TVF.</p> <p><a href="https://ganeshchandrasekaran.com/databricks-monitor-the-uc-dlt-pipelines-e1511d733af5"><strong>Read More</strong></a></p>