In this article, let's quickly discuss (hack) how to monitor UC+DLT pipelines owned by the Service Principals.
As per Databricks Documentation, it can be done using a Table Valued Function, event_log.
Start a Shared Compute Cluster or DB SQL Warehouse and execute the following query. Pass the pipeline ID as a parameter to the event_log function.
%sql
SELECT * FROM event_log("41c73963-6d78-451a-94bc-f247e36a12fd")
At a higher level, this sounds simple & straightforward. In reality, there is an underlying problem that Databricks is currently working on.
The problem is the select statement can be executed only by the Pipeline Owner. No one else can access the pipeline event log.
In a typical production environment, the Service Principal will own all the pipelines. Because of this, no one, including the Admins, can query the event_logs TVF.