Accessing Workspace Files with Shared Clusters (UC) in Databricks

<h1>The Issue</h1> <p>If you are trying to access workspace files or files located in a repository folder (as described&nbsp;<a href="https://learn.microsoft.com/en-us/azure/databricks/files/workspace-interact" rel="noopener ugc nofollow" target="_blank">here</a>) from a shared cluster in Databricks, you might run into the following error:</p> <blockquote> <p>java.lang.SecurityException: User does not have permission SELECT on any file</p> </blockquote> <p>The underlying problem has to do with the&nbsp;<a href="https://docs.databricks.com/en/clusters/configure.html#shared-access-mode-limitations" rel="noopener ugc nofollow" target="_blank">restrictions&nbsp;</a>you are facing when using shared clusters in Databricks.</p> <p>If you want to access Unity Catalog with a cluster the only two options regarding &ldquo;Access Mode&rdquo; currently are &quot;Shared&quot; or &ldquo;Single User&rdquo; though. If the latter is out of the question, as often happens in projects that i am participating in, then you will not be able to access workspace files from the only cluster option that is left.</p> <h1>The Solution</h1> <p>One way of dealing with this problem is to use&nbsp;<a href="https://databricks-sdk-py.readthedocs.io/en/latest/" rel="noopener ugc nofollow" target="_blank">Databricks Python SDK</a>.</p> <h1>(1) Install the SDK on your Cluster</h1> <p>First you need to install the SDK on your current cluster. There are numerous ways to install libraries on your cluster, quick one is to add this line of code in a notebook cell of its own:</p> <p><a href="https://medium.totter.pw/accessing-workspace-files-with-shared-clusters-uc-in-databricks-d4f0f08e0188"><strong>Website</strong></a></p>