Dynamic Row Level Filtering and Column Level Masking in Azure Databricks

<h1>Background</h1> In modern data platforms and analytics architectures, the discussion surrounding Data Security is essential. Ensuring the protection of data is paramount, and within any data architecture, the primary objective is to grant access to data only to those individuals who possess the appropriate authorization. Superusers, on the other hand, are granted access to all data. To achieve this, we need to implement the row-level filtering based on a user’s permission on the tables. If we have requirement where we would like to hide some sensitive column for the users, we need to implement the column level masking. In the past, Azure Databricks utilized Dynamic Views and Hive functions for row-level filtering and column masking. You can get more details of this older method in a <a href="https://www.linkedin.com/pulse/gdpr-compliance-using-azure-databricks-samarendra-panda/" rel="noopener ugc nofollow" target="_blank">blog post</a> I authored a couple of years ago. Although it proved effective, it involved some additional steps. Notably, the implementation of row-level filtering and column masking occurred within the Dynamic View, creating an extra object, and requiring extra maintenance effort. Furthermore, it lacked security control at the base table level, which was a notable limitation. Azure Databricks has recently released the <a href="https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/row-and-column-filters" rel="noopener ugc nofollow" target="_blank">feature</a> to perform row-level filtering and column-level masking directly on the base table. This blog post will discuss these features in-depth. <a href="https://learn-it-all.medium.com/dynamic-row-level-filtering-and-column-level-masking-in-azure-databricks-24c0c886e428">Website</a>