Azure Data Factory & Databricks. Migration of init scripts to Databricks from DBFS to Workspace 2023/2024

In this article, we will discuss how to make a simple data loading process using Microsoft Azure Data Factory and Databricks in 2023/2024. In the second part, we will analyze the migration of init scripts from DBFS to Workspace in connection with the new update from Databricks. If you don’t do it, then don’t be surprised that at the end of 2023, your ADF pipelines and clusters in Databricks will start to fail. On December 1, 2023, <a href="https://kb.databricks.com/cluster-named-init-script-migration-notebook" rel="noopener ugc nofollow" target="_blank">Databricks will disable cluster name init scripts</a> for all workspaces. This init script type was previously deprecated and will no longer be used. <img alt="" src="https://miro.medium.com/v2/resize:fit:770/1*O4R2FJTc6du6tLVascMM0A.jpeg" style="height:406px; width:700px" /> <h1>Azure Data Factory и Databricks</h1> Many projects that work with data and use Microsoft Azure cloud technologies in Pipelines Azure Data Factory contain the “Databricks Notebook” activity. It is used to launch a notebook in the Azure Databricks service, which performs computational operations on data. To connect these two services, of course, Liked Service is used, where the connection parameters are indicated. <a href="https://medium.com/@protmaks/data-factory-databricks-migrate-init-scripts-in-from-dbfs-to-workspace-2023-2024-bb4a1db22eed">Read More</a>