Using Spot Instances with Azure Databricks and Azure Data Factory

<a href="https://www.databricks.com/blog/2021/05/25/leverage-unused-compute-capacity-for-data-ai-with-azure-spot-instances-and-azure-databricks.html" rel="noopener ugc nofollow" target="_blank">Spot VMs</a> make use of unused compute capacity in Azure, allowing you to save up to 90% on compute costs for workloads, with minimal impact on workload completion times. Azure Databricks automatically handles eviction/termination of Spot VMs by replacing them with new on-demand VMs at regular pricing and restarting any tasks on survivor worker nodes. Azure Data Factory (ADF) is a popular service used to orchestrate data pipelines. But how can Spot VMs be used with ADF? It’s possible to <a href="https://learn.microsoft.com/en-us/azure/databricks/clusters/pools#spot-instances" rel="noopener ugc nofollow" target="_blank">create a pool of spot VMs in Azure Databricks</a> and create a linked service in ADF to use an existing instance pool: <a href="https://medium.com/@xsharpminor/using-spot-instances-with-azure-databricks-and-azure-data-factory-aa4d47a063cd">Visit Now</a>