Spot VMs make use of unused compute capacity in Azure, allowing you to save up to 90% on compute costs for workloads, with minimal impact on workload completion times. Azure Databricks automatically handles eviction/termination of Spot VMs by replacing them with new on-demand VMs at regular pricing and restarting any tasks on survivor worker nodes.
Azure Data Factory (ADF) is a popular service used to orchestrate data pipelines. But how can Spot VMs be used with ADF?
It’s possible to create a pool of spot VMs in Azure Databricks and create a linked service in ADF to use an existing instance pool: