Building an End-to-End Data Pipeline with Delta Lake and Databricks

Introduction

In this article, we will walk through the process of building a data pipeline using Delta Lake and Databricks. We will use COVID-19 data for the USA, available on Kaggle, as our dataset. This pipeline will demonstrate how to ingest raw data, clean and transform it, and finally visualize it.

If you are new to the concept of Delta Lake, I suggest you start with my previous article: