GCP Dataflow Flex Template Pipeline ??? Part 2 (Local Testing)

Welcome to the Part 2 of building a data pipeline using the GCP Dataflow service. In this part, we will develop & test our pipeline locally. So, lets jump right in. We will focus on below code for testing and developing the Dataflow pipeline:

Local Testing via Jupyter Notebook: https://github.com/amandeepsaluja/gcp-dataflow-pipeline/blob/master/01-flex-template/local_testing/testing.ipynb

Before we start, make sure you have created a new virtual environment and installed the below required packages as mentioned in requirements.txt

Visit Now

Tags: Flex template