GCP Dataflow Flex Template Pipeline — Part 1 (Overview)
<p>In this project, we will be building a data pipeline using the GCP Dataflow service. We will be creating a Flex template in which you can pass API endpoint response, extract keys in the response, and then export it to BigQuery.</p>
<blockquote>
<p><strong>Repository with Full Code:</strong> <a href="https://github.com/amandeepsaluja/gcp-dataflow-pipeline/tree/master/01-flex-template" rel="noopener ugc nofollow" target="_blank">https://github.com/amandeepsaluja/gcp-dataflow-pipeline/tree/master/01-flex-template</a></p>
</blockquote>
<p><strong>Steps in the Project:</strong></p>
<ol>
<li>Create the Pipeline in Python</li>
<li>Create the Docker Image</li>
<li>Setup Terraform configuration files</li>
<li>Build & deploy Docker Image of the Flex template on Artifact Registry (Terraform + GitHub Actions)</li>
<li>Create and Deploy the Dataflow job (GitHub Actions)</li>
</ol>
<p><a href="https://medium.com/@saluja.amandeep/gcp-dataflow-flex-template-pipeline-part-1-overview-b1ae8151e93d"><strong>Click Here</strong></a></p>