Simplifying Kafka Cluster Deployment: Step-by-Step Guide with Amazon MSK and Terraform
<p>This is the second article of the series “<em>Building a managed streaming data pipeline</em>”. If you haven’t read the first one, I leave the link here for you to have a look:</p>
<h2><a href="https://medium.com/datamindedbe/unleash-the-spark-create-an-amazon-emr-serverless-cluster-with-terraform-and-run-pyspark-jobs-e9882c4095b9?source=post_page-----a3643eaf903a--------------------------------" rel="noopener follow" target="_blank">Unleash the Spark: Create an Amazon EMR Serverless Cluster with Terraform and run PySpark jobs</a></h2>
<h3><a href="https://medium.com/datamindedbe/unleash-the-spark-create-an-amazon-emr-serverless-cluster-with-terraform-and-run-pyspark-jobs-e9882c4095b9?source=post_page-----a3643eaf903a--------------------------------" rel="noopener follow" target="_blank">Ignite Your Data Revolution and Harness the Power of Amazon EMR Serverless</a></h3>
<p><a href="https://medium.com/datamindedbe/unleash-the-spark-create-an-amazon-emr-serverless-cluster-with-terraform-and-run-pyspark-jobs-e9882c4095b9?source=post_page-----a3643eaf903a--------------------------------" rel="noopener follow" target="_blank">medium.com</a></p>
<p>In my first article I showed you how to use <a href="https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/emr-serverless.html" rel="noopener ugc nofollow" target="_blank">Amazon EMR Serverless</a> to run your <a href="https://spark.apache.org/docs/latest/api/python/#:~:text=PySpark%20is%20the%20Python%20API,for%20interactively%20analyzing%20your%20data." rel="noopener ugc nofollow" target="_blank">PySpark</a> job managing the infrastructure in <a href="https://www.terraform.io/" rel="noopener ugc nofollow" target="_blank">Terraform</a>. In this case, I’ll show you how to use <a href="https://aws.amazon.com/msk/" rel="noopener ugc nofollow" target="_blank">Amazon MSK</a> to deploy your Kafka Cluster using also Terraform. Finally, the next article will be about putting all the pieces together and building the <strong>streaming data pipeline</strong> using both technologies.</p>
<p>Basic knowledge and some experience with <a href="https://kafka.apache.org/" rel="noopener ugc nofollow" target="_blank">Apache Kafka</a> is recommended to fully understand all the configurations provided in the Terraform files.</p>
<p><a href="https://medium.com/datamindedbe/simplifying-kafka-cluster-deployment-step-by-step-guide-with-amazon-msk-and-terraform-a3643eaf903a"><strong>Click Here</strong></a></p>