Quick Blog: Databricks DLT consuming from AWS MSK Kafka Topics
<p>This quick blog shows an example architecture of Databricks deployed using <a href="https://docs.databricks.com/en/administration-guide/cloud-configurations/aws/customer-managed-vpc.html" rel="noopener ugc nofollow" target="_blank"><em>Customer Managed VPC</em></a>, with <em>private MSK cluster</em> in the same VPC. By providing your own VPC to host Databricks and MSK, you can customize networking objects and keep MSK traffic private. After building this platform, we use a DLT (<a href="https://www.databricks.com/product/delta-live-tables" rel="noopener ugc nofollow" target="_blank"><em>Delta Live Table</em></a>) pipeline to consume from Kafka topic and persist data to tables.</p>
<h2>Step 1: Building Databricks-MSK platform</h2>
<p><img alt="" src="https://miro.medium.com/v2/resize:fit:700/1*zlKpYnai9pOKQevNUMAHXg.jpeg" style="height:436px; width:700px" /></p>
<p>Architecture of Databricks-MSK</p>
<p>Here are the narratives on the above architecture:</p>
<p><a href="https://medium.com/@hwangdb/quick-blog-databricks-dlt-consuming-from-aws-msk-kafka-topics-4e098f905ce0"><strong>Click Here</strong></a></p>