This quick blog shows an example architecture of Databricks deployed using Customer Managed VPC, with private MSK cluster in the same VPC. By providing your own VPC to host Databricks and MSK, you can customize networking objects and keep MSK traffic private. After building this platform, we use a DLT (Delta Live Table) pipeline to consume from Kafka topic and persist data to tables.
Step 1: Building Databricks-MSK platform

Architecture of Databricks-MSK
Here are the narratives on the above architecture: