Perform “SCD Type 1” Using “MERGE” Operation on Delta Table Using “SPARK SQL” and “PySpark” in Databricks

<h1>What is &ldquo;Slowly Changing Dimension&rdquo;?</h1> <p>The &ldquo;<strong>Slowly Changing Dimension</strong>&rdquo; is &ldquo;<strong>One</strong>&rdquo; of the &ldquo;<strong>Very Important Concept</strong>&rdquo; in &ldquo;<strong>Data Warehousing Solution</strong>&rdquo;.</p> <p><strong>&ldquo;Slowly Changing Dimension</strong>&rdquo; is the &ldquo;<strong>Change</strong>&rdquo; of &ldquo;<strong>Attribute</strong>&rdquo;, or, the &ldquo;<strong>Value</strong>&rdquo; of the &ldquo;<strong>Entities</strong>&rdquo; over a &ldquo;<strong>Period</strong>&rdquo; of &ldquo;<strong>Time</strong>&rdquo;.<br /> Example:</p> <ul> <li><strong>&ldquo;Consider</strong>&rdquo; that there is a &ldquo;<strong>Delta Table</strong>&rdquo;, called &ldquo;<strong>Person</strong>&rdquo;, which has an &ldquo;<strong>Attribute</strong>&rdquo;, called as &ldquo;<strong>Address</strong>&rdquo;. Now, a &ldquo;<strong>Person</strong>&rdquo; can &ldquo;<strong>Stay</strong>&rdquo; in an &ldquo;<strong>Address</strong>&rdquo; now. But, after &ldquo;<strong>One Year</strong>&rdquo;, the &ldquo;<strong>Same Person</strong>&rdquo; can &ldquo;<strong>Change</strong>&rdquo; the &ldquo;<strong>Address</strong>&rdquo;.<br /> As a result, the &ldquo;<strong>Latest Value</strong>&rdquo; of the &ldquo;<strong>Address</strong>&rdquo; needs to be &ldquo;<strong>Updated</strong>&rdquo; in the &ldquo;<strong>Address</strong>&rdquo; &ldquo;<strong>Attribute</strong>&rdquo; of the &ldquo;<strong>Person</strong>&rdquo;&nbsp;<strong>Table</strong>.</li> <li>This is called as the &ldquo;<strong>Slowly Changing Dimension</strong>&rdquo;.</li> </ul> <h1>&ldquo;Different Methods&rdquo; of &ldquo;Handling&rdquo; the &ldquo;Slowly Changing Dimension&rdquo;</h1> <p>There are &ldquo;<strong>Various Methods</strong>&rdquo; of &ldquo;<strong>Handling</strong>&rdquo; the &ldquo;<strong>Slowly Changing Dimension</strong>&rdquo; in &ldquo;<strong>Data Warehousing Solution</strong>&rdquo;.</p> <p>The &ldquo;<strong>Commonly Used</strong>&rdquo; &ldquo;<strong>Three Approaches</strong>&rdquo; to &ldquo;<strong>Handle</strong>&rdquo; the &ldquo;<strong>Slowly Changing Dimension</strong>&rdquo; are -</p> <p><a href="https://oindrila-chakraborty88.medium.com/perform-scd-type-1-using-merge-operation-on-delta-table-using-spark-sql-and-pyspark-in-91f0e69d8fc3"><strong>Click Here</strong></a></p>
Tags: SCD Type