Perform “SCD Type 1” Using “MERGE” Operation on Delta Table Using “SPARK SQL” and “PySpark” in Databricks
<h1>What is “Slowly Changing Dimension”?</h1>
<p>The “<strong>Slowly Changing Dimension</strong>” is “<strong>One</strong>” of the “<strong>Very Important Concept</strong>” in “<strong>Data Warehousing Solution</strong>”.</p>
<p><strong>“Slowly Changing Dimension</strong>” is the “<strong>Change</strong>” of “<strong>Attribute</strong>”, or, the “<strong>Value</strong>” of the “<strong>Entities</strong>” over a “<strong>Period</strong>” of “<strong>Time</strong>”.<br />
Example:</p>
<ul>
<li><strong>“Consider</strong>” that there is a “<strong>Delta Table</strong>”, called “<strong>Person</strong>”, which has an “<strong>Attribute</strong>”, called as “<strong>Address</strong>”. Now, a “<strong>Person</strong>” can “<strong>Stay</strong>” in an “<strong>Address</strong>” now. But, after “<strong>One Year</strong>”, the “<strong>Same Person</strong>” can “<strong>Change</strong>” the “<strong>Address</strong>”.<br />
As a result, the “<strong>Latest Value</strong>” of the “<strong>Address</strong>” needs to be “<strong>Updated</strong>” in the “<strong>Address</strong>” “<strong>Attribute</strong>” of the “<strong>Person</strong>” <strong>Table</strong>.</li>
<li>This is called as the “<strong>Slowly Changing Dimension</strong>”.</li>
</ul>
<h1>“Different Methods” of “Handling” the “Slowly Changing Dimension”</h1>
<p>There are “<strong>Various Methods</strong>” of “<strong>Handling</strong>” the “<strong>Slowly Changing Dimension</strong>” in “<strong>Data Warehousing Solution</strong>”.</p>
<p>The “<strong>Commonly Used</strong>” “<strong>Three Approaches</strong>” to “<strong>Handle</strong>” the “<strong>Slowly Changing Dimension</strong>” are -</p>
<p><a href="https://oindrila-chakraborty88.medium.com/perform-scd-type-1-using-merge-operation-on-delta-table-using-spark-sql-and-pyspark-in-91f0e69d8fc3"><strong>Click Here</strong></a></p>