Missing Data Demystified: The Absolute Primer for Data Scientists

I’ve been studying this topic for many years now (I know, right?!) but along some projects I contribute to in the <a href="https://datacentricai.community/" rel="noopener ugc nofollow" target="_blank">Data-Centric Community</a>, I realized that many data scientists still haven’t fully grasped the full complexity of the problem, which inspired me to create this comprehensive tutorial. Today, we will delve into the intricacies the problem of missing data, discover the different types of missing data we may find in the wild, and explore how we can identify and mark missing values in real-world datasets. <h1>The Problem of Missing Data</h1> Missing Data is an interesting data imperfection since it may arise naturally due to the nature of the domain, or be inadvertently created during data, collection, transmission, or processing. In essence, missing data is characterized by the appearance of absent values in data, i.e., missing values in some records or observations in the dataset, and can either be univariate (one feature has missing values) or multivariate (several features have missing values): <img alt="" src="https://miro.medium.com/v2/1*rQh88nLLoMlIef9GVwdR3Q.png" style="width:700px" /> Univariate versus Multivariate missing data patterns. Image by Author. Let’s consider an example. Let’s say we are conducting a study on a patient cohort regarding diabetes, for instance. Medical data is a great example for this, because it is often highly subjected to missing values: patient values are taken from both surveys and laboratory results, can be measured several times throughout the course of diagnosis or treatment, are stored in different formats (sometimes distributed across institutions), and are often handled by different people. It can (and most certainly will) get messy! <a href="https://towardsdatascience.com/missing-data-demystified-the-absolute-primer-for-data-scientists-8c9244c764c4">Website</a>