Data Engineering Project: Twitter Airflow Data Pipeline
<p>Well, nowadays social media is abuzz with the legendary fight between Meta’s CEO Mark Zuckerberg and X’s owner Elon Musk. It has even escalated to the point of a cage fight between the two tech giants. Well we all know how that could turn out.</p>
<p> </p>
<p>Musk Vs Zuckerberg Cage Fight</p>
<blockquote>
<p>Well, their fight doesn’t at all change the daily tasks for a data engineer :p We still have to cater to irritating requests like — ‘can you quickly pull this data for me?’ at work.</p>
</blockquote>
<p>So if you want to take up a few extra projects on weekends to make yourself feel like you are a data engineer for real, then you have to follow a few legends who make great project use cases. I personally love the videos made by </p>
<p>Darshil Parmar</p>
<p> on data engineering. They will take you from a basic to an intermediate level in a matter of hours! I worked on one such project last weekend — Twitter data pipeline! It’s honestly a great pipeline for beginners in data engineering. It’s not transformation or code intensive but it helps you understand a basic flow that is required for building great, robust data pipelines. If you want to watch the entire video on youtube you can watch it here — https://www.youtube.com/watch?v=q8q3OFFfY6c. In this article I’ll be going in detail over the steps you need to follow to complete this project.</p>
<p><a href="https://medium.com/@nishasreedharan/data-engineering-twitter-x-airflow-data-pipeline-project-25b76ddfbc1a">Click Here</a></p>