Data Engineering Project: Twitter Airflow Data Pipeline
<p>Well, nowadays social media is abuzz with the legendary fight between Meta’s CEO Mark Zuckerberg and X’s owner Elon Musk. It has even escalated to the point of a cage fight between the two tech giants. Well we all know how that could turn out.</p>
<p><img alt="" src="https://miro.medium.com/v2/resize:fit:640/0*Ukw-_6DQWRZGHUTW.jpeg" style="height:793px; width:640px" /></p>
<p>Musk Vs Zuckerberg Cage Fight</p>
<blockquote>
<p>Well, their fight doesn’t at all change the daily tasks for a data engineer :p We still have to cater to irritating requests like — ‘can you quickly pull this data for me?’ at work.</p>
</blockquote>
<p>So if you want to take up a few extra projects on weekends to make yourself feel like you are a data engineer for real, then you have to follow a few legends who make great project use cases. I personally love the videos made by </p>
<p><a href="https://medium.com/u/27ed83189c91?source=post_page-----25b76ddfbc1a--------------------------------" rel="noopener" target="_blank">Darshil Parmar</a></p>
<p> on data engineering. They will take you from a basic to an intermediate level in a matter of hours! I worked on one such project last weekend — Twitter data pipeline! It’s honestly a great pipeline for beginners in data engineering. It’s not transformation or code intensive but it helps you understand a basic flow that is required for building great, robust data pipelines. If you want to watch the entire video on youtube you can watch it here — <a href="https://www.youtube.com/watch?v=q8q3OFFfY6c" rel="noopener ugc nofollow" target="_blank">https://www.youtube.com/watch?v=q8q3OFFfY6c</a>. In this article I’ll be going in detail over the steps you need to follow to complete this project.</p>
<p><a href="https://medium.com/@nishasreedharan/data-engineering-twitter-x-airflow-data-pipeline-project-25b76ddfbc1a"><strong>Website</strong></a></p>