Spark is being integrated with the cloud data platform in the modern data world. Manipulating data with Spark became curial to any data persona like data engineers, data scientists, and data analysts.
Last time, we covered a trivial exercise in big data on reading and writing static data on Spark. The previous blog on reading and writing static data can be found here. In this article, we will cover a similar topic about using Pyspark to read and write streaming data using Spark Structured Streaming through readStream and writeStream.
In this article, we will learn:
- how to read the stream data using Pyspark
- how to sink the stream data using Pyspark
- examples on reading/writing the streaming data using Pyspark on Databricks
Basic Concepts on Streaming data
Streaming data is data that is continuously generated by different sources, and such data should be processed incrementally using stream processing techniques without having access to all of the data.