Grokking the Data Engineering Interview
Vote

0% completed

​

Structured Streaming in Practice

The PySpark code below reads vending machine sales from a directory of JSON files and copies them into Parquet files. It looks like a batch job, except that it calls readStream and writeStream.

sales = (spark.readStream.schema(schema)
         .option("maxFilesPerTrigger", 1)
         .json("vend/incoming"))

query = (sales.writeStream
         .format("parquet")
         .option("path", "vend/sales")
         .option("checkpointLocation", "vend/checkpoints/sales")
         .outputMode("append")
         .trigger(availableNow=True)
         .start())
query.awaitTermination()

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%

Vote for new content