How Spark Runs a Job

The PySpark code below counts plays per title for a video streaming service. It ends by asking Spark to print its plan, and it never asks for a result.

plays = (spark.range(0, 600000, numPartitions=4)
         .withColumnRenamed("id", "play_id")
         .withColumn("title_id", F.col("play_id") % 40)
         .withColumn("seconds", (F.col("play_id") * 37) % 3600))

per_title = (plays.filter(F.col("seconds") >= 30)
             .groupBy("title_id")
             .agg(F.count("*").alias("plays"), F.sum("seconds").alias("total_seconds")))

per_title.explain()

Output:

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%

Vote for new content