Spark Joins and Skew

The PySpark job below joins 24 million play events to a table of 40 titles and totals the plays and watched seconds per genre. Spark split the join into 8 tasks. The script prints only the plan lines for joins, exchanges, and sorts. Look at the last table in the output, which lists how many rows each task read and how long it ran.

spark.conf.set("spark.sql.adaptive.enabled", "false")
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
def query():
    plays, titles = tables(spark)
    return by_genre(plays.join(titles, "title_id"))

Output:

cores: 2

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%

Vote for new content