Grokking the System Design Interview, Volume II
Vote

0% completed

​

Google News: Detailed Component Design

Step 6: Detailed Component Design

With the high-level architecture in place, we look closer at each part. We start with the pipeline that brings articles into the system.

6.1 News Ingestion and Deduplication Pipeline

The ingestion subsystem is the aggregator's web crawler and parser. Its job is to collect fresh articles from every external source, continuously and reliably. It runs as a pipeline of eight stages. The diagram below shows the whole pipeline, and we walk through the stages in order.

  1. **Source discovery and administration

.....

.....

.....

Like the course? Get enrolled and start learning!

Reading Progress

0%

Vote for new content