Google News: High-Level Architecture

Step 3 left us with a shape. The system is read-heavy, with bursts of new articles, and topics that can draw huge numbers of followers. Now we lay out the services that give that shape a design.

At a high level, the system is two pipelines working together.

The content ingestion and processing pipeline collects articles from outside sources. It normalizes, deduplicates, and categorizes them, then stores them in a content repository. Its job is to keep a clean, current collection of articles with metadata.