Let's design a news aggregator similar to Google News. This system will collect news articles from many external sources (newspapers, news channels, blogs, etc.) and present them to users in one consolidated feed. The goal is to deliver a personalized, near-real-time news feed based on each user’s preferences such as subscribed channels, categories (like politics, sports, tech), or particular entities (e.g. a celebrity, sports team, or artist). The system should support both pull (user refreshing a feed) and push (notifications for new articles) delivery modes, ensuring users quickly see breaking news or updates in their subscribed topics. We aim to achieve this at web-scale, meaning the design must accommodate hundreds of millions of users and very high data throughput, all while maintaining low latency and high availability.

Entities and Concepts

• News Article: A news item fetched from a source. Includes metadata like title, content snippet, URL, publication time, source name, category tags, and possibly a unique ID or hash. This is the core content that will be delivered to users’ feeds.
• News Source: An external publisher or feed producing articles. It could be a news website, RSS/Atom feed, or an API. Each source has attributes like source name, URL or feed endpoint, and categories it produces (e.g. CNN – categories: politics, world, sports). The aggregator tracks thousands of such sources globally (Google News, for example, tracks over 50,000 sources worldwide).
• Category: A predefined topic classification (Politics, Sports, Technology, Entertainment, etc.). Each article may belong to one or more categories. Categories help organize content and allow users to subscribe to broad topics.
• Entity/Keyword: A specific person, organization, or keyword of interest (e.g. “Taylor Swift”, “NASA”). Users may subscribe to these to get all news mentioning that entity. The system may perform entity recognition on articles to tag them with relevant names/keywords for matching subscriptions.
• User Profile: Represents a user of the system. Stores user’s preferences and subscriptions (which sources, categories, or entities they follow). Also includes info needed for push notifications (e.g. device tokens or WebSocket session info) and possibly reading history for future enhancements (like recommendations).
• Subscription: A relationship between a user and a content filter (source, category, or entity). This can be represented as (User, Topic) pairs. A user can have multiple subscriptions. For example, User123 subscribes to {“BBC News” (source), “Sports” (category), “NASA” (entity)}.
• User Feed: The personalized timeline of news articles for a user, aggregated from all the user’s subscriptions. This feed is updated in near real-time – whenever a subscribed source/category/entity publishes a new article, it should appear in the feed (and possibly trigger a push notification). The feed typically shows the latest N articles (with maybe headlines, snippet, and source info) sorted by time or relevance.
• Feed Update Event: When a new article is ingested that matches a user’s subscriptions, that is essentially an event that could trigger a push notification or at least marks that the user’s feed has a new item. The system needs to handle these events efficiently, especially if an article is relevant to millions of subscribers.

Example Scenario: A user subscribes to “Tech” category and “Apple Inc.” entity. When Apple releases a new product and various news sources publish articles about it, the aggregator ingests those articles, classifies them as “Tech” and tags “Apple Inc.” as an entity. The system will then deliver these to the user’s feed. If push is enabled, the user might get a notification on their phone saying “New article: Apple launches X” within seconds of publication.

At a high level, our news aggregator is composed of two main pipelines that work in tandem:

1. Content Ingestion & Processing Pipeline – Responsible for collecting articles from external sources, processing them (normalize, deduplicate, categorize), and storing them in a content repository. This pipeline ensures we have a clean, up-to-date collection of news articles with metadata.
2. Feed Generation & Delivery Pipeline – Responsible for delivering the right content to each user. It takes the processed articles plus the user’s subscription info to produce a personalized feed. It supports on-demand fetch (pull) and real-time push notifications. It includes services for subscription management, feed assembly, and notification dispatch.

Major Components Overview:

1. Ingestion Layer (Content Collection): External news sources feed into the system through a scalable ingestion framework:

• RSS/Feed Fetcher: A service that polls RSS/Atom feeds of news sites for new articles.

• Web Crawler/Scraper: A crawler that visits websites (for sources without feeds or for extracting the full article content). It can parse HTML to extract headline, body, etc. using rules or machine learning.

• Third-Party News APIs: Integrations with providers like NewsAPI or Google News API for sources that expose their content via JSON/XML APIs.

• We maintain a list of “seeds” or source URLs that the crawler and fetchers use. These could be feed URLs, site maps, or even other aggregators (for example, seeds like Reddit or Hacker News can be used to find content).

• The ingestion system is distributed: multiple crawler instances can run in parallel, partitioned by source or region. An Orchestrator service (scheduler) manages crawl frequency and ensures we don’t overload source sites or violate API rate limits.

• New Article Event Stream: As articles are fetched, each new article is packaged into a message/event (containing its content and metadata). These events are published to a message queue or stream (e.g. an Apache Kafka topic). This decouples upstream crawling from downstream processing – ensuring that bursts of incoming articles can be buffered and processed asynchronously. The event includes data like article ID, source, timestamp, title, content snippet, etc.

2. Processing & Storage Layer: A set of backend services consume the article events from the queue and perform processing:

• Content Parser & Normalizer: Cleans up the raw data. For example, ensures we have a consistent format (fields for title, author, publication time, etc.). It might strip HTML, encode characters properly, etc.

• Deduplication Service: Checks if the article (or an extremely similar one) was already seen recently. This could be done by computing a hash or fingerprint of the content (or using similarity detection). If it’s a duplicate, we might drop it or mark it as part of the same story cluster.

• Categorization Service: Assigns category tags (and possibly sub-topics) to the article. This might use a machine learning model (trained on news text) or simpler keyword rules. For example, an article containing “football, NFL, score” likely gets category Sports. We also detect if it’s local to some region (e.g. mentions of a city).

• Indexing & Storage: The processed article is then stored in our databases:

  • A content database stores the article records. We may use a combination of storage technologies:

    • A NoSQL document store or wide-column DB (like Cassandra or DynamoDB) to store the raw article info and allow fast key-value access by article ID or queries by category+time. Cassandra is a good choice here for high write throughput and tunable consistency.
    • A full-text search index (like Elasticsearch or Vespa) to enable text queries and complex filtering (e.g. search by keywords, or fetching top N latest in “Tech” quickly). The search index can store the necessary fields and inverted indices for article text.
    • Optionally, a relational database (SQL) for structured metadata if needed (though for mainly feed content, NoSQL/Elastic might suffice). For instance, user data might live in SQL but article data might not.

  • The search index is updated with the new article document, making it queryable within seconds. This index can later be used to fetch relevant articles for user feeds (by category or keyword).

• Media Storage: If we need to handle images or videos, those might be downloaded to a storage service (like AWS S3 or a CDN) rather than stored in the DB. We typically store just the URL or reference in the article record.

At the end of processing, the system has persisted the new article and tagged it appropriately. Now it’s ready to be delivered to users who want such content.

3. Feed Delivery Layer (Personalization & Distribution): This is the heart of how we get relevant content to users.

• User Preferences Store: We maintain a database of each user’s subscriptions and profile. This could be a relational DB (like MySQL/PostgreSQL) storing user profiles and the list of topics they follow, or a fast key-value store if we need to retrieve it very frequently. For example, a table mapping user → [list of category IDs, source IDs, topics] or a graph database if relationships were complex (friends, etc., though here it’s simpler).

• Feed Generation Service: This service is responsible for compiling the list of relevant news for a user. There are two primary strategies to generate feeds:

  • Pull model (on-demand query, a.k.a fan-out-on-read): Each time a user requests their feed, the service queries the content store for the latest articles matching that user’s interests. For example:

    • Lookup the user’s subscribed topics (e.g. Politics, and CNN, and “Climate Change”).
    • For each interest, query the index or database for recent articles in that category or from that source.
    • Merge all those articles, sort by recency or by a ranking score, then trim to the top N for display.
    • Return that list to the client.

    This approach ensures on-the-fly freshness but can be expensive at scale (many queries per feed and high latency if a user has many follows). We will mitigate that with caching and precomputation.

  • Push model (precomputed feed, a.k.a fan-out-on-write): New articles are proactively pushed into users’ feed storage as they arrive. That is, when an article is processed, the system determines which users should see it and immediately updates a stored feed timeline for those users. For example, a Feed Fan-out Service listening to the article event stream takes each new article and looks up all users following its category or source, then inserts an entry into each of those users’ feed data (e.g., in a Cassandra table partitioned by user). This makes retrieval very fast (just read a pre-built list) at the cost of many write operations up front.

  • Hybrid approach: In practice, we will use a mix of both to handle the “viral news” problem. For content with a smaller audience (niche topics or local news followed by relatively few users), a push model is efficient and ensures those users get the content instantly. For content that would require huge fan-out (e.g. general news that almost everyone sees), a pure push would be millions of writes at once – instead we can fall back to pull for those or use a shared feed. For instance, maintain a global or per-category feed that all users can read from for broad topics, rather than copying into each user’s feed. This idea aligns with using a hybrid fan-out model: push-based for normal or low-fanout updates, and pull-based for extremely high fan-out cases. In other words, push to hundreds or thousands of targets, but don’t push to 50 million – let those be fetched.

• Feed Storage and Caching: To support the above, we likely maintain:

  • A User Feed Cache/Store: This could be an in-memory cache (like Redis) or a Cassandra-style wide-column store containing precomputed feed entries for each user (sorted by time or score). This is updated by the push model. The NewsFeed service first checks here when a feed is requested.
  • A Category-wise Cache: For the pull model, we can maintain hot lists of recent articles per category or topic in a fast store. E.g., the latest 100 Politics articles in a sorted set in Redis. This way, assembling a user’s feed is a matter of pulling from a handful of these precomputed category lists (and then merging).
  • Multi-level Caching: We want to leverage caching at multiple levels to reduce latency. At the client side, the app may cache the last feed it saw. CDNs or edge caches might cache common API responses (though personalized feeds are hard to cache globally, except maybe for logged-out generic content). On the server, we can use an in-memory cache for recently accessed feeds or articles. For example, if user A and user B have very similar interests, the system might reuse some results.

• Ranking & Personalization Engine: Simply merging by timestamp isn’t always ideal; we incorporate a ranking step:

  • The ranking component takes the set of candidate articles for a user (from their followed categories) and scores or orders them. Factors could include:

    • Recency – newer is generally higher.
    • Source priority – perhaps user reads some sources more often, boost those.
    • Content popularity – if an article is trending (many users clicking it) maybe rank it higher for others.
    • Personal behavior – e.g., if user frequently clicks “Tech” articles but rarely clicks “Sports” even though they follow both, the tech ones may be ranked on top.
    • Diversity – ensure a mix of categories if user has many, rather than 10 politics articles in a row.

  • The ranking can be a simpler heuristic or a machine-learned model. In early stages, we might implement a weighted scoring (like weight = w1freshness + w2sourceQuality + w3*predictedInterest).

  • This engine might run as part of feed assembly or as a separate service. If separate, the feed service would call it with the candidate list and get back a sorted list.

  • Real-time signals: If we track in real-time which stories are receiving a lot of engagement, those signals can feed into ranking (for example, to boost a breaking news that’s clearly very important). This is akin to how Google News might rank “Top stories” using an importance metric.

• Notification/Push Service: For delivering push updates, a notification service will subscribe to new article events or feed updates. It will determine if an article warrants a push notification (perhaps user has “alerts enabled” for that topic, or it’s breaking news flagged by the system). It then sends the notification via the appropriate channel:

  • Mobile app users: through Apple APNS or Google FCM (the service prepares the payload and uses those APIs, which handle device delivery at scale).
  • Web users: possibly via Web Push API or, if the user is currently on the site, via a WebSocket/SSE connection to the client.
  • Email could be another channel (for daily digests, etc.), though that’s not real-time.

  The Notification service needs to be careful to not spam; it might apply rules (e.g., don’t send more than X pushes per user per hour, aggregate multiple new articles into one notification if appropriate).

• Real-Time Update to Clients: If a user has the app open, we don’t want them to have to pull-to-refresh to see new items. We can keep a persistent connection (WebSocket or Server-Sent Events stream). The feed service (or a dedicated real-time server) can push a message like “new article in your feed” to the client, which then either fetches it or we send the article data directly over the socket. This gives a live feed feel.

• API Gateway and Application Servers: All user interactions (feed fetch requests, search queries, etc.) go through a layer of API servers:

  • These are stateless servers (could be Node.js/Express, Java/Spring, Go, etc.) behind a load balancer. They handle authentication, validate requests, then call the appropriate backend services (feed service, search service, user profile service, etc.).
  • For example, a GET /feed request hits an API server, which authenticates the user, then calls the Feed Generation Service (which may in turn query caches/DB). The API server then formats the response (JSON) and returns it.
  • Similarly, a POST /subscribe would go to a User Pref service to add a subscription and likely trigger a recomputation or warm-up of that user’s feed.
  • This layer also integrates with CDNs for static content: e.g., image URLs in articles might be served via a CDN domain. The API can provide these URLs (perhaps pointing to a cached copy on the CDN if we do that to ensure faster image loads).

• Search Service: We will have an endpoint for searching past news by keywords, powered by the search index (Elasticsearch query, for instance). This is a separate path from the feed, but uses the same indexed data.

Data Flow Summary: When a news article is published by a source, our ingestion layer picks it up (within seconds or minutes), processes it, and stores it. This triggers updates: the feed service (push component) may update many user feeds or relevant caches, and the notification service may alert users. When a user opens their app (pull), the feed service quickly retrieves a personalized list (either from the precomputed feed or by querying recent content) and returns it. The user sees a timeline of articles and can click through to read one (likely fetched from cache or the source). If a new article arrives while they’re browsing, a live update or notification can appear.

This high-level design is modular: each component (crawling, processing, feed assembly, notifications) can be scaled independently.

Designing for web-scale involves applying many techniques to handle high load and ensure reliability. Here we highlight the key strategies and justify our choices:

• Sharding and Data Partitioning: All our major data is partitioned to spread load:

  • We partition news articles across database nodes (Cassandra automatically partitions by primary key; using article_id or category as part of key ensures distribution). This allows writing many articles per second across the cluster.
  • User-specific data (feeds, preferences) is partitioned by user ID. Thus, operations for different users naturally go to different shards. There’s no central hot spot for feed reads or writes. This is crucial given potentially 1000s of requests/sec spread across users.
  • The search index can be sharded by time or by document ID and scaled horizontally; queries will be distributed.
  • Ingestion tasks are partitioned by source to multiple crawler instances so no single machine handles all sources.

• Caching at Multiple Levels: We extensively use caching to achieve low latency:

  • At the client side (browser/app), we can cache previously loaded feed data and only fetch updates (saving server calls).

  • CDN caching is used for static assets (images, scripts) and possibly for some API endpoints that are not user-specific (like a general headlines feed).

  • Edge caching and reverse proxies could cache common API responses. However, since feeds are personalized, this is limited. But if we have any popular public feeds (like “top news for country X”), those can be cached.

  • Service-layer caching: Within our backend, we cache frequently accessed data:

    • Recent articles by category (as discussed, in Redis).
    • User preference data (so we don’t hit DB every time).
    • Completed feed results for a short time (to quickly serve repeated requests).
    • We also use in-memory caches within each service instance for ultra-fast access to hot items (with proper invalidation or short TTL).

  • Multi-tier caching ensures that by the time we hit the primary database, we’ve avoided as many queries as possible.

• Asynchronous Processing and Queues: By decoupling components with queues (Kafka), we make the system resilient to spikes. If a sudden surge of news hits, the ingestion queue will buffer it and workers will process as fast as they can – if slightly delayed, that’s acceptable. This prevents overload and crashing. Similarly, for fan-out, we can queue user feed updates. Async processing allows us to handle tasks in parallel and retry if needed without blocking the user-facing parts.

• Eventual Consistency & Relaxed Constraints: We deliberately allow stale data in exchange for performance. For example, we don’t use distributed transactions between services. Each part does its best effort. The system doesn’t wait for confirmation that every user’s feed got updated before proceeding – it’s fire-and-forget for a lot of the fan-out. This improves throughput dramatically. As noted, a known trade-off is some users might temporarily not see something, but we compensate with pull mechanisms. By prioritizing availability, the service remains responsive even under partial failures.

• Horizontal Scaling of Services: Every stateless service (API servers, feed generators, processing workers) can be cloned and load-balanced. If traffic increases, we add more instances behind the LB. For stateful components (databases, Kafka), we add nodes to the cluster and partition more – Cassandra and Elastic both scale this way. We ensure to monitor load and have auto-scaling rules where possible (like automatically add crawler instances if backlog grows, etc.).

• Load Balancing and Failover: We use load balancers at entry points (user requests distributed across many API servers). Also, Kafka and Cassandra handle their own load distribution among nodes. If a node fails, clients (with driver libraries) will automatically route to other nodes. We should design with no single point of failure – e.g., multiple Kafka brokers, multiple Cassandra replicas, multiple web servers. Use health checks to detect failures and remove them from rotation.

• High Availability Datastores: Cassandra is inherently designed for HA, replicating data across nodes and across data centers, so it can tolerate node failures without downtime. We would configure maybe replication factor 3 per data center. Elastic can be set up with primary and replica shards too. For the user relational DB, we can run a primary-replica setup with automatic failover to a replica if the primary fails (or use a distributed SQL like CockroachDB for multi-region writes if needed). The idea is to avoid any downtime in data access.

• Throughput Optimization for Feed Fan-out: The push model could be the heaviest part. To optimize:

  • We batch database writes. Instead of inserting one feed entry at a time, aggregate a batch of, say, 100 or 1000 inserts and send in one go. Cassandra, for instance, supports batch writes (though with caution). Batched writes reduce overhead per item.
  • Use multi-threading and parallelism: The fan-out service will run on many machines, and within each, use multiple threads to handle different segments of user IDs simultaneously.
  • If using Kafka for distribution: we could publish one message per new article per user, but that’s millions of messages for a big story. Instead, we might publish one message per new article (with metadata), and have consumer groups partitioned by user segments. This still ends up in processing per user. Instead, we might skip Kafka at that stage and directly use a custom fan-out dispatcher to write to DB (to avoid the overhead of producing millions of Kafka messages).
  • The hybrid approach again mitigates load by not doing push for the worst cases. That drastically cuts the peak load. The “celebrity” scenario (viral story) is handled by pull which spreads the cost over time and only for users who actually check their feed.
  • We also consider TTL (time-to-live) on feed entries: to keep the UserFeed table from growing indefinitely, we could expire entries older than X days, since typically we only show recent news. This keeps storage usage bounded per user and improves cache locality.

• Precomputation and Background Jobs: We precompute anything expensive offline if possible. For example:

  • We could precompute daily or hourly trending scores for topics (so that ranking doesn’t have to calculate popularity on the fly – it can just read a cached trend score).
  • We might periodically rebuild a user’s “interest profile” (e.g., which categories they engage with most) and store a profile vector, which the feed ranker can use quickly.
  • Generate “recommended for you” stories in advance during low-traffic times (maybe as a separate section).
  • These reduce online computation needed at request time.

• Content Delivery Network (CDN): All images and videos in articles will be served through a CDN. We might even cache article pages (if we host a reader mode) on CDN. This offloads bandwidth from our servers and gives faster response to users globally. We just need to ensure to refresh cache for images if needed (most images won’t change).

• Security and Rate Limiting: From a performance perspective, we also guard against misuse. We will implement rate limiting on API usage to prevent a single user or client from spamming the system with requests that could degrade performance. Also, bots or scrapers trying to crawl our aggregator might be served cached results or blocked, so they don’t overload us.

• Monitoring and Auto-Scaling: We set up metrics and alerts. If CPU or queue length or DB latency crosses thresholds, automated systems add more instances or trigger failovers. This dynamic scaling ensures we meet demand surges (e.g., big news event causing more user traffic and more ingestion load simultaneously). We also have fallbacks – e.g., if the feed service is ever overwhelmed, we could degrade gracefully by temporarily serving a simpler feed (maybe just top news) to all users rather than personalized, just to handle the moment – but that’s an extreme measure.

Trade-offs discussion: In our design, we traded some consistency and complexity for performance:

• We maintain possibly duplicated data (the same article ID might be stored in many user feeds and multiple DB tables). This denormalization costs storage and potential slight inconsistency (if we needed to remove/update an article, we’d have to purge many entries). We accept that cost to gain read speed. Storage is cheaper than compute at this scale.
• The push model complexity: We chose a hybrid model to avoid extremes. Pure push (fan-out-on-write) gives lowest read latency (feeds are ready) but can be wasteful (writing items for users who might not log in before those items are old) and heavy for broad fan-out. Pure pull (fan-out-on-read) saves writes but at scale gives high read latency and DB load. The hybrid tries to balance it, but is more complex to implement. This complexity is justified by the need to handle “followed by millions” scenarios efficiently.
• We emphasize availability: data is duplicated and eventually consistent, meaning in rare cases users might see slightly outdated info. We deem this acceptable for news (a 5-second or even 30-second delay in a story appearing is usually fine; we’re not doing financial transactions). This allows us to avoid distributed locks or serial processing, massively improving throughput.
• Our system will likely use more hardware (many servers) to achieve these goals. That’s expected at web-scale. We assume the cost is acceptable for a service of this user base. We also keep it somewhat efficient by using appropriate technologies (Cassandra/Redis for speed, etc.) and not over-computing on the fly.

In conclusion, this design leverages a pipeline architecture, distributed data stores, caching, and careful choice of push vs pull to meet the requirements. It should be able to deliver personalized news to users in near real-time (a few seconds latency) even under high load, with eventual consistency ensuring the system remains robust and available. The architecture is modular and scalable, much like those of large-scale social feed and aggregator systems in production today, and reflects best practices a senior engineer would apply to such a problem.