Messaging Systems: Introduction

Background

Picture a log aggregation service. It receives hundreds of log entries every second, from many different sources. It writes each entry to disk on a shared server, and builds an index so the logs can be searched later.

Three problems show up right away.

The first is a spike. Say the service can buffer 500 messages a second. What happens once traffic goes above that? If we add more instances of the service to cope, how do we split the incoming work between them?

The second is variety