Case Study: A Concurrent Web Crawler

A small site has six pages within three links of its home page, and each page takes 100 ms to download. A crawler that fetches one page at a time needs 6 times 100 ms, which is 600 ms. The links form four levels: the home page, then two pages, then two more, then one. A crawler that fetches every page as soon as it learns about it needs only 4 times 100 ms, which is 400 ms.

On a real site with thousands of pages, the same arithmetic is the difference between a crawl of minutes and a crawl of hours. A faster crawler also brings three problems that a sequential one never has