Search Engine Manifesto

Early Search Engines held an index of a few hundred thousand pages and documents, and received maybe one or two thousand inquiries each day. Today, a top Search Engine will index hundreds of millions of pages, and respond to tens of millions of queries per day.

Before a Search Engine can tell you where a file or document is, it must be found. To find information on the hundreds of millions of Web pages that exist, a Search Engine employs special software robots, called spiders, to build list...