Technology Involved.

Sieving data out from that millions and millions of web page is definitely not an easy task, aren't you curious about how search engines does it within a matter of minutes and even lesser? It would be really interesting to see how a search engine works.

Here, we'll touch on the technology involved in a search engine.

Before a search engine can sieve out the information, and let you know where you can find the information you need, it first, needs to know its locations and where to find the information needed.

In order to find information from the hundreds and millions of existing web pages, it deploys special software robots called Spiders, to collate and build lists of words found on Web sites. The process of building its list is called Web Crawling, the collated list is also called the search engine index. A spider will always begin with a popular site, then, from there indexing the words and following every link they find within the site, and the flow continues (Refer to Fig.2).





















Fig 2: How spider works (Click to Enlarge)

Search engines consist of 3 main parts. The spiders follow links on the web to pages that have not yet been indexed or have been updated since they were last crawled. These pages are than crawled and added to the search engine index. Thus, when you search using a major search engine, you were actually searching a slightly outdated index of content which roughly represents the content of the web and not searching the web itself. The third part of the search engine is actually the search interface and relevancy software.

The search engines uses search query to carry out searches. Most of the search engines will carry out the following:

• accepting the query inputted by the user, then matching it with any advanced syntax; at the same time, it checks for any spelling error and recommends the more popular/correct spelling variations.
• Check whether the query is relevant to other vertical search databases (Examples, news search, product search etc). It then place a few relevant links from the vertical searches in the regular search results.
• Gathering a list of relevant pages for the organic search results . These results are ranked by its page content, usage data and citation data.
• Request a list of revelant ads which they will place near the search results.

(Cited: HowStuffWorks: How Search Engines works?, Search Engine History: technology in seach engine)


Further Reads:
(While researching on this topics, we came across some other website that have more in-depth information.)

• How does Google collect and rank results?
• Google: A Behind-the-Scenes Look (Video)
• Google's Guide: Making Searching Even Faster