Definition

Search Engine
n.

1. A software program that searches a database and gathers and reports informations that contains or is related to specified terms.
2. A website whose primary function is providing a search engine for gathering and reporting information available on the Internet or a portion of the Internet.

Cited: Dictionary.com; Search Engine.

Search Engine, whats that?

It is almost close to impossible to search for any proper information over the net without a search engine. There are hundreds of millions of pages available over the net and almost all of them are sitting on servers with cryptic names.

This is when a search engine becomes pretty useful.

Like how the dictionary.com defines it, a Web Search Engine is actually a tool helps people find information stored on other sites over the net. Information may varies from web pages, images, information and other types of files.


Various Search Engines works differently but all of them performs three basic tasks:

• They search the Internet (or select pieces of the Internet), bases on important words.
• They keep an index of the words they find, and where they find them.
• They allow users to look for words or combinations of words found in that index.

Examples of the more well-known search engines would be:

• Google
• Yahoo! Search
• Live Search (formerly known as Msn Search)
• Ask.com (formerly known as Ask Jeeves)

(Cited: Wikipedia.com: Web Search Engine, HowStuffWorks: How search engine works? )

All good things needs a good beginning;

At xxx place not too long ago ... or maybe, really long ago.


Before the existence of Search Engines.

"There is nothing worthwhile on the web, and you will never find it anyway".

Before we have the search engines, we need to first have the Internet. At the beginning when Internet was created, there were not many people who were using it. Even so, it was rather hard to look for information because you could not search for a particular topic or interest etc. Looking for information was like playing a guessing game; you would never know what was behind the web page you click until its loaded.

Then, what was available were directories and lists on various topics which were managed and maintained by enthusiasts. (Refer to Fig.1 )












Fig 1. Net Service List (Click to enlarge)

The above is an example of a list managed by Scott Yanoff. This started off as a small list by Yanoff for his own personal use in 1991. Soon after, he started to get flooded by emails from grateful users on suggestions to the list. The list got longer and longer till Yanoff seemed to finally give up in 1995. The above (Fig 1) was what it looked like then.

By the late 1980s, the amount of web servers had gotten too huge to keep track of. The net was too loaded with information and it was almost impossible to rely on helpful hints by other friendly users.

Thus, clustering of innovations began to look for a solution to this major problem.

(Cited: Chapter4, Internet History)

Technology Involved.

Sieving data out from that millions and millions of web page is definitely not an easy task, aren't you curious about how search engines does it within a matter of minutes and even lesser? It would be really interesting to see how a search engine works.

Here, we'll touch on the technology involved in a search engine.

Before a search engine can sieve out the information, and let you know where you can find the information you need, it first, needs to know its locations and where to find the information needed.

In order to find information from the hundreds and millions of existing web pages, it deploys special software robots called Spiders, to collate and build lists of words found on Web sites. The process of building its list is called Web Crawling, the collated list is also called the search engine index. A spider will always begin with a popular site, then, from there indexing the words and following every link they find within the site, and the flow continues (Refer to Fig.2).





















Fig 2: How spider works (Click to Enlarge)

Search engines consist of 3 main parts. The spiders follow links on the web to pages that have not yet been indexed or have been updated since they were last crawled. These pages are than crawled and added to the search engine index. Thus, when you search using a major search engine, you were actually searching a slightly outdated index of content which roughly represents the content of the web and not searching the web itself. The third part of the search engine is actually the search interface and relevancy software.

The search engines uses search query to carry out searches. Most of the search engines will carry out the following:

• accepting the query inputted by the user, then matching it with any advanced syntax; at the same time, it checks for any spelling error and recommends the more popular/correct spelling variations.
• Check whether the query is relevant to other vertical search databases (Examples, news search, product search etc). It then place a few relevant links from the vertical searches in the regular search results.
• Gathering a list of relevant pages for the organic search results . These results are ranked by its page content, usage data and citation data.
• Request a list of revelant ads which they will place near the search results.

(Cited: HowStuffWorks: How Search Engines works?, Search Engine History: technology in seach engine)


Further Reads:
(While researching on this topics, we came across some other website that have more in-depth information.)

• How does Google collect and rank results?
• Google: A Behind-the-Scenes Look (Video)
• Google's Guide: Making Searching Even Faster

All good things can be manipulated;

At this point of time, you'd probably go, awwww shucks, but we're not kidding!



There are actually a lot of ethical issues behind the usage of search engine. In this particular post, we will be touching on the Ethical and Legal Issues of Search Engines.

Pitfalls of Search Engines for users:

- Users could be invading privacy
(Search engines have become a media which allows others to intrude privacy intentionally and unintentionally)

- The results may not be what the user is looking for.
(Unwanted web pages containing irrelevant information, advertisement and sometimes even viruses and Trojan)

- Restrictions may not be strong enough or there may even be lack of restrictions
( An eight year old child may be easily exposed to pornography or gruesome websites under a single search)

- Plagiarism may be encouraged due to the easy accessibility of information which is variably and publicly available. Detection is also difficult.

- Identity tracking
(One can easily obtained more information of a stranger with just his/her name and maybe minimum personal information)

All good things can be manipulated (part 2)

A search engine also can be exploited by people with ill-intentions whom manipulated it to obtain what they need. One such behaviour is know as the google hacking.


Google Hacking

It is a term used to describe when a hacker uses the search engines to find exploitable targets and sensitive data.

Sensitive data can be identified by a set of queries from the Google Hacking Database (GHDB).

Google hacking may seems like a scanty issue, but in actual fact, hackers can obtain sensitive data that allow them to cause harm to your system or sell data that are confidential.

The Google Hacking Database can identify information like that following:

• Advisories and server vulnerabilities
• Error messages that contain too much information
• Files containing passwords
• Sensitive directories
• Pages containing logon portals
• Pages containing network or vulnerability data such as firewall logs.

(Cited: Acunetix; Google Hacking, Wikipedia; Google Hacking. )

Video

Reflections

The conclusion that we have come up with is that even though there may be ethical and legal issues involved in using search engines, we have reached a point where internet users will not be able to do without them. Just think of trying to find a product on the internet: how would you go about to find it without using a search engine? We dare say that the benefits of having search engines significantly outweigh the problems that exist and such is the importance of their standing that saying the internet world will be crippled without search engines may even be an understatement. The solution to solving the loopholes that exist in search engines is not to do away with the search engines, but to educate the users on protecting themselves against the vulnerabilities that they face. It is easier for people to learn to safeguard themselves and their computing resources rather than for the major search engine companies to come out with a new way their products process. Governments must also lend their hands in solving these problems by passing laws that tackle copyright issues, and step up in their actions in eradicating the persistent offenders.

Before doing this project, we did not know that there was so much to search engines than we ever thought. They are highly useful tools that we have been using ever since we started using the internet. Our group believes that the search engines have “shrunk” the world in the same manner that airplanes did when they were invented. We have picked up useful information in using search engines along the way in doing our project and also found out about the malicious “google hacking”. As quoted from spiderman’s uncle: “With great power comes great responsibility”. The same thing applies to anyone who uses the power of the search engine to sieve through the millions of items that can be found on the internet.

So next time you use any search engine, take a while to admire the wonderful program while it searches and returns your search in the blink of an eye!

Comments + Disclaimer

Hi everyone! Whoa, we must be really interesting for you to read till the end. Apart from all this, all the sources of our research were linked at the end of each relevant post. If you're looking for something more extensive and in depth, do click on the links and visit the sources. (: All information found on this web do not completely belong to us. Sources are credited. Thanks!

Comments:
We accept constructive criticism or comments of any kind.

Do post your questions, comments, constructive criticism etc, etc, etc on the comment link under this post.

Much loves.