Buley Library

How Searching the Web Works

Search Engines are searchable databases generated by software "robots" or spiders that roam the World Wide Web looking for pages. With the returns from these spiders a database is created that attempts to provide a picture of what material is available on the web. You can search these databases (Search Engines) which return the sites that most closely match your search query.

Building the Search Engine

Search Engine sends spider/agent to a web page for indexing and inclusion of the page in the search engine database.
Spider analyzes page based on criteria below and sends back a report/record for the page it has analyzed.
Spiders then follow the URLs it has found on the page to more pages and then creates records for these pages.
And so on ...

Using the Search Engine

User goes to Search engine and types in a query.
Search Engine analyzes the query and searches its database of what the web looks like and returns the results. It then ranks the results based on it's own criteria and finally sends the user a ranked list of results.

Ranking Criteria

Are the criteria that the search engines use to determine the order in which results are displayed. Every search engine is different and use different criteria and cover different sites. Each of these aspects is given a level of importance which helps the search engine return the most relevant sites to the search.

Domain Name
Meta Tags - keywords and description
Title of the page
Text, words on the pages

Proximity of searched words to each other
Frequency page is referred to by other indexed pages
Alt tags on image

Search Engine Issues:

Search engine are a picture of what the web looked like when it last explored a site, which could have been many days ago.

No search engine is perfect.

None of the databases contains ALL the documents on the Internet.

All search engines have a help page. This is the first place to visit when trying a new search engine.

Every search tool is different. They all have syntax that must be followed for an effective search.

Beware of the default Boolean logic.

Search engines use different criteria to determine relevancy rating of each hit. The most relevant hits may or may not appear in the first few screens. Or the most helpful hits may not be linked to by many others so do not appear early on in the rankings.

Spiders cannot read image

Some search engines sell their top positions to the highest bidder, beware!

Search engines are constantly updated and new search engines developed. Check out the SEARCH ENGINE WATCH by Danny Sullivan. This site includes information on search engine reviews, tutorials on engines, insight into search engine technology, and more.

Some Highlights of Search Engine Watch:

What they don't Search

Any site that specifies that it not be indexed (a no robots command in the HTML)

Sites that require a password, so most journals and library databases as well as almost all sites that charge for their content. Also many private sites such as internal company sites.

Sites that are not linked to by other sites

For more on how Search Engines work see:

Interesting Info about Search Engines:

70% of users typically type in only one keyword.

Over 75% of web users use search engines to traverse the web

Over 75% of web users use search engines to traverse the web. April 2000, RealNames Survey 46% of Internet users find new web sites via search engines. Word of mouth (20%) and random searching (20%) were the next most popular methods.

57% of Internet users search the web each day, making search the second most popular Internet activity. 46% say they look for product info, making this the third most popular activity. Email is the most popular activity, with 81% checking each day.

1 in every 28 page views on the Web is a search results page (3.5% of all page views)

84.8% of people use search engines to find new web sites

Things you need to know when using a particular Search Engine

Size: this represents the number of web pages indexed.

Updated: Web spiders take time to crawl the web, and to send back their finding. This field shows how current this search engine is.

Phrase Searching: Phrase searching is used to find words together as a phrase; for instance a search on "the heart of darkness" will return only pages that have the words heart of darkness in them in that order next to each other. But the and of are stop words so they are not searched. This means that this will really be a search for "heart darkness".

Requires word in search: The required words must be somewhere in the retrieved documents.

Exclude word in search: Any retrieved document cannot have this word in it.

Default multiword operator: When you just type a couple of words in with no Boolean operators the default operator will be used to connect these words together. Knowing the default operator can make a HUGE difference in the quality of your search.

Look for either word: This means one or the other or both words must be in the document.

Truncation: Truncation is used to indicate ending on a word. For instance the truncated word cat will find cats, category, catalog etc. You should, as this example shows, be careful when using truncation. Truncation is most useful to catch plurals.

EXTRA FUN INFO!!
How the Web is like a Bowtie

A study of the web's structure, five times larger than any attempted previously, reveals that it isn't the fully interconnected network that we've been led to believe. The study suggests that the chance of being able to surf between two randomly chosen pages is less than one in four. Researchers from three Californian groups at IBM's Almaden Research Center in San Jose, the Altavista search engine in San Mateo and Compaq Systems Research Center in Palo Alto have analysed 200 million web pages and 1.5 billion hyperlinks. Their results, which will be presented next week at the World Wide Web 9 Conference in Amsterdam, indicate that the web is made up of four distinct components. A central core contains pages between which users can surf easily. Another large cluster, labeled 'in', contains pages that link to the core but cannot be reached from it. These are often new pages that have not yet been linked to. A separate 'out' cluster consists of pages that can be reached from the core but do not link to it, such as corporate websites containing only internal links. Other groups of pages, called 'tendrils' and 'tubes', connect to either the in or out clusters, or both, but not to the core, whereas some pages are completely unconnected. To illustrate this structure, the researchers picture the web as a plot shaped like a bow tie with finger-like projections. (from Nature 405, 113 (2000) © Macmillan Publishers Ltd. )