Dirty Blue Widgets SEO Tutorial # 4 (Text)

How Do Search Engines function - Web Crawlers

It’s the search engines that eventually bring your internet site to the attention of the prospective clients. Therefore it’s more effective to know how these search engines really work and how they portray data to the client starting a search.

There are essentially two kinds of search engines. The first is by robots known as crawlers or spiders.

Search Engines use spiders to index websites. When you submit your internet site pages to a search engine by filling out their compulsory submission form, the search engine spider will index your whole site. A ‘spider’ is an machine-driven program that is controlled by the search engine system. Spider visits an internet site, scan the content on the existent site, the site’s Meta tags and also follow the links that the site connects. The spider then brings back all that data back to a central depository, where the information is indexed. It will go to every link you have on your internet site and index those sites too. A few spiders will only index a designated amount of pages on your internet site, so don’t make a site with 500 pages!

The spider will sporadically come back to the sites to look for any data that’s changed. The frequency with which this occurs is ascertained by the moderators of the search engine.

A spider is virtually like a book where it comprises the table of contents, the literal content and the links and references for all the internet sites it discovers throughout its search, and it could index up to a million pages daily.

Examples:  Excite, Lycos, AltaVista and Google.

Once you ask a search engine to find data, it’s really searching through the index which it’s produced and not really searching the Web. Various search engines give various rankings as not all search engines use the same algorithm to search through the indices.

Among the things that a search engine algorithm scans for is the oftenness and position of keywords on a web page, although it may also notice contrived keyword stuffing or spamdexing. Then the algorithms analyze the manner that pages link to other pages in the internet. By ascertaining how pages link to one another, an engine may both ascertain what a page is about, if the keywords of the linked pages are alike to the keywords on the main page.