Most of the major search engines like Google, Yahoo, MSN, Altavista uses a crawler known as "SPIDER" to crawl(to follow) the websites for indexing.
- Spiders are robots or crawlers, that are actually an automated program which follows the links found in the internet and the web servers.
- Spiders crawl from one link to another and also from one website to another. For this reason links are considered to be an important asset for a website.
- To make spiders crawl a website is to submit the site to the search engines and also by getting back links(inbound links) from a reputed recognized website.
- The best way for submitting the website to the search engines is the manual way of submission and not by means of any automated tool or software programs.
- The links submitted by the automated tool to the search engines are mostly not crawled by the search engines since they don't have the capacity to recognize the anti-bot codes and security codes provided by the search engines and the submitting sites.
- The important thing is submitting a URL to a search engine is well considered. But there are also some search engines that may crawl your website this day or some other day even without submitting it to them and this may take any period of time for the search engines.
- some of the major search engines's spiders names are mentioned here and they can be noted for references.
- www.google.com - googlebot.
- www.yahoo.com - yahooslurp.
- www.msn.com - scooter.
- www.altavista.com - scooterbot.
No comments:
Post a Comment