| Robots
Meta Tag
The Robots Tag declares to search engines the content you want
them to index and spider.
Robots, also known as
spiders, are mechanisms that automatically spider your site,
or search your site to determine how to categorize the information
you submitted to the search engine. Typically, a website owner
would submit the main page and the robots would visit your site
and collect all subpages and related links from your main page.
However, this tag enables you to control which pages you would
like spidered, and which to ignore. For instance, certain webpages
and directories (ie: CGI Scripts) you may not want indexed in
the search engines. Using the robots tag, you can define which
pages to follow, which to index and which to ignore completely.
Example:
<META NAME="robots" CONTENT="index, follow">
|
