Meta tags are an under-used feature of every web page. Users cannot normally see the information stored in Meta tags, except when third parties access that information. A good example of this is the ‘Description’ meta tag, which search engines sometimes use as the snippet that appears in search results. The ‘Robots’ meta tag is a good example of a tag that users don’t see, however, search engines do see it, and the contents will help to control how a search engine treats your page.
There are several different forms of the Robots meta tag, and good SEO practices would suggest that you use all of them. Ideally, a webmaster would make use of a ‘Robots.txt’ file in order to control how search engines deal with individual pages, however, it also makes sense to use the Robots meta tag to reinforce the information in the Robots.txt file.
The Robots meta tag is quite a clever tag really. It appears in three formats: the standard ‘Robots’ tag is designed to cover all search engines, the ‘MSNBot’ tag is designed to talk to Bing, and the ‘Googlebot’ tag is designed to talk to Google. You can essentially tell Bing to do one thing and Google to do another. There is also a train of thought that suggests using only the ‘Robots.txt’ file, and not including a Robots meta tag. Since including a Robots meta tag isn’t going to hurt your website, you can still include them.
So what information does a Robots meta tag hold? When a search engine crawls your website, it will read and index each page. The search engine will also define a snippet for the page, and look at (and follow) any links on the page. The Robots meta tag can be used to control each of those areas using specific codes.
The Robots meta tag appears in thesection of your page’s code. It’s not important where so long as it’s before the tag. The following tags are standard for the three major Robots meta tags used:
checkout pages, you don't necessarily want these pages appearing in search results, especially if they are drawing value away from important product pages. NoFollow tells a search engine you don't want the internal links on that page followed. NoArchive tells the search engine not to archive a page while NoSnippet indicates you don't want a snippet to appear in the search results. At present, only Google recognizes the NoSnippet command.Robots meta tag commands can be used together; for example, you could have Content="NoFollow, NoIndex, NoArchive". Search engines will generally abide by each of those commands.When completed, a typical Robots meta tag should have the following format - – obviously you would use commands that suit your pages. When used properly, a webmaster can control which pages appear in search results, and whether or not the links on a page are followed.
A final word of caution. Meta tags do add to a page’s overall content (and size). If you can put together a good ‘robots.txt’ file, then the meta tag is not necessary. You can over-ride meta tag commands by adding new commands within your content. For example, if you have not specified NoFollow in your meta tag, then you can add the ‘nofollow’ command to any links (or vice-versa).