If you have a new website, what’s the easiest way to tell the search engines about that website? Search engines generally ‘discover’ pages by following links, however, a new website is unlikely to have many incoming links so a webmasters needs to tell the search engines about their site. The easiest way to achieve this is by having a Sitemap. As the name suggests, this is a map of your website that includes all of the URLs that you want a search engine to crawl and index.
The Sitemap used is referred to as an XML Sitemap file; XML being the language format, and it is a separate file that is generally saved in the root directory of your website. If you have a Google account, then SEO best practices suggests creating a Google Webmaster Tools account, and using that account to submit your sitemaps. Google’s Webmaster tools has several benefits, one of which is a record of when the sitemap was last accessed, how many pages have been indexed, and whether there were any problems detected.
An XML Sitemap can be difficult to produce by hand. If you have a website with less than 500 pages, then you can access several free XML Sitemap creation programs at no cost. These programs will create a properly formatted sitemap, all you need do is download it to your website’s root directory and to tell the search engines where to find it.
There are several dedicated programs that will create much larger XML Sitemaps, however, there are some points that should be understood when creating a sitemap. The first is size. Each sitemap should have less than 50,000 URLs, and should be smaller than 10mb. You can break sitemaps up into smaller sitemaps, then use a ‘sitemap index’ as the master – this points to each sitemap by name and location. When it comes to SEO, if you have a website that deals with several different topics, then a sitemap for each topic can increase the number of URLs indexed for each topic.
XML sitemaps contain data in a tight format that provides information to a search engine about an individual page. The sitemap’s format includes information on:
- the exact URL
- the last time a page was modified
- how often the page is modified
- the page’s priority
Search engines crawling a website can bump up bandwidth use significantly. By controlling how a search engine crawls your website, you can significantly reduce that bandwidth. At the same time, if you have important pages, you can ensure they are crawled more regularly. This is very useful on websites that include product descriptions with stock numbers and/or user generated feedback. You can set that page to be crawled hourly if you wish. At the same time, a page such as your ‘About’ page may only get updated every three months – so you can set coding to three months.
A sitemap is generally used to provide information about web pages, however, you can put together a sitemap that is specific for:
- software source code,
- and geographical (KML).
These each require their own sitemap created in the correct format for each area. Each sitemap should be submitted to the major search engines manually to begin with. Once they know your sitemap exists, they will continue to visit your website and to index pages. It is worth pointing out that a sitemap needs to be updated everytime you add a new page, however, you don’t need to resubmit the sitemap. Even then, search engines clearly state that they may not index every page, even if a sitemap or robots file indicates their presence.
Who needs an XML Sitemap? Really, every website should have an XML Sitemap. New websites and those with dynamic content notice real benefits very quickly. If you use a third party program, then you can create a new properly coded XML Sitemap in minutes. Add a couple of more minutes and you have your sitemap submitted to the major search engines. Program such as WordPress make the process much easier as there are plugins that automatically create a new sitemap every time a new page has been created.
Add an XML Sitemap to your website, submit the sitemap, then watch as more of your pages are indexed each day.