The Sitemap
Those who don’t practice SEO or study how search engine spiders work rarely see the use of a properly configured, well-maintained sitemap. After all, most human beings would rather use a search utility or organized navigation system rather than sift through links to every document on a website.
However, sitemaps are of vital importance to proper indexing. Search engine spiders find pages by following links. If you navigational system is comprehensive, then there should be no need for a sitemap, as every page on your site will be reachable. However, for older and/or larger sites, a sitemap helps to ensure that the spiders find and crawl every page, even those that may no longer be accessed through standard navigation. It can be a chore, but maintaining a complete sitemap could mean the difference between a page finding its way into indexes or dropping out of visibility entirely.
On that note, Google recently came out with a new service geared toward helping webmasters control how their sites are indexed without resorting to convoluted robots.txt files. Google Sitemaps allows users to configure XML feeds that tell Google where all of their pages are, how frequently they are updated, when they were last updated, and so forth. And since Google holds anywhere from 50% to 75% of the search market, depending on who you talk to, many webmasters would do well to latch on to this service before the competition does.