Web Blog of Greg Mathews

Before I start please let me clarify, when I ask the question ‘What is the point of a Google Sitemap?’ I am refering to the Google Sitemap and not an HTML sitemap you have on your site for your visitors to browse the site easier.

I have always beleived that a Google sitemap is of use when launching a rapidly growing site, I do not think they are beneficial to sites with last than half a dozen pages or so.

I launched a site back in October 2006 and regularly submited a sitemap containing all of the pages on the site, this sitemap was changing on a daily basis as the site grew. Within 10 days the site was indexed and showing well in the Google SERPs, (Search Engine Results Pages) far quicker than I ever expected or would have seen without the use of a Google Sitemap in my opinion.

Things went well for the first couple of months until Google appeared to Drop the site from its index in January, but this appeared to be a glitch and within a couple of weeks the site started to appear in the SERPs again.

Then this month (March 2007) a similar situation happened, although using the site command on the domain in Google still showed over 100,000 pages in the index only about 50 where included, the rest never appear, not even in the Suplimental results.

When I looked at the SERPs for my site it appeared the only pages indexed where the html sitemaps, which are just pages of links to other pages on the site. When I checked my server logs it these where also the only html pages being crawled by the Googlebot. Other files Googlebot was also crawling where for a cgi script that the html pages had in the form of <script ‘type=’text/javascript’ xsrc=’url to script’> which is just a logging script that returns nothing.

From this I am assuming that Google is now no longer using my Google Sitemap but instead are doing a more thorough crawl of my site by following the links, both internal and external. This is a good thing in that it means real pages with links to them will be indexed and obscure pages will not, however I do have the following questions :

1. Did Google need to remove all of my pages from the SERPs before changing the method which they use to find pages on my site? this has cost me about 75% of my traffic/income.

2. Why oh Why is Googlebot following javascript links? each of my 100,000+ pages has the same javascript link but each with a different parameter being passed to it in the url, this means that the Google bot is crawling the same script 1000’s of times. I have blocked Googlebot from crawling this script now via the robots.txt file but this could have implications as the script is used for other tasks (non Javascript) which I am happy for Google to crawl.

3. Now that my site is established, do I need to continue using/creating a Google Sitemap? What is the point?

It’s been a while I know but I thought I would post about Yzoo.co.uk

In January I posted here about the site being dropped by Google and not being listed in the SERP’s. Well things did pick up again and January matched novembers traffic and earnings - Great. February did even better, reaching more than 1,000,000 (one million) visitors and matched the earnings of December, well February beat Decembers earnings by a few dollars.

Read the rest of this entry »