Google Webmaster Central offers a nice graphing feature so you can see the crawl trends but there is no way to access meaningful data. Studying crawl rate is hugely important because it helps you find out which pages are trusted most by the search engines. The more Google trusts a page the greater the frequency it will re-visit that page. If you build a few deep links and see your crawl rate for the target page suddenly increase you can be confident that the links are highly trusted and that you should try to get some more.
A plugin for Wordpress that allows you to track crawl activity for MSN, Yahoo & Google from your dashboard. Using the plugin, you can:-
- Access crawl data for 3 main search engines
- See the exact times a spider accessed any page on your site
- Monitor your crawl rate over time
- Discover how crawl rate is related to PageRank
If you enjoyed this post, make sure you subscribe to my RSS feed!