Hello,
I am having problems with web server(s) getting bogged down by google bots. I am looking for more information and help on this specific topic.
The site that the google bots are looking up is HUGE- dozens of pages with hundreds of links. Google appears to be crawling the site at 5 minute intervals and all of this indexing (or whatever its doing) is taking up resources nad costing big money. I am afraid if I put a robots.txt file that the site will not come up as high on the google search page.
Any help or info? I have found plenty of info that tells me "this is what a web crawler is" but nothing goes past that.
All you computer gurus please help!!!!
2007-07-09 05:56:24 · 3 answers · asked by Anonymous in Computers & Internet ➔ Internet ➔ Google
If you have a Google account and if you have submitted your site to them then go into 'Webmaster Tools, and click on your site.
On the left side of the page you will see a number of options and one of them is 'Tools' and under that 'Crawl Rate'.
Click on that and you will get an option to vary the crawl rate of your site.
If you choose slower, it will reduce the load on your webserver.
Regards, Zyfert
http://webhosting.cybersprout.com
2007-07-09 13:27:19 · answer #1 · answered by Zyfert 7 · 1⤊ 0⤋
The best thing you can do is to create a google sitemap for your site. The sitemap is an xml file that contain links to all your sites pages. IF you site is dynamic you can probably automate the process of creating a sitemap. In the sitemap you can list how frequently you want individual pages to be crawled thereby reducing your bandwidth wastage.
There are other advantages like the fact that googlebot will not have to crawl your entire website each time it visits. The sitemap is supposed to list the last update time of each page and so googlebot will only crawl the updated pages.
2007-07-09 13:13:44 · answer #2 · answered by Abdussamad 2 · 0⤊ 0⤋
Short of disallowing the google bots, there is not a lot you can do about this. To be honest, most web masters would give their right arm to be getting so much attention from Google. And you're right, if you put up a robots.txt, your ratings will plummet!
Live with it and make sure you have enough server space to cope.
2007-07-09 13:10:22 · answer #3 · answered by liverpoolscousermarch 5 · 0⤊ 2⤋