Waste data Wath is an error "200"?

Question

I had problem with my server. It replies error "200" for every file that bots try to find. The problem is that when bots looking for robots.files server says 200 and because we don't have a robots.txt files bots said we will back later crawling your contents.

Answer 1

Best Answer - Chosen by Asker

I had a similar problem. Server reply 200 so it says that's ok there is no problem with the file. But the files doesn't exist...
SEO strategies are “the talk of the web”. All companies may make up their mind to that. We did it.
International waste exchange is a very important service for International community, up to 10.000 companies use it every month and we didn’t understand why www.wastexchange.co.uk was no in the top of all search engines search results.
We had noticed that our waste management web site was not listed well by Google. The most part of waste offers and waste requests published on www.wastexchange.co.uk was not listed. We waste d a lot of time understanding why. Finally we tried the “Google sitemap” service and we understood something more. First of all we understood that our waste management site was hidden to crawlers because we didn’t have a “robots.txt” and a sitemap.xml files. Google Bots that had come into our waste management site found that the robots.txt files was not available. According to this fact they decided to list the wastexchange.co.uk contents later when the robots.txt file would be available. For the same reason they was waiting for our sitemap file. That was for two fatcs:
1- We didn’t have any robots.txt or sitemap.xml files
2- Our server answer “200″ that means that a file exist but it’s not reachable.
So Bots said “Well now it’s not reachable, we don’t know where we can go and where we don’t, we’ll back later to list contents of this waste management web site.
We always have thought that robots and sitemap files was not important. We thought that Bots don’t mind about these informations because these are information that web master provides them. We wrong. Even we had submitted both files Google start to list our content.
We verified it by subscribing to Google sitemap service.
Now we know we have to do a lot of work to improve our SEO but we believe we just did something easy but important. I hope.

For more details about the sitemap.xml siles structure check:

http://www.wastexchange.eu

Answer 2

according to this there no error: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html