Do I need a robots txt file for my simple one page website?

Question

I have a simple one page text only website. I went to submit it to "Scrub the Web" search engine, and the "meta tag analizer" said I need to construct a "robots txt file" for my website. Should I do it?

Answer 1

You don't need a robots.txt file at all . . . it's optional. And since your site is text only, and you want a search engine to find it, you definitely don't need it. The robots.txt file is to stop search engine crawlers from getting tangled in parts of the site meant only for humans (like the forms for answering questions, or signing up for new accounts, that sort of thing). Also, if you're on Google and Yahoo you probably don't need to sign up anywhere else. Have a nice day!

Answer 2

You may as well have one in any case. If not you'll see plenty of 404 errors in your logs from spiders requesting the file that isn't there. It's actually used not just for blocking crawlers but to allow or disallow certain search engines or certain directories on the site.

It can also be used to provide a crawler with a sitemap file so that it can find and index parts of your web site that may be more difficult to find.

Here's a robots generator;

http://www.mcanerin.com/EN/search-engine/robots-txt.asp