what do robots.txt files do and do i need on on my site www.comberonline.org?

Question

some say i need it for search engines, but what does it do? and can i do with out it

Answer 1

robots.txt tell the web crawlers how to deal with your site. The simplest form is tell the search engines to ignore certain content on the site. It can be quite specific telling different crawlers different things to ignore or use.

Answer 2

It stops Search Engine Spiders from indexing the pages listed in the file, comes in handy if you have things on your site that you dont want a search engine to list, an admin or members only area for example

Answer 3

As stated above, robots.txt tell web robots (spiders whatever you want to call them) What parts of your web site they can and can't index for search engines. There is more info on them at http://www.robotstxt.org or
http://en.wikipedia.org/wiki/Robots.txt

Answer 4

Well the only way to find out if you can go without it, is back it up onto your computer, then delete it, if nothing screws up, then you didn't need it,, if anything did screw up, then you do need it, so upload it back to the server.