If you have a site and it requires a username/password, will bots be able to index the contents?
Also for secure sites, will it be able to index?
2007-03-05 20:46:53 · 5 answers · asked by b1911 2 in Business & Finance ➔ Advertising & Marketing ➔ Search Engine Optimization
Password protected: No.
Secure: Yes, as long as the content isn't behind a password. (Google, for example, is able to crawl at least the initial page, before the login.)
2007-03-06 00:06:14 · answer #1 · answered by strayinma 4 · 0⤊ 0⤋
Secure: yes. Password protected: no. Also, as a rule, I add a robots noindex/nofollow to the login page so as to let the search engine spiders know they shouldn't even try.
2007-03-06 16:38:34 · answer #2 · answered by trentrockport 5 · 0⤊ 0⤋
The use of robots.txt will exclude bots that crawl for major search engines. However, there are many 'guerilla bots' that dont observe these.
2007-03-08 09:12:13 · answer #3 · answered by Megan C 1 · 0⤊ 0⤋
If they could, what would be the sense in the pw protection?
2007-03-06 01:18:24 · answer #4 · answered by thomy8s 4 · 0⤊ 0⤋
Hello Keep reading my blog
http://www.praveenkodur.com/blog/
http://www.praveenkodur.com/
2007-03-07 20:29:22 · answer #5 · answered by Anonymous · 0⤊ 0⤋