Can i impersonate a search engine identity to access some web pages that can't be accessed by a common user?

Question

Have you heard of cloaking?

Answer 1

Cloaking is a technique used in Search Engine optimisation by webmasters to fool search engine bots. They plan to show a different page to a different search engine bot, because they believe certain layouts will rank higher in Google than say in MSN, so depending on which bot they detect visiting their site they will show a different page.

How do they detect what bot is visting? By looking at the user agent string!

When someone visits a site, typically the client will send a user-agent string. For most normal users using say Internet explorer 6.0 it will be.

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)

Googlebot will be for example Googlebot/2.1 (+http://www.googlebot.com/bot.html) among others

You can try to change the user agent that your client sends , a method known as user agent spoofing, so for example you can pretend to be googlebot.

With firefox you can do it manually as http://johnbokma.com/mexit/2004/04/24/changinguseragent.html
or you can install the extension useragent switcher

With Internet explorer it is harder, among other methods you can use a local web proxy like proxomitron or privoxy to change it.

Answer 2

Erm, you can't

Seatch engines USE the same pages we do, they just keep an index.