Web Crawler visiting a website?

Question

How does the Google crawler crawl the website?

Answer 1

Google crawls the site using a program - which is commonly know as "Google Bot". This program typically reads the url, reads the page and captures the content (after removing all programming related code, common words ).

The captured content is stored in an index which is then used to calculate the importance of your site for a given keyword.