Crawler

Crawler

A crawler is also known as spider or robot, it’s a search engine program that literally “crawls” the web, collecting data, following links, making copies of new and updated sites, and storing millions of URLs in the search engine’s index. This allows the search engines to respond with faster and more relevant or timely listings.

Back to top