Robots.txt

Robots.txt

Also known as robots exclusion protocol, Robots.txt is a text file stored in a site’s root directory that tells a search engine crawler which site pages and sub-folders should not be included in the search engine Index. However, there is no guarantee that a crawler will comply with this request. robots.txt is an alternative to a meta robots tag or password protection.

Back to top