Fill in the blank: The primary function of a robots.txt file is_______________.
- to prevent pages from appearing in search results
- to tell search engines about which of your site’s pages you’d like them to crawl
- to prevent your site from being overloaded with requests by crawlers
- to control exactly which pages appear in search results
