How to Block Pages and Directories with robots.txt

When a search robot visits a website, it first looks for the robots.txt file. This text file is located in the root directory of the site, next to the main index file (for example, in public_html in many control panels). It contains instructions for search robots.

With robots.txt you can ask robots not to crawl certain folders or pages, set crawling rules for specific robots and point them to your sitemap.

Important: Disallow blocks crawling, not indexing. If other sites link to a blocked page, it can still appear in search results without a description. To keep a page out of search results, use <meta name="robots" content="noindex"> on the page and do not block it in robots.txt, otherwise the robot will never see this tag.

If the site has no robots.txt file, create it. Two directives are used to block crawling: User-agent and Disallow.

User-agent: NAME_OF_THE_ROBOT
Disallow: /         # block the entire site
Disallow: /page/   # block only /page/

Examples

Block the site for Bingbot

User-agent: Bingbot
Disallow: /

Block the site for Yandex

User-agent: Yandex
Disallow: /

Block the site for Googlebot

User-agent: Googlebot
Disallow: /

Block the site for all robots

User-agent: *
Disallow: /

Block the cgi-bin and images folders for all robots

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/

Allow crawling of the whole site

An empty robots.txt file has the same effect as these lines:

User-agent: *
Disallow:

Allow only Google and Yandex, with a crawl delay for Yandex

The following file blocks all robots except Googlebot and Yandex. The Crawl-delay directive asks the robot to wait 4 seconds between requests. Google ignores Crawl-delay, so it is set only for Yandex:

User-agent: *
Disallow: /

User-agent: Yandex
Crawl-delay: 4
Disallow:

User-agent: Googlebot
Disallow:

Point robots to your sitemap

Sitemap: https://example.com/sitemap.xml

After editing robots.txt, check it with the robots.txt report in Google Search Console.