By Shad Super in Questions —

What Is the Syntax Structure of the robots.txt File, and How Can I Properly Format Directives for Different User Agents?

Summary

The robots.txt file is used to manage and control the behavior of web crawlers or bots on your site. This guide outlines the syntax structure of the robots.txt file, detailing the correct formatting of directives for different user agents to ensure optimal and efficient bot management.

Basic Syntax Structure

The robots.txt file is placed in the root directory of your website and consists of one or more groups of directives. Each group has a User-agent line followed by one or more Disallow or Allow directives.

User-agent

The User-agent directive specifies the name of the bot to which the subsequent rules apply. To apply rules to all bots, use an asterisk (*).

User-agent: *

Disallow

The Disallow directive tells the specified bot which directories or files it should not crawl. An empty Disallow value means the bot is allowed to crawl all pages of the site.

Disallow: /private/

Allow

The Allow directive is used to allow crawling of a subdirectory or page inside a disallowed directory.

Allow: /public/page.html

Example Structure for Different User Agents

Here’s a practical example demonstrating how to create rules for different user agents:

# Block all web crawlers from accessing the private directory
User-agent: *
Disallow: /private/

# Allow Googlebot to access a specific page in the private directory
User-agent: Googlebot
Allow: /private/public-page.html

# Do not allow Bingbot to access the entire site
User-agent: Bingbot
Disallow: /

Special Directives

There are special directives such as Sitemap, Crawl-delay, and others that provide extended functionalities:

Sitemap

The Sitemap directive informs bots where your XML sitemap is located, which helps them discover your site’s URLs more efficiently.

Sitemap: https://www.example.com/sitemap.xml

Crawl-delay

The Crawl-delay directive specifies the number of seconds a bot should wait between requests to the server. This is especially useful for reducing server load from web crawlers.

User-agent: Bingbot
Crawl-delay: 10

Comments

Comments can be included in the robots.txt file by using the # symbol. These lines are ignored by the web crawlers and can be used to annotate the file for human readers.

# This is a comment
User-agent: *
Disallow: /tmp/

Best Practices

Testing

After creating or modifying your robots.txt file, it’s important to test it. Google provides a Robots.txt Tester tool which can be used to ensure that your file is correctly formatted and functions as expected.

Keep It Simple

Maintain a clean and simple structure in your robots.txt file, avoiding overly complex rulesets which could lead to unintended behavior.

Regular Updates

Review and update your robots.txt file periodically, especially after significant website updates, to ensure it continues to function correctly and reflects your current crawling preferences.

References