Neatbo.

robots.txt rule tester

Test pasted or imported robots.txt rules for one URL and crawler token, with an allow decision, selected group, winning rule and line number.

Browser-local processingInputPasted robots.txt or UTF-8 .txt file + URL and crawler tokenOutputAllow decision and JSON rule reportUp to 1 MiB per file · File limit: 1
  1. 1Add input
  2. 2Adjust settings
  3. 3Get your result

Tool input and files are processed in this browser without being uploaded.

Your input

Inputs are kept temporarily in this tab when switching tools. Refreshing or closing clears them; large results may need to be regenerated.

⌘ / Ctrl + Enter to run
robots.txt source
0 characters · 0 bytes
Options

Complete the required options first. You can keep the defaults for the rest.

Enter a crawler product token such as Googlebot, not a full browser User-Agent string.

Preparing the tool…

Before you start

Test pasted or imported robots.txt rules for one URL and crawler token, with an allow decision, selected group, winning rule and line number. Paste rules or choose a UTF-8 .txt file, then enter a complete URL and crawler product token.

How to use this tool

  1. Paste robots.txt or choose a UTF-8 .txt file; enter a complete URL and crawler product token.
  2. Run the test and read the allow decision, selected group and winning rule with its source line.
  3. Download JSON for every matching rule and ignored line number. Check the live site separately before relying on this decision.

Supported inputs and limits

Tests groups, wildcards, end anchors and UTF-8 path matching locally for one HTTP(S) URL. It does not fetch robots.txt, verify its host or server status, or prove indexing.

Paste up to 1 MiB or import one UTF-8 .txt file up to 1 MiB, with at most 5,000 lines. Unsupported or invalid lines are listed; complex matching has a calculation limit.

Worked example

Example input

User-agent: *
Disallow: /private/
Allow: /private/public/
Example options
{"agent": "ExampleBot", "url": "https://example.com/private/file"}

Example output

{
  "allowed": false,
  "path": "/private/file",
  "agent": "examplebot",
  "implicitRobotsTxt": false,
  "selectedGroups": [
    {
      "agents": [
        "*"
      ],
      "line": 1
    }
  ],
  "matchedRule": {
    "allow": false,
    "pattern": "/private/",
    "line": 2
  },
  "matchedRules": [
    {
      "allow": false,
      "pattern": "/private/",
      "line": 2
    }
  ],
  "ignoredLines": []
}

When something does not work

Tests groups, wildcards, end anchors and UTF-8 path matching locally for one HTTP(S) URL. It does not fetch robots.txt, verify its host or server status, or prove indexing. Paste up to 1 MiB or import one UTF-8 .txt file up to 1 MiB, with at most 5,000 lines. Unsupported or invalid lines are listed; complex matching has a calculation limit.

Frequently asked questions

How do I find the matching robots rule?

Run the test and read the allow decision, selected group and winning rule with its source line. Download JSON for every matching rule and ignored line number. Check the live site separately before relying on this decision.

What should the example produce?

/private/file is blocked; /private/public/file is allowed by the more specific rule. Equivalent Allow and Disallow rules favor Allow.

Which cases are outside its scope?

Tests groups, wildcards, end anchors and UTF-8 path matching locally for one HTTP(S) URL. It does not fetch robots.txt, verify its host or server status, or prove indexing. Paste up to 1 MiB or import one UTF-8 .txt file up to 1 MiB, with at most 5,000 lines. Unsupported or invalid lines are listed; complex matching has a calculation limit.

Documentation & further reading

Website configuration guide: meta tags, robots, sitemaps and CSP →SHA256SUMS and JWT signatures: what verification can prove →

Related tools