Start free (+500 credits)Talk to a data expert

Robots.txt Analyzer

Parse robots.txt rules and test whether a URL path is allowed for scraping.

robots.txt content

Path to testUser-agent

Allowed · No matching rule (default allow) (agent: *)

Parsed rules

[
  {
    "userAgent": "*",
    "allow": [
      "/public/"
    ],
    "disallow": [
      "/admin/"
    ],
    "crawlDelay": 1,
    "sitemap": [
      "https://example.com/sitemap.xml"
    ]
  }
]

Sitemaps

https://example.com/sitemap.xml

What is this tool?

Parse robots.txt rules and test whether a URL path is allowed for scraping.

How to use

Paste your input in the widget above. Everything runs locally in your browser, no data is sent to Piloterr servers.

Related tools

scraping parser
sitemap url extractor

Piloterr resources

Scraper APIs
Website Crawler API
Website Rendering API
What is web scraping?
All scraping tools

Start free (+500 credits)Browse API library