Why would a path be blocked even though I don't see a rule for it directly?

A broader wildcard rule earlier in the file can block a path even without an exact match — rule precedence and pattern matching aren't always obvious just from reading the file.

Can different user agents get different results for the same path?

Yes, if your robots.txt has user-agent-specific rules, the same path can be allowed for one crawler and blocked for another.

Does this test against my live robots.txt, or one I paste in?

You paste in the robots.txt content to test, so you can check changes before actually publishing them to your live site.

Does Robots.txt Tester upload my files to a server?

No. Robots.txt Tester runs entirely in your browser using on-device processing, your files never leave your device.

Robots.txt Tester

Paste a robots.txt file and test whether a path is allowed for a given user agent.

Runs in your browser

robots.txt content

Path to testUser-agent

Allowed — * can crawl /blog/post

About this tool

A robots.txt rule that looks correct can still block or allow paths differently than you intended, since crawler rule matching (wildcards, rule precedence, user-agent-specific blocks) doesn't always work the way it reads at a glance. This tool tests whether a specific path is allowed or blocked for a given user agent against your actual robots.txt rules, confirming what a crawler would really do rather than assuming from reading the rules yourself.

Frequently asked questions

Robots.txt Generator

Generate a robots.txt file to control search engine crawling.

Sitemap Generator

Generate an XML sitemap from a list of page URLs.

Meta Tag Generator

Generate title, description, Open Graph and Twitter Card meta tags.

SERP Preview

Preview how your page will look in Google search results.

Open Graph & Twitter Card Preview

Preview how a link will look when shared on Facebook, LinkedIn and X.

Schema Markup Generator

Generate JSON-LD structured data for FAQs, products, articles and more.

JSON-LD Validator

Check JSON-LD structured data for common errors.

Web Manifest Generator

Generate a web app manifest.webmanifest file.

Meta Title & Description Checker

Check whether your title and meta description fit Google's recommended length.

Canonical URL Checker

Check a page's HTML for a missing, duplicate, or mismatched canonical tag.

Aspect Ratio Calculator

Simplify an aspect ratio and scale dimensions to a target width or height.

Keyword Density Checker

Find how often each word appears in your content, as a percentage.

PDF Tools

Merge, split, compress and convert PDF files.

Image Tools

Compress, resize, crop and convert images.

Link & QR Tools

Shorten links and generate QR codes and barcodes.

Audio Tools

Convert audio files and turn text into speech, or speech into text.

Video Tools

Compress and process video files, right in your browser.

Developer Tools

JSON, encoding, hashing and text utilities for developers.

Data & Spreadsheet Tools

Clean, convert and inspect CSV, TSV and tabular data.

File Tools

Archive, inspect and convert files, right in your browser.

Business Tools

Invoices, quotations, receipts and other business documents.

Social Media Tools

Resize images for social platforms, count hashtags and preview posts.