Can robots.txt actually prevent a page from appearing in Google?

Not reliably on its own — robots.txt blocks crawling, but a blocked page can still get indexed from other signals like links to it. For guaranteed removal, use a noindex meta tag instead.

Where does robots.txt need to be placed on my site?

At the root of your domain (yoursite.com/robots.txt) — search engines only check that exact location.

Can I set different rules for different search engine crawlers?

Yes, you can specify rules for individual user agents (like Googlebot) separately from the general wildcard rule.

Does Robots.txt Generator upload my files to a server?

No. Robots.txt Generator runs entirely in your browser using on-device processing, your files never leave your device.

Robots.txt Generator

Generate a robots.txt file to control search engine crawling.

Runs in your browser

Allow all crawlers by defaultDisallowed paths (one per line)Sitemap URL (optional)

robots.txt

User-agent: *
Allow: /
Disallow: /admin
Disallow: /api

About this tool

A robots.txt file tells search engine crawlers which parts of your site they're allowed to access, using a specific plain-text syntax that's easy to get subtly wrong — a misplaced rule can accidentally block your entire site from being indexed. This tool generates a correctly formatted robots.txt file from simple options you choose, reducing the risk of a syntax mistake that silently affects how your whole site gets crawled.

Frequently asked questions

Sitemap Generator

Generate an XML sitemap from a list of page URLs.

Meta Tag Generator

Generate title, description, Open Graph and Twitter Card meta tags.

SERP Preview

Preview how your page will look in Google search results.

Open Graph & Twitter Card Preview

Preview how a link will look when shared on Facebook, LinkedIn and X.

Schema Markup Generator

Generate JSON-LD structured data for FAQs, products, articles and more.

JSON-LD Validator

Check JSON-LD structured data for common errors.

Web Manifest Generator

Generate a web app manifest.webmanifest file.

Meta Title & Description Checker

Check whether your title and meta description fit Google's recommended length.

Canonical URL Checker

Check a page's HTML for a missing, duplicate, or mismatched canonical tag.

Aspect Ratio Calculator

Simplify an aspect ratio and scale dimensions to a target width or height.

Keyword Density Checker

Find how often each word appears in your content, as a percentage.

Hreflang Tag Generator

Generate hreflang link tags for multi-language and multi-region pages.

PDF Tools

Merge, split, compress and convert PDF files.

Image Tools

Compress, resize, crop and convert images.

Link & QR Tools

Shorten links and generate QR codes and barcodes.

Audio Tools

Convert audio files and turn text into speech, or speech into text.

Video Tools

Compress and process video files, right in your browser.

Developer Tools

JSON, encoding, hashing and text utilities for developers.

Data & Spreadsheet Tools

Clean, convert and inspect CSV, TSV and tabular data.

File Tools

Archive, inspect and convert files, right in your browser.

Business Tools

Invoices, quotations, receipts and other business documents.

Social Media Tools

Resize images for social platforms, count hashtags and preview posts.