🔎 SEO tools

Robots.txt generator

Choose a starting point, add the paths you want crawlers to skip and your sitemap address, then copy a ready-made robots.txt.

Free · no signup Runs in your browser

Your robots.txt—

How do I create a robots.txt file?

Choose a preset (allow everything, block everything or a typical WordPress site) or write your own rules, add your sitemap address, and copy the text. Save it as robots.txt in the root folder of your site, so that it opens at your-domain.com/robots.txt. It tells crawlers which paths to skip. It is not a way to keep a page out of Google's results.

What a robots.txt file does

A robots.txt file sits at the top of your site and tells well-behaved crawlers which parts they may visit. It is made of groups: a User-agent line naming the crawler (an asterisk means all of them), followed by Disallow lines for paths to skip and Allow lines for exceptions. A Sitemap line points to your sitemap so that crawlers find it.

Google's own documentation is clear about the limits: robots.txt is used mainly to avoid overloading your site with requests, and it is not a mechanism for keeping a web page out of Google. A blocked page can still appear in results if other pages link to it. To keep a page out of results, use a noindex rule or protect it with a password. It is also not a security measure, because anyone can read the file.

It runs in your browser. What you enter is never sent anywhere, and there is no sign-up.

When to use it

  • Setting up a new site
  • Blocking crawlers from admin, cart or search pages
  • Adding your sitemap address so it is easy to find
  • Checking the syntax of rules before you upload them

Mistakes it helps you avoid

  • Blocking the whole site by leaving 'Disallow: /' in place after launch.
  • Using robots.txt to hide private pages: it is public and not a lock.
  • Forgetting the leading slash on a path.
  • Blocking the CSS and scripts a page needs to display, which can stop Google seeing it properly.

How it works

Everything happens in your browser as you type.

  • User-agent: * = the rules below apply to every crawler (or name one, such as Googlebot)
  • Disallow: /private/ = do not crawl paths that start with /private/
  • Allow: /private/ok.html = an exception to a Disallow rule
  • Disallow: (empty) = nothing is blocked
  • Sitemap: https://example.com/sitemap.xml = the full address of your sitemap

The file is plain text and the rules are case-sensitive in paths. A more specific path wins over a shorter one, and Allow wins over Disallow when they are equally specific. This tool builds one rule group; a site can have several groups, one per crawler, which you can add by hand. Wildcards (*) and the end-of-path marker ($) are supported by Google and other major crawlers.

Example: a WordPress site

You run a WordPress site and want crawlers to skip the admin area but still reach the AJAX file some themes need.

  1. PresetWordPress
  2. Disallow/wp-admin/
  3. Allow/wp-admin/admin-ajax.php
  4. Sitemaphttps://www.example.com/sitemap.xml

A file with User-agent: *, Allow and Disallow lines and a Sitemap line, ready to save as robots.txt

Add more paths as your site needs them.

Terms on this page

Crawler
A program that visits web pages to read them, such as Googlebot.
User-agent
The name a crawler uses to identify itself in robots.txt rules.

Common questions

Where do I put the robots.txt file?

In the root of your site, so it opens at your-domain.com/robots.txt. A robots.txt inside a folder is ignored.

Does robots.txt remove a page from Google?

No. Google says it is not a mechanism for keeping a web page out of Google. Use a noindex rule or a password instead.

What does 'Disallow: /' do?

It asks every crawler covered by that group not to visit any page on your site. Use it only on a site that is not ready to be found.

Is robots.txt a security measure?

No. The file is public and only polite crawlers follow it. Never rely on it to hide private pages.

Do I need a Sitemap line?

It is optional, but it makes your sitemap easy for crawlers to find.

What to work out next

Where these numbers come from

Last verified 2026-10-25 The method on this page is checked against the sources above at least once a year. Spotted something out of date? Tell us and we will fix it.

Related tools

Sitemap generator

Paste your page addresses and get a valid sitemap.xml file to upload, with an optional last-modified date.

Open tool

Meta checker

Check the length of a page title and meta description and see a search result preview before you publish.

Open tool

FAQ schema

Write your questions and answers and get FAQPage JSON-LD markup to paste into a page, with a note on when Google shows it.

Open tool

Schema markup generator

Generate JSON-LD structured data for an organization, local business, article or breadcrumb trail, with checks for missing fields.

Open tool

Meta tag generator

Create the title, description, canonical, robots, Open Graph and Twitter card tags for a page, with a share preview, and copy the HTML.

Open tool

Hreflang generator

Generate hreflang link tags, sitemap entries or HTTP headers for the language and country versions of a page, with checks for common mistakes.

Open tool

🔎 SEO tools

Small SEO jobs done in your browser: previews, checkers and generators. Nothing you enter is sent anywhere.

See all 12 tools in this section