robots.txt settings

Used for the Sitemap line. https:// is added if missing and any path is removed.

Preset

One per line. A leading / is added if missing.

Google ignores Crawl-delay. Some other crawlers, such as Bing, respect it.

AI crawlers

Tick a crawler to block it with its own Disallow: / group. Blocking AI crawlers may reduce how often your content appears in AI answers; whether to block is your call.

Your robots.txt

Upload to your site root so it loads at yourdomain.com/robots.txt. Many SEO plugins edit this file under SEO → Tools.

Runs in your browser. Nothing you enter is uploaded or stored.

How it works

Four steps. No signup.

01

Enter your website address

Used for the sitemap line and to resolve page paths.

10 seconds
02

Pick a preset

WordPress, allow everything, block everything or custom.

10 seconds
03

Adjust the rules

Add paths to block or allow, and choose any crawlers to block.

1 minute
04

Copy or download

Upload the file to your site root, or enter the rules in your SEO plugin.

2 minutes
Questions

Questions about robots.txt.

Straight answers about this tool. If yours isn't here, ask me directly.

Average response·< 1 business day

What is a robots.txt file?

It is a text file at the root of your site that tells crawlers which parts they may visit. It guides crawling, but it is not a security tool.

What does the WordPress preset do?

It blocks the admin area, keeps admin-ajax.php available, blocks internal search pages and adds your sitemap line.

Will blocking a page remove it from Google?

Not always. A blocked page can still be indexed if other sites link to it. Use a noindex tag to keep a page out of results.

Should I block AI crawlers?

That is your call. Blocking them can stop your content being used for training, and may reduce how often you appear in AI answers.

Do I need a sitemap if I use an SEO plugin?

No. Plugins such as Yoast and Rank Math create one automatically. This tool is useful for small or static sites.

Is my data stored?

No. The files are built in your browser.

The hosting I use

I host my own sites on Hostinger.

In my experience it's one of the best hosting options for small business websites, and it's the one I suggest. If you sign up through my link, I earn a small commission at no extra cost to you.

Check Hostinger plansAffiliate link
Free tools, plus a free website audit

Want help with the rest of your website?

These tools are free to use. If your site needs more than a new icon or better markup, I design and build custom websites for small businesses, and I offer a free audit that shows you what to fix first.