1 · Your site

The main address, without a page name. Used for the sitemap link and the sitemap file.

2 · robots.txt

Start from

AI and data crawlers

Tick the ones you want to ask to stay out. Well-behaved crawlers follow this. It does not stop Google or Bing search.

3 · sitemap.xml

Full addresses or paths such as /about/. Only list pages you want found in search.
Last modified date
Only add a date if it is true. Google ignores dates that are always changing.

Your files




Runs in your browser. Nothing you type or upload is stored or sent.
How it works
Four steps. No signup.
01
Pick a starting point
WordPress, open to everything, or closed to everything.
10 seconds
02
Adjust the rules
Add paths to block or allow, choose AI crawlers to block and add your sitemap address.
2 minutes
03
Add your pages
Paste your page addresses to build an XML sitemap.
2 minutes
04
Copy or download
Upload robots.txt and sitemap.xml to your site's root folder.
2 minutes
Questions
Robots.txt questions, answered.
Straight answers about crawling and sitemaps. If yours is not here, send it to me directly.
What does robots.txt do?
It tells crawlers which parts of your site they may visit. It is a request that well-behaved crawlers follow, not a lock, so never use it to hide private information.
Will blocking a page remove it from Google?
Not reliably. A blocked page can still appear in results if other sites link to it. To keep a page out of results, use a noindex tag instead and leave the page crawlable.
Should I block AI crawlers?
That is your choice. Blocking them asks those crawlers not to use your content. It does not affect your Google rankings, but it may reduce how often AI tools mention your site.
Where do the files go?
In the root of your domain, so they load at yourdomain.com/robots.txt and yourdomain.com/sitemap.xml. If an SEO plugin already makes them, use its settings instead of uploading files.
Do I need a sitemap?
Small sites are found without one, but a sitemap helps search engines discover new pages faster. Submit it in Google Search Console.
Can a wrong robots.txt hurt me?
Yes. A single line that blocks everything can remove your site from search over time. Check the warnings in the tool before you publish.
Free tools, plus a free website audit
Want help with the rest of your website?
These tools are free to use. If your site needs more than a quick fix, I design and build custom websites for small businesses, and I offer a free audit that shows you what to fix first.