robots.txt Generator/Validator
User-agent: * Disallow: /admin/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
Build a robots.txt file visually — add one or more user-agent groups with their own Allow/Disallow rules, list your sitemap URLs, and optionally set a crawl delay — or switch to Validate mode and paste in an existing robots.txt to check it for common mistakes like malformed directives or relative sitemap URLs.
Frequently asked questions
Where does robots.txt need to live?
It must be placed at the root of your domain, exactly at /robots.txt (e.g. https://example.com/robots.txt) — search engines won't look for it anywhere else.
Does robots.txt actually block pages from appearing in search results?
Not reliably on its own — Disallow tells well-behaved crawlers not to fetch a URL, but a disallowed URL can still be indexed (without its content) if other sites link to it. To truly keep a page out of search results, use a noindex meta tag or HTTP header instead.
What does the validator check for?
It flags lines that don't follow the 'Directive: value' format, unknown directives, Allow/Disallow rules that appear before any User-agent line, paths missing a leading slash, non-absolute Sitemap URLs, and User-agent groups with no rules underneath them.