# Robots.txt Generator: Allow, Disallow, and Sitemap Without Guessing the Syntax

> A bad robots.txt can hide your shop or invite crawlers into staging. Generate a valid file with allow, disallow, and sitemap — then put it at the site root.

- **URL**: https://brytetools.com/post/robots-txt-generator-guide
- **Markdown summary**: https://brytetools.com/llms/posts/robots-txt-generator-guide.md
- **Meta description**: Generate robots.txt with allow, disallow, crawl-delay, and sitemap. Free SEO tool — copy or download, no account.

## Article

Staging got indexed because robots.txt lived in a subfolder. Or the live shop blocked /products during a migration and traffic fell off a cliff. The file is small. The blast radius is not.

A robots.txt generator will not replace a crawl strategy. It will stop you shipping a typo in Disallow at 2am.

## Where the file must live

Crawlers look at the site root: https://yoursite.com/robots.txt. A file at /blog/robots.txt is ignored for the host. Subdomains need their own file.

## Directives that actually get used

- User-agent — which bot the rules apply to (* for everyone)

- Disallow — paths you ask well-behaved bots not to crawl

- Allow — exceptions inside a disallowed tree (support varies)

- Sitemap — absolute URL to your XML sitemap

- Crawl-delay — not honoured by Google; some other bots still read it

Robots.txt is a request, not a lock. Private pages need auth, not Disallow.

## Generate a valid file in the browser

Use [BryteTools Robots.txt Generator](https://brytetools.com/tools/robots-txt-generator) to add allow/disallow rules, optional crawl-delay, and a sitemap URL. Copy or download. No account.

Typical production pattern: allow the public site, disallow admin, cart, and search-result junk, point Sitemap: at the real sitemap. Typical staging pattern: Disallow: / for User-agent: * so the duplicate property does not rank.

## Rules that backfire

RuleWhat happens

Disallow: / on productionGoogle may drop URLs from the index over time

Blocking CSS/JS needed to renderPoor rendering, messy Search Console

No sitemap lineYou rely only on links and luck

Robots as securityAnyone can still fetch the URL

## After you deploy the file

Confirm it returns 200 at the root. Add the sitemap in Search Console too — the robots line is a hint, not a substitute. Pair with [Meta Tag Inspector](https://brytetools.com/tools/meta-tag-inspector) so you are not blocking a URL that also has a contradictory noindex story you did not intend.

## FAQ

Where do I put robots.txt?Site root. Subfolder paths are ignored by crawlers looking for the root file.

Can robots.txt hide private pages from people?No. It only asks well-behaved bots not to crawl. Use authentication for private content.

Does Google honour Crawl-delay?No. Some other bots do. Do not rely on it for Googlebot.

Should I block parameters and faceted search?Often yes, if they create duplicate thin URLs. Test with a crawl of a staging copy first.

Is my site configuration stored?No. Generation runs in your browser.

## Try it now

[Generate robots.txt free →](https://brytetools.com/tools/robots-txt-generator)

Copy, deploy at the root, then verify the URL in the browser.

## Site context

BryteTools provides free private browser tools. See https://brytetools.com/llms.txt for the full catalog.