Skip to content
Allin
Sign up · Sign in

Robots.txt generator

Build a correct robots.txt file from structured fields: the crawler to target, the paths to allow and disallow, one or more sitemaps (relative paths are made absolute), and optional crawl-delay and host lines. It manages how search engines crawl your site — remember it is a crawl directive, not a security barrier.

Related tools

All SEO tools →
Hreflang Tag GeneratorGenerate rel=alternate hreflang link tags for multilingual sites, with presets and URL auto-fill.Keyword Density CheckerPaste text to see word counts and the most frequent 1, 2 and 3-word phrases with their density.Meta tags generatorGenerate SEO and Open Graph meta tags for your pages.Schema Markup GeneratorGenerate valid JSON-LD structured data for Article, Product and FAQ pages.Slug generatorTurn any title into a clean, URL-friendly slug.UTM builderBuild tracked campaign URLs with utm_source, medium and campaign.

Robots.txt generator works straight from this page — free, instant, nothing to install. It covers the crawler to target, the paths to allow and disallow, one or more sitemaps (relative paths are made absolute), and optional crawl-delay and host lines. It manages how search engines crawl your site — remember it is a crawl directive, not a security barrier — adjust any of them and the result follows immediately.

How to use it

  1. Open the tool — no signup or install needed.
  2. Enter your input or adjust the available options.
  3. Get your result instantly, then copy or download it.

Frequently asked questions

What is Robots.txt generator?

Build a correct robots.txt file from structured fields: the crawler to target, the paths to allow and disallow, one or more sitemaps (relative paths are made absolute), and optional crawl-delay and host lines. It manages how search engines crawl your site — remember it is a crawl directive, not a security barrier.

What does a concrete case look like?

User-agent: * · Disallow: /admin/ · Sitemap: https://example.com/sitemap.xml — the tool shows every step in between, not just the final figure.

What does it take into account?

It factors in the crawler to target, the paths to allow and disallow, one or more sitemaps (relative paths are made absolute), and optional crawl-delay and host lines. It manages how search engines crawl your site — remember it is a crawl directive, not a security barrier. Change any of them and the output follows immediately.

When would I actually use this?

Before publishing: checking a title and description will not be truncated, tagging a campaign link consistently, and spotting keyword stuffing before a reader does.

What is the most common mistake?

Optimising for a character count. Search engines truncate by pixel width, not by character, so a title of capitals and wide letters is cut well before one of the same length in narrow ones.

What else is filed next to it?

Hreflang Tag Generator, Keyword Density Checker and Meta tags generator share its section. They are not variants of it — being filed together is not the same as being alike — but that is where to look if this turned out not to be the tool you wanted.

Where do the figures come from?

Length guidance reflects what search results have displayed in recent testing, not a documented limit — engines rewrite titles freely and the thresholds move. Treat them as a safe range, not a rule.

Further reading

All guides →
How-toHow to Write a robots.txt: Directives, Matching, and What It Cannot HideFour directives, two wildcards, one file at the host root. It is a crawl instruction and nothing more — it does not remove a page from search results, it does not restrict access, and it publishes every path you list in it.GuideHTTP Status Codes Explained: The Ones That Actually Get Confused301 against 308, 302 against 307, 401 against 403, 404 against 410 — plus what Retry-After on a 429 or a 503 actually promises. The pairs where picking the wrong code changes behaviour, not just wording.GuideWhat Makes a URL Slug Good: Stability, Readability and the Conflict Between ThemA slug has two jobs that pull against each other: it is a permanent identifier and it is a piece of readable text. Length, hyphens, stop words, dates, non-ASCII characters and the identifier-plus-slug pattern that gets both properties — with the real numbers from a site that localises 1,736 tool slugs into six languages.GuideTitle Tags, Meta Descriptions, and What Search Engines Do With ThemWhat Google's own documentation says about rewriting title links and about the meta description, rather than what SEO folklore says. Then the measurable part: titles are truncated by pixel width, so two titles of exactly sixty characters can render 204.55 pixels apart and only one of them survives.ExplainerKeyword Density Is a Dead Metric, and What Replaced ItDensity counted occurrences because retrieval once counted occurrences. TF-IDF, then BM25 with its saturation curve, then embeddings replaced it. Here is the same 800-word page scored three ways, and why the three disagree.GuideStructured Data: What Search Engines Actually Use It ForSchema.org markup buys eligibility for a rich result, never a guarantee and never a ranking boost. Here is which types still produce something visible in 2026, what each one requires, and the one rule that gets sites penalised.