Can I use robots.txt to keep a private page out of Google?

No, on both counts. Google says outright that robots.txt is not a mechanism for keeping a web page out of Google, and a disallowed URL linked from any other site can still be listed using nothing but the link and its anchor text. Nor is the page private in any real sense: robots.txt is only a request, obeyed by search engines and ignored by everything with an interest in ignoring it, and the file itself broadcasts the path. For search visibility use a noindex, delivered as a meta robots tag or an X-Robots-Tag header. For actual privacy use authentication. The two are separate problems and robots.txt solves neither.

Do subdomains and http versus https share one robots.txt?

No. The rules apply only to the exact host, protocol and port that served the file, so https://example.com, http://example.com, https://www.example.com and https://api.example.com are four separate scopes needing four separate files — even if they resolve to the same server and the same document root. This is the second most common misconfiguration after putting the file in a subdirectory, where it is never read at all. A practical consequence: if you redirect http to https, the crawler follows the redirect to fetch the file, so the https copy governs both in effect; if you do not redirect, the http origin has no rules and is crawled freely.

I disallowed a page but it is still in the results. What now?

That is the expected behaviour, and the fix is to reverse the order of operations. Remove the Disallow so the page becomes crawlable again, add a noindex — a meta robots tag or an X-Robots-Tag header — and wait for the page to be recrawled and dropped. The reason the original approach failed is circular: while the URL is disallowed the crawler never fetches it, so it never sees any noindex you put on the page, and the listing built from inbound links persists indefinitely. Once the page has fallen out of the index you may reinstate the Disallow if you want to save crawl budget, though at that point it usually buys you very little.

How to Write a robots.txt: Directives, Matching, and What It Cannot Hide

Published 5/21/2026 · 9 min read · Developer tools

Daniel Okonkwo — Front-end developer and tech writer at OneKitly

Web performance · File formats

Checked against 2 sources

View profile →

In short

A robots.txt is a plain UTF-8 text file that must sit at the top-level path of a host, at /robots.txt in lowercase, and it applies only to that exact host, protocol and port — https and http, and every subdomain, each need their own. It contains groups. A group starts with one or more User-agent lines naming the crawlers it addresses, with * meaning all of them, and continues with Disallow and Allow lines whose values are URL path prefixes. A Sitemap line gives the absolute URL of a sitemap and is independent of any group. Google honours only user-agent, disallow, allow and sitemap, and ignores every other field. Two special characters are supported in paths: * matches zero or more of any character, and $ anchors the end of the URL. When several rules match, the most specific one wins, measured by the length of the rule path in octets; when two rules are equally specific, the least restrictive one wins, which means an Allow beats a Disallow of the same length. The critical point is what the file is not. It is a crawl instruction, not access control and not a de-indexing mechanism. A disallowed URL can still appear in search results if other sites link to it, because the link and its anchor text are enough to index the address without fetching the page. A noindex line in robots.txt is not supported and does nothing. And the file is public by definition, so every path you disallow is a path you have advertised.

Four directives, two wildcards, one file at the host root. It is a crawl instruction and nothing more — it does not remove a page from search results, it does not restrict access, and it publishes every path you list in it.

A crawl instruction, not a lock and not an eraser

Google states it plainly: robots.txt is not a mechanism for keeping a web page out of Google, and a page disallowed in robots.txt can still be indexed if other sites link to it. The mechanism is easy to see once stated. Disallow tells a well-behaved crawler not to fetch the URL. It says nothing about whether the address may appear in an index, and a link from another site supplies enough — the URL itself, plus the anchor text pointing at it — to list the page without ever downloading it. The result is the familiar search listing with a bare URL, no title from the page and no description, sitting exactly where the site owner believed they had removed it.

The correct tools depend on what you actually want. To keep a page out of search results, serve a noindex — either a meta robots tag in the head of the document or an X-Robots-Tag response header, which also works for files that have no head, such as PDFs. To keep content away from anyone at all, put it behind authentication; nothing declarative in a public text file has ever restricted access, and RFC 9309 says so outright, calling the protocol no substitute for valid content security measures. And there is a sequencing trap that catches people combining the two: if you disallow a URL, the crawler will never fetch it, will therefore never see the noindex you added to it, and the page can persist in the index indefinitely. Let it be crawled until it drops out, then disallow it if you still want to.

How matching works: prefixes, two wildcards, and the longest rule

Every Disallow and Allow value is matched as a prefix of the URL path, so Disallow: /admin blocks /admin, /admin/, /administrator and /admin-tools alike. Adding a trailing slash narrows it to the directory. Two special characters refine this: * stands for zero or more of any character, and $ anchors the end of the URL, so Disallow: /*.pdf$ blocks every URL ending in .pdf while leaving /report.pdf?download=1 reachable, because the query string comes after the anchor. A trailing wildcard adds nothing — /* is the same rule as /. Paths are case-sensitive, so /Admin and /admin are two different rules, though the directive names themselves are not.

When more than one rule matches a URL, the winner is the most specific one, and specificity here means nothing more sophisticated than the length of the rule path in octets. Disallow: /reports/ and Allow: /reports/public/ both match /reports/public/q3.html; the Allow is longer, so the file is crawlable. Reverse the lengths and the Disallow wins. If two matching rules are exactly the same length, the tie goes to the least restrictive rule, which means the Allow. Order in the file is irrelevant — a Disallow written after an Allow does not override it, and this alone accounts for a large share of robots.txt files that do not behave as their author reads them. Two more limits worth knowing: Google reads at most 500 kibibytes and ignores everything after that, and an empty Disallow value means nothing is blocked, which is the idiomatic way to write an allow-everything group.

Where the file lives, and why it advertises what you meant to hide

The location rule is absolute and admits no configuration. RFC 9309 requires the rules to be accessible in a file named /robots.txt, all lowercase, at the top-level path of the service, and Google adds that the rules apply only to the host, protocol and port where the file is hosted. Read that carefully, because it has consequences people trip over constantly. A file at https://example.com/robots.txt governs nothing on http://example.com, nothing on https://shop.example.com and nothing on https://example.com:8443 — each of those is a separate origin needing its own file. A file placed in a subdirectory is not read at all. And a site behind a CDN or a reverse proxy is only as correct as its routing: if the platform serves its own robots.txt at the root, yours in the application never runs.

Now the blunt part. The file is fetched by anyone who asks, has no authentication, and is by far the first thing an automated scanner reads. Writing Disallow: /internal/backup-2019/ does not hide that directory — it publishes its existence, its exact path and the fact that you considered it worth hiding, in a document you have invited the entire internet to read. Every reconnaissance tool ever written starts there for exactly this reason. If a path must not be reached, put it behind authentication or move it off the public host; if it merely must not be crawled, disallow a broad parent prefix rather than naming the sensitive leaf. And treat the file as code: keep it in version control, review changes like any other deployment, and check it after every platform migration, because a stray Disallow: / shipped to production is the fastest way to remove an entire site from search and one of the slowest to recover from.

The lines you can put in a robots.txt, and the two that people write anyway
Line	What it does	Status	The trap
User-agent: *	Opens a group and names the crawlers it addresses; * addresses any crawler with no group of its own	Honoured	A crawler obeys exactly one group — the most specific one that names it — and ignores the * group entirely once it has its own
Disallow: /path	Asks the crawlers in this group not to fetch any URL whose path starts with that prefix	Honoured	It blocks fetching, not indexing — a blocked URL linked from elsewhere can still be listed in results, with no snippet
Allow: /path/file	Carves an exception out of a broader Disallow in the same group	Honoured	It only wins if its path is longer than the Disallow it fights; an equally long pair goes to the Allow, a shorter one loses
Sitemap: https://…/sitemap.xml	Points crawlers at a sitemap; may appear anywhere in the file and belongs to no group	Honoured	The value must be a full absolute URL including the scheme; a relative path is discarded silently
Crawl-delay: 10	Asks a crawler to wait between requests; a vendor extension, never part of the protocol	Ignored by Google	Some crawlers respect it and some do not, so it cannot be relied on for load control; rate-limit at the server instead
Noindex: /path	Nothing. It looks like it should remove a page from the index and it does not	Unsupported	Use a meta robots noindex tag in the page head, or an X-Robots-Tag response header, and leave the URL crawlable so the rule can be read

Frequently asked questions

Can I use robots.txt to keep a private page out of Google?: No, on both counts. Google says outright that robots.txt is not a mechanism for keeping a web page out of Google, and a disallowed URL linked from any other site can still be listed using nothing but the link and its anchor text. Nor is the page private in any real sense: robots.txt is only a request, obeyed by search engines and ignored by everything with an interest in ignoring it, and the file itself broadcasts the path. For search visibility use a noindex, delivered as a meta robots tag or an X-Robots-Tag header. For actual privacy use authentication. The two are separate problems and robots.txt solves neither.
Do subdomains and http versus https share one robots.txt?: No. The rules apply only to the exact host, protocol and port that served the file, so https://example.com, http://example.com, https://www.example.com and https://api.example.com are four separate scopes needing four separate files — even if they resolve to the same server and the same document root. This is the second most common misconfiguration after putting the file in a subdirectory, where it is never read at all. A practical consequence: if you redirect http to https, the crawler follows the redirect to fetch the file, so the https copy governs both in effect; if you do not redirect, the http origin has no rules and is crawled freely.
I disallowed a page but it is still in the results. What now?: That is the expected behaviour, and the fix is to reverse the order of operations. Remove the Disallow so the page becomes crawlable again, add a noindex — a meta robots tag or an X-Robots-Tag header — and wait for the page to be recrawled and dropped. The reason the original approach failed is circular: while the URL is disallowed the crawler never fetches it, so it never sees any noindex you put on the page, and the listing built from inbound links persists indefinitely. Once the page has fallen out of the index you may reinstate the Disallow if you want to save crawl budget, though at that point it usually buys you very little.

Articles you may find interesting

All guides →

GuideHTTP Status Codes Explained: The Ones That Actually Get Confused301 against 308, 302 against 307, 401 against 403, 404 against 410 — plus what Retry-After on a 429 or a 503 actually promises. The pairs where picking the wrong code changes behaviour, not just wording.How-toHow to Write a Cron Expression: Five Fields and the OR Rule Nobody MentionsMinute, hour, day of month, month, day of week. The traps are that a step is a stride through a range and not an interval, and that the two day fields are combined with OR — so 0 0 1 * 1 fires on the 1st and on every Monday.ExplainerHow Unix File Permissions Work: Reading 755 Without GuessingRead is 4, write is 2, execute is 1, and each of the three digits describes a different party. The part most explanations get wrong is what the execute bit does on a directory — it grants traversal, not the right to run anything.GuideWhat Makes a URL Slug Good: Stability, Readability and the Conflict Between ThemA slug has two jobs that pull against each other: it is a permanent identifier and it is a piece of readable text. Length, hyphens, stop words, dates, non-ASCII characters and the identifier-plus-slug pattern that gets both properties — with the real numbers from a site that localises 1,736 tool slugs into six languages.GuideTitle Tags, Meta Descriptions, and What Search Engines Do With ThemWhat Google's own documentation says about rewriting title links and about the meta description, rather than what SEO folklore says. Then the measurable part: titles are truncated by pixel width, so two titles of exactly sixty characters can render 204.55 pixels apart and only one of them survives.ExplainerKeyword Density Is a Dead Metric, and What Replaced ItDensity counted occurrences because retrieval once counted occurrences. TF-IDF, then BM25 with its saturation curve, then embeddings replaced it. Here is the same 800-word page scored three ways, and why the three disagree.

Sources

Spotted a mistake in this article?