Keyword Density Is a Dead Metric, and What Replaced It

What density was actually a proxy for

Keyword density is one division: the number of times a term appears, over the number of words on the page. Sixteen appearances of a word in an 800-word article is 2.00%. That is the whole metric, and it was never invented by search engines — it was invented by the people trying to reverse-engineer them, as a guess at what the engine was counting.

The guess was reasonable, because for a while it was right. Early retrieval treated a document as a bag of words: order discarded, structure discarded, the document reduced to a vector of term counts. Score a document against a query by adding up how many query terms it contained and how often, and repetition is directly rewarded. Density is that model's rule of thumb, normalised for length so that a long page and a short page can be compared. It was a proxy for one specific thing — term frequency in a bag-of-words scoring function — and it stopped being useful the moment the scoring function changed.

Two things killed it. The first was that the proxy became a target, and a metric that is also a target stops measuring: pages padded with repetitions were, by density, excellent, and by any human standard unreadable. The second was slower and more interesting — the scoring functions stopped counting that way, first through inverse document frequency, then through saturation, then by abandoning term counting altogether. The rest of this article is those three steps, run on one concrete page.

TF-IDF: rarity is worth more than repetition

The first correction came from a 1972 observation of Karen Spärck Jones: the specificity of a term is inversely related to how many documents contain it. A term appearing in almost every document tells you almost nothing about which document to return. That becomes inverse document frequency, idf = ln(N ÷ df), with N the number of documents in the corpus and df the number containing the term. Multiply it by term frequency and you have TF-IDF, which is still the mental model most working editors actually need.

Put numbers on it. Take an 800-word page about espresso machines. The word machine appears 24 times — a 3.00% density, the highest on the page. The word espresso appears 16 times, a 2.00% density. The word descaling appears 4 times, a 0.50% density. Now assume a corpus of 10,000,000 documents in which machine appears in 2,000,000, espresso in 200,000 and descaling in 50,000. The idf values are ln(5) = 1.6094, ln(50) = 3.9120 and ln(200) = 5.2983. With raw term frequency, TF-IDF gives machine 24 × 1.6094 = 38.63, espresso 16 × 3.9120 = 62.59 and descaling 4 × 5.2983 = 21.19.

The ordering has already flipped once. Density says machine is the strongest signal on the page. TF-IDF says espresso is, by a margin of 62 to 39, even though it appears eight fewer times. And if you use the log-damped variant of term frequency that classical TF-IDF usually applies, 1 + ln(f), the scores become 6.72 for machine, 14.76 for espresso and 12.64 for descaling — a term at a 0.50% density now nearly outscores a term at 2.00%, and comfortably outscores the one at 3.00%. Same page, same words, three orderings, no repetition added.

BM25 and the saturation curve, which is the interesting part

BM25 came out of the Okapi work at City University London and the TREC evaluations of the 1990s, and it is still the default ranking function in Lucene, Elasticsearch and most of the search infrastructure running today. Its term-frequency component is where the real idea lives: instead of using f directly, it uses f × (k1 + 1) ÷ (f + k1 × (1 − b + b × dl ÷ avgdl)), where dl is the document length, avgdl the average length in the corpus, and k1 and b are tuning constants conventionally set to 1.2 and 0.75. Read it as a fraction and the behaviour falls out: as f grows, numerator and denominator both grow, and the ratio converges on k1 + 1.

On a page of exactly average length the length term collapses to 1 and the weight is simply f × 2.2 ÷ (f + 1.2). Run it: one occurrence scores 1.0000, two score 1.3750, three score 1.5714, five score 1.7742, ten score 1.9643, twenty score 2.0755, fifty score 2.1484. The ceiling is 2.2 and nothing reaches it. As a fraction of that ceiling, a single occurrence already collects 45.5% of everything the term can ever be worth; two collect 62.5%, five collect 80.6%, ten collect 89.3%.

The marginal view is the one that should end the argument. Going from one occurrence to two adds 0.3750. Going from nine to ten adds 0.0231. The tenth occurrence is worth about one sixteenth of the second, and the fiftieth is worth 0.0010 — 365 times less than the second. This is what saturation means, and it is a deliberate design choice, not an accident: Robertson and Zaragoza describe the shape as the point of the function — the first occurrence establishes that the document is about the term at all, and further occurrences can only confirm what is already established. Writing a word a tenth time to rank for it is optimising a quantity the retrieval function has already stopped listening to.

The b parameter handles length, and it does so far more gently than density does. Take those same 16 occurrences of espresso and put them in a 1,600-word page instead of an 800-word one, in a corpus whose average document is 800 words. Density halves, from 2.00% to 1.00%. The BM25 weight falls from 2.0465 to 1.9448 — a drop of 5.0%. To get back to the 800-word score at double the length you would need 28 occurrences, not 32. Density says you were penalised by half; BM25 says you were penalised by a twentieth. They are not measuring the same thing and never were.

Then embeddings stopped counting terms at all

TF-IDF and BM25 are both lexical: they need the query term to be present in the document, in some form, for the document to score at all. Embedding-based retrieval breaks that assumption entirely. A model maps the query and the document into the same high-dimensional vector space, and relevance becomes the geometric closeness of two vectors — usually cosine similarity — rather than any overlap of strings. Occurrence counts play no role in the calculation. There is no f in the formula.

The practical consequence is the one that should reframe how you write. A page about descaling a coffee machine can be retrieved for a query about limescale in a kettle, or about white vinegar in an espresso maker, without containing any of those words, because the vectors land close together. Conversely, a page that repeats an exact phrase forty times but says nothing else about the subject sits in a region of the space that no genuine question points at. Modern systems generally run both families and fuse the results, so lexical matching has not disappeared — but the lexical half is BM25, which saturates, and the semantic half does not count at all.

What density is still worth: exactly one look

Here is the honest defence of the metric. Density is a diagnostic, not a dial. Run it once on a finished draft and read it the way you read a spelling checker: not to hit a number, but to be surprised. A phrase sitting at 5% of an 800-word page is forty occurrences, and nobody writes forty deliberate repetitions of anything — so it is a template artefact, a boilerplate block counted into the body text, a product name repeated by a generator, or an editor who lost the thread. All four are real defects, and density finds them in one pass.

What it is not worth is a target, and this is where you should be suspicious of any tool or consultant that offers one. There is no published recommended percentage, from Google or from anyone with access to a ranking function, and the reason is structural rather than secretive: BM25 saturates, so there is no percentage that maximises it; embeddings ignore counts entirely, so there is no percentage they respond to; and the optimal number of occurrences of a term depends on corpus statistics you do not have. A tool that tells you to aim for a band is reporting a convention, not a finding. Google's own spam policies address the same behaviour from the other end — keyword stuffing, defined as filling a page with repeated words to manipulate rankings, is a violation, and it is described in terms of the reading experience, not a threshold.

So the replacement for a density target is not a different number. It is a different question: does this page cover the distinct sub-topics that a person asking this question would need answered, in language they would recognise. That is what a term-weighted model rewards without anybody optimising for it, because a page that genuinely covers a subject naturally contains its rare and specific vocabulary — the descaling, the portafilter, the grind size — and rare specific vocabulary is precisely what idf is built to reward. Word frequency counts and readability scores are more useful companions to a draft than a density percentage, because both describe the text rather than pretending to describe the engine.

Frequently asked questions

What is the ideal keyword density?

There isn't one, and the absence is mathematical rather than secretive. BM25 saturates: with k1 = 1.2, one occurrence collects 45.5% of everything the term can be worth and ten collect 89.3%, converging on a ceiling nothing reaches — so there is no count that maximises the score. Embedding retrieval ignores counts entirely. And the number of occurrences that would help most depends on corpus statistics you cannot see. Any tool quoting a band is quoting a convention that predates every retrieval model in use.

If density is dead, why does your tool still calculate it?

Because a dead target can still be a live diagnostic. Nobody should write toward a density figure, but reading one on a finished draft catches accidents no other check catches: boilerplate counted into the body, a product name repeated by a generator, a template variable that expanded forty times. Those are real defects and density surfaces them in one pass. Use it the way you use a spell checker — to be told something you did not intend, not to hit a score.

What are k1 and b in BM25, and do I need to care?

k1 controls how fast term frequency saturates and sets the ceiling at k1 + 1; b controls how strongly document length is normalised, from 0 for no normalisation to 1 for full. The conventional defaults are 1.2 and 0.75, which is what Lucene and Elasticsearch ship. As a writer you do not tune them and cannot know what any given engine uses. What you should take from them is the shape they produce, not the values: repetition has steeply diminishing returns, and length is penalised gently rather than proportionally.

Does using my keyword in the title and headings still matter?

Yes, but not for the reason density people usually give. Field-weighted retrieval treats a match in a title differently from a match in body text, and BM25F — the fielded extension of BM25 — formalises exactly that. The reason is not that the term appears again; it is that where it appears carries information about what the document is about. The practical rule is unchanged from good editing: say what the page is about in the title, once, in the words a reader would use. That also happens to be what makes a title work as a link and as a snippet.

Can a page rank for a phrase it never contains?

Yes, and that is the sharpest illustration of why density has no target. Embedding retrieval maps query and document into the same vector space and compares them geometrically, with no requirement that any string match. A page about removing limescale from an espresso machine can be retrieved for a query about descaling a coffee maker with vinegar, because the vectors land close together. Purely lexical systems could never do this, and it is why writing for coverage of a subject now outperforms writing for occurrences of a string.

Is keyword stuffing still a penalty, or just useless?

Both, and the useless part comes first. BM25 saturation means the fortieth occurrence contributes essentially nothing over the tenth, so the practice buys no score even before any policy applies. On top of that, Google's spam policies list keyword stuffing — filling a page with repeated words to manipulate rankings — as a violation, described in terms of what it does to a reader rather than as a numeric threshold. So the honest summary is that stuffing is a way to make a page worse for humans in exchange for a scoring gain that the retrieval function stopped paying out decades ago.