Word Frequency and Zipf's Law: We Counted Six Books in Six Languages and Fitted the Slope

The claim, and how we tested it

The claim is easy to state: sort every distinct word of a long text by how often it occurs, and the word in position n will appear roughly 1/n as often as the word in position 1. The second commonest word appears about half as often as the first, the tenth about a tenth as often, the hundredth about a hundredth. It is usually asserted; we counted instead. Six novels came from Project Gutenberg, one per language of this site, with the licence header and footer stripped. Every text went through the same tokeniser — lowercase, then match runs of Unicode letters and marks plus the apostrophe — so no language got a friendlier treatment than any other.

Tokenising is already a decision, not a neutral act. Splitting on Unicode letters merges "Don't" and "dont" only if you keep the apostrophe out, treats "co-operate" as two words, and counts inflected forms separately, so German and Italian — languages that spread their grammar across many word forms — end up with more distinct words for the same amount of text. Buddenbrooks has 25,071 distinct words in 230,086 running words; Quincas Borba has 10,329 in 78,361. Those are properties of our tokeniser as much as of the languages, and the same caveat applies to any word-frequency tool, including ours. What matters here is that the choice was held constant across all six.

Rank × frequency stays almost constant

If frequency really goes as 1/rank, then rank multiplied by frequency should be the same number everywhere. In Moby-Dick it very nearly is. Rank 20 is "at" with 1,320 occurrences: 20 × 1,320 = 26,400. Rank 50 is "what" with 565: 28,250. Rank 100 is "white" with 281: 28,100. Rank 500 is "somehow" with 44: 22,000. Rank 1,000 is "gunwale" with 21: 21,000. Rank 5,000 is "sporting" with 4: 20,000. Across ranks that differ by a factor of 250, the product varies by about a third — that is what "roughly constant" looks like in real data, and it is a strong result for a one-parameter claim about a novel nobody wrote to be countable.

The tidier way to see it is on log-log axes. Taking base-10 logarithms of both rank and frequency turns f = C / rank^s into a straight line with slope -s, and a least-squares fit recovers s. Over ranks 10 to 1,000 in Moby-Dick the fit gives a slope of -1.083 and an R² of 0.998 across 991 points — a very good straight line. The other five books give -1.068, -1.054, -1.021, -1.079 and -1.031, all with R² between 0.996 and 0.999. Zipf's law in its bare form predicts exactly -1; every one of our six texts sits a little steeper, between 2% and 8% below it. That systematic excess is well documented and is one reason the generalised form writes the exponent as a free parameter instead of fixing it at one.

The same shape in six languages, and across two centuries

The six books span 1605 to 1901 and six languages with very different grammars, and their slopes fall in a band 0.06 wide. That is the interesting part: the shape does not come from English, or from novels, or from the nineteenth century. To check that it survives into modern non-fiction we ran the same count on this site's own English articles — 533 of them, 513,756 running words written in 2026 about tax rates, unit conversions and file formats. The slope over ranks 10 to 1,000 is -0.915 with R² 0.997. Flatter than the novels, but the same straight line on the same axes, 175 years and one genre later.

The top-ten lists differ in the obvious way and agree in the deep way. English gives the, of, and, a, to; French gives de, la, et, il, le; Spanish gives que, de, y, la, a; Portuguese a, que, o, de, e; German und, die, der, er, in; Italian e, che, di, a, il. Not a noun among them. Every one of these is an article, a preposition, a conjunction or a pronoun — grammatical glue rather than subject matter. That is the observation the rest of this article rests on.

Where the law fails: both tails, and any short text

Fit only the top ten words of Moby-Dick and the slope comes out -0.793 instead of -1.083; the head of the distribution is flatter than the law predicts. It is flatter still in the other five: -0.541 in French, -0.693 in Spanish, -0.595 in Portuguese, -0.545 in German, -0.485 in Italian. The single most frequent word is never as dominant as a strict 1/n rule would demand, which is precisely why the Zipf–Mandelbrot form adds a constant to the rank before raising it to a power. At the other end the fit degrades differently: over ranks 1,000 to the end, R² drops from 0.998 to 0.962 in English and to 0.895 in Portuguese, because the tail is a staircase of words that occur four, three, two and finally one time, and a line through a staircase is a poor line.

The more useful failure is length. Take the first 200 words of Moby-Dick and count them: 100 distinct words, 86 of which occur exactly once, and the most frequent token is "chapter" at 56 occurrences because the passage is mostly a table of contents. Fitting that gives a slope of -0.614 with R² 0.685 — no law visible at all. At 1,000 words the fit is -0.751 with R² 0.920; at 5,000 words -0.897 and 0.969; at 20,000 words -0.991 and 0.991. The distribution only becomes recognisable somewhere in the low tens of thousands of words. A blog post, a product description or an email is far below that. Running a word-frequency count on 300 words tells you what that text happens to repeat; it does not tell you anything about the language.

Why stop words exist, and why compression works

If a hundred words are half your text, then a hundred entries in a list will remove half the tokens from an index. That is all a stop-word list ever was: an economy measure that Zipf's law makes enormous. In Moby-Dick the ten commonest words are 23.6% of the running text and the hundred commonest are 50.6%; 96 distinct words cover half the book. The other languages agree on the shape and differ on the count: 90 word forms for half of the French novel, 59 for the Spanish, 97 for the Portuguese, 132 for the German, 141 for the Italian. Note that this is exactly why the stop-word approach is now out of favour in search — those words carry a little information, and "to be or not to be" is entirely made of them.

The compression connection is measurable. Moby-Dick uses 17,422 distinct words; if every word were equally likely, naming one would cost log2(17,422) = 14.09 bits. The actual unigram distribution has an entropy of 10.00 bits per word, 29.1% less, and that saving is the skew — the fact that "the" turns up every fifteenth word while "gunwale" turns up twenty-one times in a novel. General-purpose compressors exploit the same skew at the byte level: gzip -9 takes the 1,206 KiB book down to 486 KiB. Shuffling all 216,603 words into random order and compressing again gives 459 KiB out of 1,152 KiB, essentially the same ratio, which tells you how much of gzip's win here comes from the frequency distribution rather than from word order.

Raw frequency is a bad relevance signal — a worked example

Split Moby-Dick into its chapters, treat each as a document, and ask the frequency counter what one chapter is about. We used the 134 chapters long enough to count and picked the one titled "Queen Mab", 863 words long. Ranked by raw frequency its top terms are a (39), i (36), the (26), he (22), of (20) — a perfect description of English and a useless description of the chapter. Weight the same counts by inverse document frequency and the top terms become kicked (8), kick (7), wise (9), pyramid (5), says (10), flask (10), which is a fair summary of a chapter in which Stubb has been kicked and asks Flask about it.

The arithmetic behind the switch is one logarithm. "The" appears in 134 of the 134 chapters, so its inverse document frequency is ln(134/134) = 0.000 and its weight collapses to zero no matter how often it occurs. "Cetology" appears in 3 chapters, so its idf is ln(134/3) = 3.799 and three occurrences beat twenty-six of "the". This is the same reasoning behind tf-idf and behind BM25, the ranking function most search engines start from, and it is the reason the keyword-density number that marketing tools report has no counterpart inside a modern retrieval system. Density measures the wrong axis; what matters is how unusual a term is in the collection, not how often it repeats in the page.

What a word-frequency count is actually good for

Once you stop reading the top of the list, the count becomes genuinely useful. Compare a document against a reference frequency list and the words that are unusually common in it are its subject — that is the tf-idf calculation above with a corpus you already have. Look for repetition you did not intend: an author's tic, a term repeated eleven times in a 400-word product page, a name spelled two ways. Count the hapax legomena, the words occurring exactly once, which are 44.2% of the distinct words in Moby-Dick and 55.9% in Quincas Borba; a sudden drop in that share across drafts is a decent proxy for vocabulary flattening. And check coverage: if 100 words carry half your text, the other half is where the meaning is.

One habit is worth keeping from all of this: report the denominator. "Appears 12 times" means nothing without "in 400 words", and 12 in 400 is 3%, which for a content word is very high and for "the" is very low. Our word-frequency counter shows the count, the share and the rank together for that reason, and the readability tool next door works from the same token stream.

Frequently asked questions

How long does a text have to be before Zipf's law shows up?

Tens of thousands of words. Truncating Moby-Dick and refitting gives R² 0.685 at 200 words, 0.920 at 1,000, 0.969 at 5,000 and 0.991 at 20,000, with the slope climbing from -0.614 towards -1.0 as the text grows. Below a few thousand words most distinct words occur exactly once — 86 of the 100 distinct words in a 200-word sample — and a rank list made mostly of ties has no slope to measure.

Should I remove stop words before counting?

It depends on the question. To find out what a document is about, yes — or better, weight by inverse document frequency, which removes them automatically: "the" appears in all 134 Moby-Dick chapters, so its idf is exactly zero. To check whether a text is unusual, no: an abnormally low rate of function words is itself a signal, and stop-word lists silently delete real content, since "to be or not to be" is made entirely of stop words.

Why is the fitted slope steeper than -1?

Because the bare 1/n law is an idealisation. Every one of our six novels fits between -1.021 and -1.083 over ranks 10 to 1,000, so the excess is systematic rather than noise, and it is why the generalised Zipf–Mandelbrot form treats the exponent as a free parameter and adds a constant to the rank to bend the flat head into place. Which range you fit also changes the answer: fitting the full rank list instead of ranks 10 to 1,000 moves the English slope from -1.083 to -1.154, because the noisy tail is the majority of the points.

Does the tokeniser change the result?

It changes the counts, not the shape. Whether you keep the apostrophe, split hyphenated words, or lemmatise inflected forms moves the number of distinct words a lot — 25,071 for the German novel against 10,329 for the Portuguese one, on texts of very different lengths — while the log-log slope stays in the same narrow band. Compare frequency lists only when they came from the same tokeniser, and say which one you used.

Is keyword density a useful number?

Only as a self-check on repetition. Density is a within-document number and relevance is a between-document one: in our chapter test, weighting by inverse document frequency turned a top-five of a, i, the, he, of into kicked, kick, wise, pyramid, flask. A word that appears in every document in a collection has an idf of exactly zero however dense it is on your page. Use the count to catch accidental repetition; do not use it as a target.