Counting Words Is Ambiguous, and Every Tool Answers Differently

Four methods, one paragraph, four answers

Here is the paragraph we used, 188 characters long and typical of technical writing: "The state-of-the-art model doesn't stop at 1,234 tokens—it reads the whole page, including https://example.com/docs, and re-runs the check twice. That's 98.6% coverage, up from two-thirds." Nothing about it is contrived; every feature in it appears in ordinary product documentation.

Splitting on runs of whitespace gives 25 tokens. This is the simplest possible definition and the one most people would arrive at unaided: a word is what sits between two spaces. It treats state-of-the-art as one word, the whole URL as one word, and tokens—it as one word, because the em dash has no spaces around it.

The regular expression that matches word characters between word boundaries gives 38. It splits state-of-the-art into four, doesn't into two, 1,234 into two, 98.6 into two, and the URL into https, example, com and docs. Every one of those splits is what the pattern says it should do — a word character in that expression is a letter, a digit or an underscore, so a hyphen, an apostrophe and a comma are all boundaries. The result is a count half as large again as the whitespace one.

Unicode word segmentation, available in the browser and in Node as Intl.Segmenter with granularity "word", gives 33. It is the only one of the four with a specification behind it, and its choices are noticeably more human: it keeps doesn't and That's whole, keeps 1,234 whole, keeps example.com whole, and splits state-of-the-art into four because the standard treats the hyphen as a break. A fourth rule, written to keep letters and digits together across internal hyphens, apostrophes, full stops and commas, gives 28 — the closest of the machine rules to what a person would say if you asked them to count out loud.

The honest correction: on plain prose they all agree

It would be tidier to claim that word counts disagree everywhere, and it is not true. We counted a 160-character paragraph of ordinary narrative prose — no compounds, no figures, no addresses — and all four methods returned exactly 29. Then we counted 49,616 characters of this site's own English article text and got 8,622 from whitespace splitting, 8,537 from the hyphen-aware rule, 8,729 from segmentation and 8,917 from the regular expression. Lowest to highest is a spread of 4.5%.

So the real statement is narrower and more useful than "word counts are unreliable". They are reliable on prose and unreliable on anything that mixes prose with machine-readable material. Our technical paragraph spread from 25 to 38, a 52% gap, on 188 characters. A contract clause containing a figure, a compound adjective, a URL and a time of day spread from 24 to 33, a 38% gap. The predictor is not the length of the text; it is the density of things that are not plain words.

Where exactly the methods part company

Hyphenated compounds are the largest single source of disagreement. state-of-the-art counts as one under whitespace splitting and under the hyphen-aware rule, and as four under both the regular expression and Unicode segmentation. Any text with several such compounds — a specification, a product page, a legal document — will show a visible gap between tools for that reason alone.

Contractions split the field differently. doesn't is one word for whitespace splitting, one for segmentation and one for the hyphen-aware rule, but two for the regular expression, which treats the apostrophe as a boundary and leaves a stray t. The curly apostrophe behaves the same way, so switching a document to typographic quotes does not fix it. The possessive plural is quieter: the students' work counts as three under every method, because the apostrophe is at the end of a token rather than inside it.

Numbers and addresses cause the rest. 1,234 and 98.6 each count as one for three methods and two for the regular expression, which cannot tell a thousands separator or a decimal point from punctuation. An email address counts as one, two or three depending on the method; a URL counts as one, three or four. And an acronym written with full stops — U.S.A. — is one word for whitespace splitting, segmentation and the hyphen-aware rule but three for the regular expression. Punctuation joining two words with no space around it splits everything except whitespace splitting: tokens—it, 2024–2026, and/or and wait…what all count as two for three methods and one for the fourth.

Chinese and Japanese, where whitespace splitting returns 1

Take a sixteen-character Chinese sentence with no spaces in it, as Chinese sentences generally have none. Splitting on whitespace returns 1 — the whole sentence is one token. The word-character regular expression returns 0, because the word-character class in a pattern without the Unicode flag covers only ASCII letters, digits and the underscore, and a Chinese character is none of those. The hyphen-aware Unicode rule returns 2, because the sentence has one internal comma and the rule sees two runs of letters.

Unicode word segmentation returns 8, and the eight tokens it finds are the words a reader would identify. The equivalent Japanese sentence of twenty-one characters segments into eleven. This is dictionary work rather than pattern matching, and it is the reason segmentation is worth reaching for even in a Latin-script product: it is the only method in the list that does not silently return a wrong answer on text it was not designed for. It also does not require you to know the language in advance — segmenting the Chinese sentence with the English locale gave the same eight tokens.

Why a publisher's count and a word processor's count differ

A word processor is counting what its author decided a word is, which in practice is close to whitespace splitting with a few adjustments — it is optimised for a person writing prose and it wants the number in the status bar to feel right. A content management system is often counting with a regular expression, because that is what was available in the language it was written in. A publisher who pays by the word may be counting neither, and instead dividing the character count by a fixed figure, because that removes the argument about compounds entirely.

None of these is cheating. They are three answers to a question that has no canonical form, and the only thing that goes wrong is comparing them. If you are told a piece is 1,500 words, that number is meaningless until you know which count produced it. On our test paragraph, a thousand words means 7,520 characters of text if the count is whitespace-based and 4,947 characters if it is regex-based — the same nominal quantity delivering half as much again in one reading as in the other.

Character counts do not escape the problem, they relocate it. A character is as contested a unit as a word — code units, code points and grapheme clusters give three different numbers for the same string, and any text containing an emoji or a combining accent will produce all three. We took that apart in a separate piece on character limits; the short version is that both counts are definitions, and a limit stated in either is incomplete until it names the definition.

What to do when the limit is contractual

Name the counter, not just the number. "Not more than 2,000 words" is not a specification; "not more than 2,000 words as counted by X" is. Agreeing on a tool is enough — both sides paste the text into the same counter and get the same answer, whatever definition it implements. What must not happen is each side using its own and discovering the gap after delivery.

If you cannot name a tool, name a rule in prose: whether hyphenated compounds count as one or as several, whether a URL counts as one, whether figures count at all. Those three sentences remove most of the disagreement, because those three categories account for nearly all of the 52% spread we measured. And if you want a unit that is genuinely hard to argue about, use characters excluding spaces and state which character you mean — it is a worse proxy for effort and a much better one for length.

Frequently asked questions

Which word count is the correct one?

There is no correct one, only a fitting one. If the count exists so a reader knows how long a piece feels, whitespace splitting or Unicode segmentation both work and segmentation handles more languages. If it exists so a machine can index or truncate, segmentation is the only one with a published specification behind it. If it exists so money changes hands, the correct count is whichever one both parties named in advance.

Why does my word count jump when I paste from a website?

Usually because the pasted text brought URLs, figures and hyphenated terms with it, and those are the three categories on which the counting methods disagree most. Our technical test paragraph varies by 52% between the lowest and highest method while an ordinary narrative paragraph gives the identical number from all four. A second cause is invisible characters — no-break spaces from a formatted number, or a zero-width joiner from an emoji — which can merge or split tokens without showing anything on screen.

Does the word-character regex work for accented languages?

Not without help. In its plain form the word-character class covers ASCII letters, digits and the underscore, which means an accented letter is a boundary rather than part of a word. A French or Portuguese text will therefore be over-counted, with each accented word split at the accent. The fix is to write the pattern against Unicode letter and number properties instead, which is what the hyphen-aware rule in this article does, or to use segmentation and skip the question.

Is Intl.Segmenter available everywhere?

It is part of the ECMAScript internationalisation standard and is present in current browsers and in Node, which is where we ran every number in this article. The practical caveat is not availability but data: segmentation quality depends on the locale data the runtime shipped with, so two environments can disagree slightly on the same text. For counting Latin-script prose the difference is negligible; for Japanese, where segmentation is a genuine dictionary problem, it is not.

How should a word counter treat emoji?

Consistently, and usually by not counting them as words. In our test, "Great work 👍 thanks" gives four from whitespace splitting, which counts the emoji as a word, and three from the other methods, which do not. Neither is wrong, but a counter that reports four should say so, because a writer working to a word limit will otherwise budget for a word that carries no text. Whichever choice you make, apply it to the whole document rather than to individual emoji.

Can I convert between word counts with a fixed ratio?

Only very roughly, and only for prose. Over 50,000 characters of ordinary article text the four methods stayed within 4.5% of each other, so a single ratio is a workable approximation there. On a technical paragraph the same four spread by 52%, which no ratio survives. If you must estimate, estimate from characters: our English article prose runs 5.75 characters per whitespace-word, and that figure is far more stable across text types than the ratio between two word-counting rules.