Cleaning Messy Text: The Order of Operations That Actually Matters

Cleanup steps do not commute

A text cleaner looks like a menu: remove extra spaces, remove line breaks, remove duplicate lines, strip HTML tags. Each item sounds self-contained, so people tick them in whatever order the interface happens to list them. They are not self-contained. Every step rewrites the input that the next step sees, and several pairs produce different output depending on which one runs first. The failures are quiet: you still get text, it still looks plausible, and the part you lost is the part you were not looking at.

Three pairs cause almost all the real damage: tags against entities, whitespace against line breaks, and deduplication against trimming. Each one is demonstrated below on an actual string, run rather than reasoned about. The nine ordered steps are simply what falls out of those three constraints once you respect all of them at the same time.

Tags before entities, and decode exactly once

Take one HTML fragment: <p>Terms &amp;amp; conditions: write &lt;b&gt; to show a bold tag.</p>. A browser renders it as: Terms &amp; conditions: write <b> to show a bold tag. The escaped sequences are text — the author wanted a literal ampersand and a literal, visible bold tag.

Strip the tags first and decode once, and you get exactly that browser output. Decode first and then strip, and the &lt;b&gt; has already become a real <b> element by the time the stripper runs, so the stripper deletes it: the result is "write to show a bold tag", with a double space where the subject of the sentence used to be. Strip first but decode twice and you get the opposite failure: "write <b> to show a bold tag" now contains a live tag that nothing escaped, which is how a plain-text extract turns back into markup the next time somebody pastes it into a page.

The rule behind all three outcomes is short: an entity is escaped text, and decoding promotes text to markup. Anything that treats markup specially must therefore run before the promotion. Browsers avoid the whole question by tokenising once, in a single pass, which is why a real parser never has to decide this. A regex pipeline does, and the decision is: strip, then decode, exactly one decode.

Whitespace after line breaks, never across them

Take a document with three paragraphs, some of them wrapped across several lines, with a stray blank line or three between them. Run /\s+/ replaced by a single space over it first, because "remove extra spaces" was the first checkbox, and you get one 80-character run with zero paragraph boundaries left. Nothing downstream can restore them: the newlines that carried the structure were whitespace, and you asked for whitespace to be collapsed.

Run the same document through the correct order — normalise line endings, trim each line, join the wrapped lines, collapse only horizontal runs with /[^\S\n]+/, then reduce three or more newlines to one blank line — and it comes out at 58 characters across two paragraphs, structure intact. The difference is not that one regex is better than the other. It is that /\s+/ includes \n and \r, and a class like [^\S\n] deliberately does not.

There is a second, subtler ordering inside this one. Take "Paragraph one text.", a line containing a single space, and "Paragraph two text.". Split it on /\n\n/ and you find one paragraph, not two, because the separator line is not empty — it holds a space. Trim each line first and the same split finds two. That is why trimming lines comes before any paragraph decision, while collapsing runs comes after it: the two whitespace operations sit on opposite sides of the line-break step.

Deduplicate last, because equality is a downstream property

Line deduplication compares whole strings. Give it the five lines "alpha " (with a trailing space), "beta", "alpha", "beta" followed by a tab, and "gamma", and it removes nothing: five lines in, five lines out, because "alpha " and "alpha" are simply different strings and so are "beta" and "beta\t". Trim the trailing whitespace first and the same deduplication takes the five down to three. The duplicates were always there; they were invisible in exactly the way a trailing space is invisible.

The same thing happens on the worked sample at the end of this article. Deduplication placed second in the pipeline, right after the tags are stripped, removes zero lines. The identical deduplication placed eighth removes one, because by then the zero-width space has gone and the double space has been collapsed, so the two lines that were always the same sentence have finally become the same string. Deduplication does not find duplicates; normalisation creates them, and deduplication then collects them.

One decision stays yours: case. Deduplication compares exactly, so "Total" and "total" are two lines. Case folding is a separate, lossy choice that belongs in its own step and should be visible in the interface rather than baked into the deduplicator — the same argument that applies to naming conventions, where the transformation is only safe if you know which convention you started from.

The characters that survive every naive cleanup

Three code points cause most of the residue. U+00A0, the non-breaking space, arrives from word processors, from web pages and from French and Spanish typography, where it sits before a colon or after an opening quotation mark. U+200B, the zero-width space, arrives from CMS editors and from copy-pasted rich text as an invisible line-break hint. U+FEFF, the zero-width no-break space, is what a UTF-8 byte-order mark decodes to; it turns up at the very start of files written by spreadsheet exports and by Windows tools, and sometimes in the middle after a naive concatenation.

What matters for a cleanup pipeline is which of them your tools can see, and the answer is not intuitive. Run in Node 22: /\s/ matches U+00A0 and U+FEFF but not U+200B. The Unicode property escape /\p{White_Space}/u matches U+00A0 but not U+FEFF. They disagree, in both directions — U+0085, the NEL control, is matched by the Unicode property and not by /\s/. The reason is that the ECMAScript grammar defines its own WhiteSpace production, which adds the byte-order mark for historical reasons, while the Unicode Character Database assigns White_Space on its own criteria and does not give it to U+FEFF.

Normalisation is not a substitute. Applying NFKC maps U+00A0 to a plain U+0020, which is genuinely useful, but it leaves U+200B exactly where it was: the string is still one character long afterwards. So the pipeline needs an explicit deletion step for the format characters — U+200B, U+200C, U+200D, U+FEFF, U+00AD — and a separate mapping step for the exotic spaces. Neither can be delegated to a whitespace regex, because to a whitespace regex half of them are not whitespace.

One messy sample, nine steps, before and after

The sample is 126 characters and contains, on purpose, every problem discussed above: a leading pair of spaces, CRLF line endings, an h2 and three p elements, a double-encoded ampersand, a zero-width space glued to the end of a sentence, three consecutive newlines, a sentence repeated verbatim, a doubled internal space, a paragraph wrapped across two lines with the continuation indented, and a byte-order mark plus two spaces at the very end.

Run through the nine steps in order it becomes 58 characters: a heading line reading Q3 &amp; Q4 report, a line reading Revenue rose 12%., a blank line, then Costs fell and slightly. as two lines of one paragraph. The intermediate lengths are 122 after the line endings are normalised, 92 after the tags go, 88 after the single entity decode, 86 once the zero-width space and the byte-order mark are deleted, 80 after each line is trimmed, 77 after the horizontal collapse, 76 after the triple newline is reduced, and 58 after deduplication.

Two of those numbers are the argument of the whole article. The drop from 76 to 58 is the duplicate line, and it only exists because steps 4 and 8 ran first; move deduplication to position two and that drop is zero. The 86 to 80 step is the per-line trim, and it is what later lets the paragraph split see a genuinely empty separator line. Everything else is bookkeeping.

Before you ship the cleaned text

Keep the original. Every step in this pipeline is lossy by design, and none of them is reversible: you cannot recover which spaces were non-breaking, which line break was a wrap and which was a paragraph, or which of two identical lines was the one you meant to keep. Cleaned text is a derived artefact, and derived artefacts should never be the only copy.

Then run the pipeline twice on its own output. A correctly ordered cleanup is idempotent: the second pass must change nothing. If it does, you have a step that is not stable under repetition, and in practice it is almost always the entity decode — the one operation in the list that can create new work for itself. An idempotence check costs one line of code and catches the class of bug that only appears when a document goes through the tool a second time, six months later, because somebody re-imported it.

Finally, count things only at the end. Any length, word count or readability figure taken mid-pipeline is measuring a string that no longer exists, and a length in particular depends on what you are willing to call a character — a question worth settling separately before you trust any number that a counter gives you.

Frequently asked questions

If I run the cleaner twice, does the order still matter?

Yes, and running it twice can make things worse rather than better. Repetition cannot recreate information that an earlier step destroyed — collapsed paragraph boundaries stay collapsed no matter how many passes you make. Meanwhile the entity decode is not idempotent: a second pass decodes &amp;amp; a second time, turning what was a deliberate literal ampersand in the text into a structural one. The right test is not to run it twice for a better result, but to run it twice to confirm the second pass changes nothing.

Why does .trim() remove the byte-order mark but not the zero-width space?

Because trim is defined against the ECMAScript WhiteSpace production, not against the Unicode White_Space property, and that production explicitly lists U+FEFF for historical reasons dating from when the byte-order mark was routinely found at the start of a stream. U+200B has never been on that list: Unicode classifies it as a format character in the Cf general category, on the grounds that it marks a line-break opportunity rather than inter-word space. So trim removes one and not the other, and neither /\s/ nor trim will ever help you with U+200B. Delete it explicitly.

Is there ever a good reason to use /\s+/ on a whole document?

Yes, in exactly one situation: when you have decided that the output is a single line and structure is irrelevant — a search key, a fingerprint used for comparison, a value going into a one-line CSV cell. There, flattening everything to single spaces is the point. Anywhere the output will be read by a person, /\s+/ is the wrong class, because it treats the newline that separates two paragraphs and the two spaces after a full stop as the same kind of thing. Use it deliberately for keys and never as a default for prose.

How do I even see that an invisible character is there?

Compare the length you expect with the length you get, then dump the code points. Two strings that render identically on screen can have different lengths — that is the whole tell. Once the length surprises you, list each character with its code point in hexadecimal and the offender is obvious: a U+00A0 sitting where you assumed U+0020, or a U+200B glued to the end of a sentence. Doing this once on a sample from each source you import from will tell you which producers in your pipeline emit which characters, and you can then delete exactly those.

Should deduplication preserve the original order of the lines?

Almost always yes, and it should keep the first occurrence rather than the last. Sorting to find duplicates is a habit inherited from command-line pipelines, and it silently reorders a document whose order carried meaning — a list of steps, a changelog, a transcript. Keeping the first occurrence also matches how people read: the earlier line is usually the one with the context around it. If a tool offers to sort while it deduplicates, treat that as two separate operations and only ask for the one you want.