Removing Accents Breaks Search — Until You Do It on Both Sides

Normalisation is a function, and it has to run on both sides

Here is the whole argument in one experiment. Take an index of six names — Café Amélie, Cafe Central, Zoë Müller, Zoe Miller, Łukasz Nowak, Ørsted Energi — and six queries: cafe, café, muller, müller, lukasz, orsted. With no normalisation anywhere, cafe finds one entry and café finds a different one, muller finds nothing and müller finds Zoë Müller. Six queries, two hits. That is the problem people set out to fix.

Now apply the strip to the query only, which is the change people make first because the query is the thing they control. The result is worse, not better. The query café now folds to cafe and matches Cafe Central, while Café Amélie — the entry it matched perfectly a moment ago — no longer matches at all, because the index still holds the accented form. The query müller loses its single hit for the same reason. Two hits become two hits, but they are not the same two, and one of the losses was an exact match.

Strip the index only and the mirror image happens: cafe now finds both cafés, but café finds nothing, because the accented query can no longer match the folded index. Strip both sides and the table finally behaves: all four spellings of the café query return both entries, and both spellings of Müller return the one entry. The rule this establishes is not "strip accents". It is: whatever function you apply, apply the same one, at index time and at query time, from the same code path. A normalisation applied on one side is not a weaker version of the correct thing — it is a different and generally worse matching rule.

NFC, NFD, and what the strip actually is

Unicode allows an accented letter to be written two ways. In composed form, NFC, the letter é is a single code point, U+00E9. In decomposed form, NFD, it is two: U+0065, the plain e, followed by U+0301, the combining acute accent. Both render identically on screen. Both are correct. They are not the same string: in JavaScript the composed é has length 1 and the decomposed é has length 2, and the strict equality of the two is false. A word like café taken from an NFC source and searched for in an NFD document simply does not appear, and the developer sees a search that fails on text they can plainly read on the page.

The strip exploits this. Normalise to NFD, which pulls every canonically decomposable letter apart into a base letter plus its marks, then delete everything in the Unicode general category M — the combining marks. é becomes U+0065 U+0301 becomes e. ñ becomes U+006E U+0303 becomes n. å becomes U+0061 U+030A becomes a. That is the entire mechanism, and it is why the operation is properly called folding rather than removing: nothing is being removed from the alphabet, a distinction is being discarded.

One practical consequence: before you fold anything, normalise everything to one form. If your index is NFC and your incoming text is NFD, folding both to the same base letters papers over the difference by accident — but any comparison you make before folding, or on a field you chose not to fold, will still be wrong. Normalise to NFC on ingest, as a storage rule, and treat folding as a separate index built on top of it.

The letters that carry no mark at all

The mechanism has an obvious failure mode: if a letter has no canonical decomposition, NFD leaves it alone and there is no combining mark to delete, so the fold does nothing. Run it and you can see exactly which letters those are. ß stays ß. ø stays ø. ł stays ł. œ stays œ. æ stays æ. ı stays ı. đ stays đ. In every one of these cases the diacritic is not a mark applied to a base letter; the stroke through the l, the slash through the o, the tie between the o and the e are part of the letterform, and Unicode encodes each as a character in its own right.

This is why the six-name experiment still returns nothing for lukasz and orsted even after both sides are folded. Łukasz Nowak folds to Łukasz Nowak, unchanged; Ørsted Energi folds to Ørsted Energi, unchanged. The two entries a user is least likely to type correctly are precisely the two the fold cannot help with. A word like Łódź is the clearest case: fold it and you get Łodz, because ó and ź decompose and ł does not, so the result is neither the original nor the plain-ASCII spelling anybody would search for.

The fix is a small explicit map applied before the NFD pass: ł to l, ø to oe or o, œ to oe, æ to ae, đ to d, ß to ss. Compatibility decomposition, NFKD, handles the ligatures œ and æ but not the stroked letters, so it is not a general answer either. There is no way around knowing which language you are indexing, which is exactly what the next three sections are about.

German: oe, not o — and ss, not s

German umlauts do decompose, so the naive fold produces something. It produces the wrong thing. Größe folds to Große — the umlaut goes, the sharp s stays, and the result is a real German word meaning something else. Müller folds to Muller, Öl folds to Ol. The convention German readers actually expect, codified in DIN 5007 variant 2 and used in telephone directories, is the two-letter expansion: ä to ae, ö to oe, ü to ue, ß to ss. Größe becomes Groesse, Müller becomes Mueller, Straße becomes Strasse.

The sharp s deserves its own paragraph because it behaves unlike anything else on this list. It has no canonical decomposition, so NFD leaves it; NFKC leaves it too, so Straße stays Straße. But its uppercase mapping in JavaScript is the two-character string SS, which means a naive uppercase-then-compare pipeline already folds it correctly while a naive accent strip does not. The capital form U+1E9E exists and lowercases back to ß. If your German index is built by uppercasing and your query is built by accent-stripping, ß will match in one direction and not the other — the one-sided problem again, wearing a different hat.

There is a way to get this right without writing a table at all. An Intl.Collator for German at base sensitivity already treats Grosse and Größe as equal — that is standard German collation, where ß has a secondary difference from ss. Ask for the phonebook variant, the locale de-u-co-phonebk, and Groesse and Größe compare equal too, because that collation is precisely the one that treats ö as oe. CLDR maintains those tables; you do not have to.

Scandinavian and Turkish: letters, not decorations

In Danish, Norwegian and Swedish, æ, ø and å are letters of the alphabet, and they come after z. You can watch a collator prove it: sort A, Aa, Å, Ø and Z with a Danish collator and you get A, Z, Ø, Å, Aa; sort the same five with an English collator and you get A, Å, Aa, Ø, Z. In Danish, å has not been quietly filed next to a — it has its own position at the end, and the digraph Aa sorts with it. Folding å to a therefore does not remove an accent, it merges two different letters, and folding ø to o merges two more.

Turkish has the sharpest case of all, and it is about casing rather than accents. Turkish distinguishes a dotless ı, U+0131, from a dotted i, and correspondingly a dotted capital İ, U+0130, from the plain I. The lowercase of I is ı and the lowercase of İ is i — but only in the Turkish locale. Run it in JavaScript and the trap is visible: the string İSTANBUL lowercased with the default rules becomes nine characters long, because İ maps to i followed by the combining dot above U+0307. Lowercased with toLocaleLowerCase("tr") it becomes eight characters, the plain istanbul a user would type. A search index built with the default lowercase will never match that user's query, and the two strings look identical on screen.

Both cases point the same way. The letters that break a naive fold are the letters that a language treats as full members of its alphabet, and the transformation that a language expects is a property of the language, not of the character. That is what a locale argument is for, and passing one costs nothing.

The words your own language will not let you fold

Folding is lossy in a way that is easy to prove in any language that uses diacritics. In English, which mostly borrows them, résumé folds to resume — a curriculum vitae and the verb meaning to start again become the same token, which is exactly the kind of collision that makes a search result list look broken to a reader. Naïve folds to naive, which is harmless because the two are the same word; résumé and resume are not.

This is not an argument against folding. It is an argument for keeping the unfolded form as well. The pattern that works is two fields: store the original text exactly as written, normalised to NFC and nothing else, and build a second folded field beside it for matching. Rank exact matches on the original above folded matches, and the reader who typed the accent gets the entry they meant first while the reader who did not still finds something. Folding as an additional index is a feature; folding as a destructive replacement of your data is a bug you will find out about later.

What to use instead of a hand-written strip

For comparison and sorting, use a collator rather than a transformation. An Intl.Collator at sensitivity base reports cafe and café as equal, Muller and Müller as equal, Orsted and Ørsted as equal, and Lukasz and Łukasz as equal — including the four letters the NFD strip cannot touch, because the collation tables know what those letters are. It also does this without producing a mangled intermediate string that then has to be stored somewhere.

For URL slugs, where you genuinely need a bounded ASCII output, keep folding — but drive it from an explicit language map first and NFD second, and check the result. A slug generator is one of the few places where a lossy, irreversible fold is correct, because a slug is not data: it is a label you can regenerate, and it is allowed to lose distinctions the original text carried. Just make sure the map is applied before NFD, or ł and ø will fall straight through into a slug and out again as unusable characters.

And whatever you settle on, write it once. The single most common cause of the bug in the title is not a bad fold — it is a good fold implemented twice, once in the indexer and once in the search box, by two people, six months apart. One function, exported from one module, called from both sides.

Frequently asked questions

Should I store the folded text or fold on the fly?

Store it, as an extra field, and never as a replacement. Folding on the fly means folding the entire index on every query, which is slow, and it makes it far too easy for one code path to fold and another to forget. A stored folded field is cheap, is built once by the same function the search box calls, and leaves the original intact for exact matching and for display. The one thing you must not do is fold in place: once the accented form is gone from your database you cannot render the name correctly on a page, and no amount of later cleverness brings it back.

Is NFKD better than NFD for this, since it also decomposes ligatures?

It solves one problem and creates several. NFKD does turn œ into oe and fi into fi, which is genuinely what you want in a search index. But compatibility decomposition also rewrites superscripts into ordinary digits, fullwidth Latin letters into ASCII, the ohm sign into an omega, and various spacing characters into plain spaces. In a display field that is destructive in ways you did not sign up for. In a matching field it is usually acceptable and often helpful. So: NFC for storage, NFKD as one input to a folded matching field if you want the ligature behaviour, and an explicit map for ø, ł and ß, which neither form will fix.

Does a database do this for me if I pick the right collation?

Largely yes, and that is usually the better answer than folding in application code. An accent-insensitive collation implements the same idea the collator does, at the level where the comparison actually happens, so the index and the query are compared under one rule by construction. Two cautions. First, collation is per column or per comparison, so a query that compares a collated column against an expression you folded yourself is back to the one-sided problem. Second, accent-insensitive collations are language-specific in exactly the ways described above, so choose the one that matches your content rather than a generic default.

Why do two strings that look identical fail an equality test?

Because one is composed and the other is decomposed. Strict equality compares code units, and a composed é is one code unit while a decomposed é is two — so the test fails while the rendering is pixel-identical. This is the single most common Unicode surprise in a search feature, and it is also the easiest to fix: normalise both operands to NFC before comparing. Note that a locale-aware comparison already reports the two as equivalent, which is a useful diagnostic — if localeCompare says zero and strict equality says false, you have found a normalisation mismatch and not a data error.

Is it ever right to fold a person's name?

For matching, yes. For display, no. Someone called Zoë Müller is entitled to see her name spelled correctly on the screen, on the invoice and in the email, and a system that stores only the folded form cannot deliver that no matter what it does later. Fold into a search key beside the record, never over it, and make sure every output path reads the original field. This is also the practical reason the two-field pattern wins: it makes the display path and the matching path structurally different, so nobody can accidentally print the search key.