Finding Duplicates in a List Without a Spreadsheet

What the tool actually returns

It counts every line, keeps the ones whose count is greater than one, and prints each of those once. On the input b, a, b, a, c it returned b then a — in that order, because b appears first, not because b repeats more often. Nothing is sorted, nothing is removed from your list, and the number of times each value repeats is not shown. If you want the list with the repeats taken out, that is a different tool, remove-duplicate-lines, and it keeps the first occurrence of each value in its original position.

The text it prints is the first occurrence, verbatim. That sounds like a detail until you turn on Ignore case: given Apple, banana, APPLE it prints Apple, not APPLE and not apple. It has decided the two are the same value, and it has to show you one of them, so it shows the one it met first. If your list is a mix of shouted and normal spellings, this means the output tells you which spelling arrived first, not which is more common or which is correct.

The trailing space, and the sibling tool that disagrees

Paris above Paris-with-one-trailing-space produced no duplicates, and so did Paris above space-Paris. The two lines differ by one character, that character is invisible, and the tool has no trim option to switch on. Meanwhile remove-duplicate-lines carries a Trim lines toggle that is on out of the box, so the very same two lines go in and one line comes out. Two tools in the same category, one page apart, giving opposite answers about the same input. Both are defensible: a duplicate finder that trimmed would be lying about what is in your file, and a duplicate remover that did not trim would leave you with a list that still looks duplicated.

The practical consequence is an order of operations. Run trim-lines over the list first, then the finder. On a deliberately messy input — two spaces, alpha, two spaces, then a tab-only line, then two spaces, beta, one space, then an empty line, then alpha — the finder reported no duplicates on the raw text and reported alpha on the same text after trim-lines and remove-blank-lines had been through it. Nothing about the list changed in any way a reader would notice. What changed is whether the tool could see what the reader could.

Two spellings of the same accented word

Unicode can write an accented letter two ways. Either as one code point that already carries the accent, or as the plain letter followed by a separate combining mark. Both render as the same glyph, both are correct, and which one you get depends on where the text came from — some operating systems and some text-input paths produce the decomposed form by default. Unicode Standard Annex #15 defines the normalisation forms that map one onto the other, and JavaScript exposes them through a string method. duplicate-line-finder does not call it.

So a list holding the word for coffee written once each way came back as no duplicates. The two lines are visually indistinguishable and four versus five code points long. Then the wrinkle: add a third copy in the first spelling and the tool suddenly reports a duplicate, because two of the three now match each other exactly. That is the shape of the failure to watch for. It is not that the tool never finds accented duplicates; it is that it finds some of them, which makes the misses much harder to notice. If your list is French, Spanish, Portuguese, German or Italian and the counts look slightly low, this is the first thing to test — and the way to test it is to sort the list first, where the two spellings will land next to each other and reveal themselves by refusing to look different.

Ignore case is not the same as fold the language

The toggle lower-cases both sides with the language-independent mapping JavaScript provides. That covers the ordinary cases in all six of our languages, and it covers accented letters correctly — the capital and lower-case forms of an accented vowel fold together. It does not do anything a specific language would want beyond that. The German capital form written as two letters and the sharp s do not fold together, which was checked: a two-line list holding the upper-case and the lower-case spelling of a common German street word came back as no duplicates, because lower-casing the shouted form gives two letters and the other line has one.

The general point is that Ignore case answers a typographic question, not a linguistic one. Two lines that a French reader would call the same entry — one with the accent typed and one without — are not duplicates here and never will be, because removing accents is a different operation with a different tool. If that is the comparison you need, strip the accents from a copy of the list first, run the finder on the copy, and use its output as a list of things to go and check in the original.

A workable routine

Clean, then find, then decide, then remove. trim-lines and remove-blank-lines put every line into a comparable shape. duplicate-line-finder tells you which values repeat, without touching anything. You look at that short list and decide whether each repeat is a genuine double entry or two different things that happen to share a name — the tool cannot know, and this is the step people skip. Only then does remove-duplicate-lines go over the original. Doing it in that order means you never delete a line you have not looked at, which matters when the list is customer references rather than test data.

One last detail from the runs. Both tools split on a line feed with an optional carriage return in front, so a list saved on Windows and the same list saved on a Mac give identical results — a CRLF file with two copies of the letter a reported one duplicate, exactly as the plain version did. And a trailing newline at the end of your list creates one final empty line, which is counted but, on its own, never printed.

Frequently asked questions

The finder says there are no duplicates but I can see two identical lines. What is going on?

Three candidates, in order of likelihood. A trailing or leading space on one of them, which the tool does not trim. A no-break space instead of an ordinary one somewhere inside, which happens constantly in text pasted out of documents. Or two different Unicode spellings of the same accented letter. Test them in that order: run trim-lines over the list and try again, which settles the first candidate in seconds. If it still finds nothing and the lines contain accents, that is the third candidate, and sorting the list will put the two spellings adjacent so you can compare them side by side.

Why is there both a duplicate finder and a duplicate remover?

Because the two answer different questions and have different default settings. The finder reports what repeats and changes nothing, so you can look before you cut; it does not trim, which makes it a faithful report of what is literally in the file. The remover produces a cleaned list and trims by default, which makes it forgiving of the invisible junk a paste brings along. On a list of two lines differing only by a trailing space, the finder says there is nothing to report and the remover deletes one of them. That is not a bug in either; it is the reason to run the finder first when the list matters.

Does it tell me how many times each value repeats?

No. It counts internally, uses the count to decide whether a value is worth printing, and then prints only the value. On b, a, b, a, c it returned b and a with no numbers attached, and there is no option to show them. If you need the counts, the practical route is to sort the list and read the runs, or to use a word or occurrence counter on the values you already know repeat. The finder's job is to give you a short list of suspects, and it is deliberately narrow about it.

Will it treat an email address in two different cases as one address?

Only if you switch Ignore case on, and you should — the domain half of an address is case-insensitive by specification, so the same mailbox written two ways is genuinely one address. With the toggle off, the tool reports nothing on a two-line list holding a mixed-case and a lower-case spelling of the same address, which was checked directly. This is worth knowing before you deduplicate a contact export, because a list that looks clean at 1 200 rows may hold two dozen of these, and every one of them is a person who receives the mailing twice.

Can I use this on a list of thousands of lines?

Yes. The work is one pass to count and one pass to collect, using a hash map keyed on the line, so the time grows roughly in step with the number of lines rather than with the square of it — a naive comparison of every line against every other would not survive a few thousand rows, and this is not that. Everything runs in the page, on your machine, with no upload, so the practical ceiling is your browser's memory rather than any server limit. The part that will slow you down is not the tool; it is reading the output and deciding what each repeat means.