Duplicate Line Remover

About this tool

The Duplicate Line Remover treats your text as a list, one item per line, and keeps only the first appearance of each distinct line. It is handy for email lists, keyword lists, product codes, tags, or any pasted column where each entry should appear once. Everything runs in your browser; the list is never sent anywhere.

How to use the Duplicate Line Remover

1. Paste your list, one item per line.
2. Set your options — ignore case, trim whitespace, or sort alphabetically.
3. Click "Remove duplicates" and copy the cleaned-up result.

How lines are matched

The tool splits your text at each line break and reads the lines from top to bottom, remembering every line it has already seen. A line that matches an earlier one is dropped; anything new is kept. Matching is exact, character for character, after your options are applied:

• Trim whitespace (on by default) removes spaces and tabs from the start and end of each line before comparing — and the trimmed version is what appears in the result. With it off, omar@example.com and the same address followed by a stray space count as different lines.
• Ignore case compares lowercase copies, so Apple and apple match. The result keeps the capitalisation of whichever version came first.
• Sort alphabetically runs after duplicates are removed and uses your browser's language-aware comparison, so b and B sit next to each other and accented letters sort beside their plain forms.

Without sorting, the surviving lines stay in their original order. The count above the result is the number of unique non-empty lines; a kept blank line isn't counted.

Worked example

Suppose you've combined signups from two forms into this nine-line list. The fourth line starts with two spaces, the sixth has a trailing space after the address, and there are two blank lines:

Jane@Example.com
omar@example.com
jane@example.com
  lee@example.com

omar@example.com 
Lee@example.com

zoe@example.com

With only the default Trim whitespace option on, the tool reports 6 unique lines. The second omar@example.com is removed once its trailing space is trimmed, and the second blank line goes too, but Jane@Example.com and jane@example.com both survive, as do lee@ and Lee@, because their capitalisation differs. Turn on Ignore case as well and the count drops to 4:

Jane@Example.com
omar@example.com
lee@example.com

zoe@example.com

Adding Sort alphabetically gives the same four addresses in A–Z order, with the one remaining blank line moved to the very top, since an empty line sorts before any text.

A note on case and email addresses

Ignore case is usually the right choice for email lists, but it is a convention rather than a rule. Domain names are not case-sensitive, while the email standard RFC 5321 says the part before the @ must be treated as case-sensitive — though it also discourages anyone from relying on that. In practice most mail services treat Jane@ and jane@ as the same mailbox, so merging them avoids sending the same message twice. For data where case really does matter, such as product codes where ab12 and AB12 are different items or file paths on Linux, leave Ignore case off.

Common use cases

• Cleaning up an email list before a mail merge or campaign send
• Deduplicating a keyword or tag list for SEO or content planning
• Merging two lists of names or IDs into one unique set
• Tidying a list pasted from a spreadsheet with accidental repeats

Edge cases and limits

• Blank lines are treated like any other line, so several collapse into one. A line break at the very end of your paste also counts as an empty line and shows up at the end of the result.
• Windows line endings. Text copied from some Windows programs ends each line with an invisible carriage return. Trim whitespace strips it; with trim off, the last line of a list can fail to match an identical line above it.
• Numbers sort as text. Alphabetical sorting puts 10 before 9.
• Only whole lines are compared. Repeats inside a single line, such as a comma-separated list, aren't detected — put one item per line first. Near-duplicates like "123 Main St." and "123 Main Street" aren't matched either; there's no fuzzy matching.
• The first occurrence wins. If the later entries are the corrected ones, reverse the list before pasting it in.

When to use another tool

If you want to de-duplicate and sort in one pass with more control over the order — natural order (file2 before file10), by number, by length, or Z–A — the Text Sorter has a remove-duplicates option alongside its sort modes. To remove blank lines and extra spaces entirely rather than collapsing them, use the Text Cleaner.

When identical-looking lines are both kept

The tool compares characters, not appearance, so two lines that look the same on screen can still be different underneath. We tested the common causes against the tool's own matching rules:

• Non-breaking spaces (U+00A0), often copied from web pages and word processors, are removed by Trim whitespace when they sit at the start or end of a line, the same as ordinary spaces. Inside a line they still count as a different character from a normal space.
• Zero-width spaces (U+200B) are invisible and are not treated as whitespace, so trimming does not remove them. A line with one hidden at the start will never match the clean version. The Text Cleaner can help strip hidden formatting first.
• A byte order mark (U+FEFF) at the very start of a pasted file is removed by trimming.
• Accents typed two ways. Unicode can store é as one character (U+00E9) or as e followed by a combining accent (U+0301). They display the same, but the tool does not normalize text, so they count as different lines.

Limits of Ignore case

Ignore case works by converting each line to lowercase with your browser's standard rules before comparing. That handles almost every everyday list, but a few letters do not have simple one-to-one pairs. The German ß stays ß in lowercase, so STRASSE and straße are not matched. The Turkish capital İ (with a dot) lowercases to two characters, an i plus a combining dot, so it does not match a plain i. If your list is in a language with these letters, check the result by eye.

Removing duplicate words rather than lines

The tool only compares whole lines, so repeated words inside one sentence or a comma-separated list are not touched. To deduplicate words, first put each word or item on its own line. Most text editors and spreadsheet programs can replace commas or spaces with line breaks. Then paste the list here, remove duplicates, and join the items again if you need them on one line. Keep Trim whitespace on so the spaces left over from the commas are ignored.

Seeing how many duplicates were removed

The result shows the number of unique lines, not the number removed. To get the removed count, count the lines you pasted and subtract the unique count. Remember that blank lines are included in your input but not in the unique total. To see exactly which lines disappeared, paste the original and the result into the Text Diff Checker.