How to Remove Duplicate Lines From a List

A mailing list exported twice, a contact list merged from two sources, a log file with repeated entries — duplicate lines are one of the most common data-cleanup problems, and also one of the easiest to get subtly wrong by hand. Here's how to remove them properly.

The Two Decisions That Actually Matter

Deduplicating a list sounds like it only has one setting (remove the duplicates), but there are two choices that change the result:

Case sensitivity. Should "Apple" and "apple" count as the same line, or different lines? If your list has inconsistent capitalization from being merged across sources, case-insensitive comparison catches duplicates a strict comparison would miss.

Which occurrence survives. When a line appears more than once, do you keep the first one you see, or the last one? This matters more than it sounds — if a later duplicate has slightly more complete data (a contact list where the second entry has a phone number the first one is missing), you'd want to keep the last occurrence instead of the default first.

Step by Step

1. Open Remove Duplicate Lines.
2. Paste your list, one item per line.
3. Choose case sensitivity and whether to trim whitespace before comparing.
4. Choose whether to keep the first or last occurrence of each duplicate.
5. Copy or download the result — the number of lines removed is shown above it.

A Worked Example

Input:

apple
banana
apple
orange
banana

With case-sensitive comparison and "keep first occurrence," the result is:

apple
banana
orange

Two duplicate lines were removed, and the surviving lines kept their original relative order — "apple" and "banana" stayed at the positions where they first appeared, rather than getting shuffled to wherever the last occurrence happened to be.

Why "Trim Whitespace" Matters

If your list came from a spreadsheet export or was pasted from somewhere with inconsistent formatting, two lines that look identical might not actually be — one could have a trailing space the other doesn't. Without trimming, those count as different lines and don't get deduplicated, which is a common source of "why didn't this remove my duplicate?" confusion. Turning on "Trim whitespace" fixes this by ignoring leading/trailing spaces during comparison.

Frequently Asked Questions

Does the order of my list change? No — surviving lines always keep their original relative order; only the duplicate lines are removed.

What's the difference between keeping the first vs. last occurrence? If a line appears more than once, "first occurrence" keeps it at the earliest position and removes later copies. "Last occurrence" does the reverse — the version and position that survive are the one that appeared last.

Can I ignore case when comparing? Yes — turn off "Case sensitive" to treat "Apple" and "apple" as duplicates.

Does this upload my list anywhere? No. Deduplication happens entirely in your browser.

What if two lines look the same but aren't being caught as duplicates? Turn on "Trim whitespace" — a trailing space or tab that isn't visible can make two visually identical lines register as different.

Try It

Cleaning up a list with repeated entries? QR Ivify's free Remove Duplicate Lines tool handles case sensitivity and whitespace correctly. Need to sort the result afterward? Try Text Sorter.