Remove duplicate lines
Delete repeated lines and keep the first of each, in the order they were already in. Useful for email lists, log extracts, keyword lists and anything else that came out of a merge.
Order is preserved, which sort | uniq cannot do
The standard shell idiom for this is sort file | uniq, and the sort is not optional: uniq only removes duplicates that are adjacent, so the list has to be sorted first. That works, and it destroys the order.
For a lot of lists the order is the content. A playlist, a set of steps, a changelog, a keyword list ranked by volume, a set of URLs in crawl order — sorting them alphabetically to remove three duplicates loses more than it fixes. This keeps the first occurrence of each line exactly where it was and drops the rest.
What counts as a duplicate
By default, an exact match: same characters, same case, same leading and trailing spaces. That is the safe default, because a tool that silently treats Apple and apple as the same line has made an editorial decision on your behalf.
Turn off "match case" when the list is of things that are case-insensitive in reality — email addresses, domain names, tags, SQL column names — where two spellings genuinely are one entry.
The one that catches people is invisible: two lines that look identical but differ by a trailing space are not duplicates. If a list refuses to dedupe when you can see the repeats, run "remove extra spaces" over it first and try again.
Worked example
Before
newsletter sales@example.com newsletter support@example.com sales@example.com
After
newsletter sales@example.com support@example.com
Common questions
How do I remove duplicate lines but keep the order?
Paste the list and click Remove duplicate lines. The first occurrence of each line stays exactly where it was and later repeats are dropped — nothing is sorted.
Is this the same as sort and uniq?
It produces the same set of lines but keeps the original order. The shell idiom sort | uniq has to sort the list first, because uniq only removes duplicates that are next to each other.
Why are lines that look identical not being removed?
Almost always a trailing space, a tab, or a difference in case. Run Remove extra spaces first, or turn off "match case", and try again.
Other text tools
- Case converterAll tools
- Sentence case converterCase
- Title case converterCase
- Uppercase converterCase
- Lowercase converterCase
- camelCase converterCode
- Snake case converterCode
- Kebab case converterCode
- Remove line breaksLines
- Remove extra spacesLines
- Sort lines alphabeticallyLines
- Reverse textLines
- Word counterCount