Input
Unique Lines
Paste a list and remove the duplicate lines without reordering what is left. Ignore case and stray spaces so Bob@Example.com and bob@example.com count as one line, or set Output to Only duplicates to list the lines that repeat.
alice@example.com
bob@example.com
Alice@Example.com
carol@example.com
bob@example.com
dave@example.com
bob@example.comalice@example.com
bob@example.com
carol@example.com
dave@example.comFirst occurrence keeps each line where it first appeared. Last occurrence keeps the final copy at that copy's position. Use Last for a log or a settings file, where the later entry wins.
Compares lowercased copies but outputs the line exactly as it was written in the copy you kept. For example, Alice@Example.com stays capitalized if it came first.
Strips leading and trailing spaces and tabs before comparing, and outputs the trimmed line. Leave Trim whitespace on. A trailing space from a spreadsheet export is the usual reason a duplicate survives.
Lists each line that appears more than once, one copy each. Use it to find the double-booked slot or the email that signed up twice, instead of dropping the repeat without seeing it.
sort -u dedupes but also sorts, and
uniq only removes repeats that sit next to each
other, which is why the two are usually piped together. To keep
the original order, use awk '!seen[$0]++' file.txt.
It prints a line only while its counter is still zero, which is
the first time awk sees it. Use tolower($0) as the
key to ignore case.
Two lines that look identical but both survive usually differ
in something invisible: a trailing space, a Windows
\r, or a non-breaking space pasted from a web page.
This tool splits on CRLF, CR and LF alike, and Trim whitespace
removes ordinary and non-breaking spaces at the ends. However,
a zero-width space in the middle of a line still makes it a
different line.
const text = "alice@example.com\nbob@example.com\nAlice@Example.com \nbob@example.com\n";
const lines = text.split(/\r?\n/).filter((l) => l.trim() !== "");
const key = (l) => l.trim().toLowerCase();
// Keep the first copy, preserve order
const seen = new Set();
const unique = lines.filter((l) => !seen.has(key(l)) && seen.add(key(l)));
console.log(unique.join("\n"));
// Keep the last copy: dedupe the reversed list, then flip it back
const seenLast = new Set();
const lastWins = lines.toReversed().filter((l) => !seenLast.has(key(l)) && seenLast.add(key(l))).reverse();
console.log(lastWins);
// Only the lines that repeat
const counts = new Map();
for (const l of lines) counts.set(key(l), (counts.get(key(l)) ?? 0) + 1);
console.log([...counts].filter(([, n]) => n > 1).map(([k]) => k));Set.prototype.add returns the set, which is truthy, so the filter keeps a line the first time it appears and skips it afterwards.
# Keep the first copy of each line, original order
awk '!seen[$0]++' emails.txt
# Same, ignoring case and trailing whitespace
awk '{ k = tolower($0); sub(/[ \t\r]+$/, "", k) } !seen[k]++' emails.txt
# Sorted and unique (loses the original order)
sort -u emails.txt
# Only the lines that appear more than once, with counts
sort emails.txt | uniq -cduniq on its own removes only repeats that sit next to each other, which is why it is usually paired with sort.
from collections import Counter
lines = ["alice@example.com", "bob@example.com", "Alice@Example.com ", "bob@example.com"]
# dict keeps insertion order, so this dedupes and keeps the first copy
print(list(dict.fromkeys(lines)))
# Case- and whitespace-insensitive, keeping the original text of the first copy
unique = {}
for line in lines:
unique.setdefault(line.strip().casefold(), line)
print(list(unique.values()))
# Lines that repeat, with counts
counts = Counter(l.strip().casefold() for l in lines)
print({k: n for k, n in counts.items() if n > 1})Two different email strings can reach the same mailbox. The domain part is case-insensitive, but the local part (before the @) technically is not, and Gmail ignores dots and anything after a plus sign. Ignore case is right for most sign-up lists. Provider-specific rules such as Gmail's need code.
This tool follows the specifications below. Read them when you need the exact normative wording, for example to settle an edge case between two implementations.
That is the default. Each line stays where its first copy appeared and later repeats are dropped. Tick Sort result if you also want the output in order.
Yes. Ignore case compares lowercased copies, and Trim whitespace (on by default) strips leading and trailing spaces before comparing. With Trim on, the output lines are trimmed too.
First occurrence keeps each line at the position of its first copy. Last occurrence keeps the final copy at that copy's position. Use Last when later entries override earlier ones, as in a log or a settings file.
Set Output to Only duplicates. Each line that appears more than once is listed once, and the counter shows how many distinct lines repeat.
With Remove empty lines on (the default) they are dropped. With it off, a blank line is treated like any other line, so several blank lines collapse into one.
Sort lines puts the list in order and can dedupe in the same pass. To see what changed between two lists, use the diff checker, and for bulk edits to every line there is find and replace.