🧹 Remove Duplicate Lines
By ToolNimba Editorial Team · Updated 2026-06-19
Paste a list above to remove duplicate lines.
This tool removes duplicate lines from any list you paste in. It keeps the first occurrence of each line and strips out the repeats, then shows you how many lines came in, how many were unique, and how many were removed. You can match lines case-insensitively, trim whitespace, drop blank lines, and either keep the original order or sort the result. Everything runs in your browser, so your text never leaves your device.
What is the Remove Duplicate Lines?
A duplicate line is a line whose text matches another line you have already seen. Deduplicating a list means scanning it top to bottom and keeping only the first time each distinct line appears, discarding every later copy. This is one of the most common cleanup jobs in everyday data work: lists of emails, URLs, product codes, names, and log entries all tend to pick up repeats when you merge files or copy across sources.
The tricky part is deciding what counts as the same line. Are "Apple" and "apple" duplicates? Is " apple" (with a leading space) the same as "apple"? The honest answer is that it depends on your data, which is why this tool gives you switches rather than guessing. Turn on case-insensitive matching to fold differences in capitalisation together, and turn on trimming to ignore stray leading and trailing spaces that often sneak in when text is pasted from spreadsheets or PDFs.
Once the duplicates are gone you can leave the lines in their original order, which matters when sequence carries meaning (a playlist, a step list, a changelog), or sort them alphabetically to make the list easier to scan and compare. The line counts update live so you can immediately see how much was trimmed, which is a quick sanity check that the dedupe did roughly what you expected.
When to use it
- Cleaning a mailing list so the same email address is not sent two copies of a message.
- Combining several exported CSV or text files and stripping out the rows that overlap.
- Tidying a list of keywords, tags, or URLs before importing them into another tool.
- Removing repeated entries from a log or error dump to see the distinct messages.
How to use the Remove Duplicate Lines
- Paste your list into the input box, with one item per line.
- Turn on "Ignore case" if capitalisation should not matter.
- Keep "Trim whitespace" on to ignore stray spaces around each line.
- Choose whether to keep the original order or sort the result A to Z or Z to A.
- Read the line counts, then click Copy to grab the cleaned list.
Formula & method
Worked examples
A 5-line list with two exact repeats, keeping the original order.
- Input lines: apple, banana, apple, cherry, banana
- Line 1 "apple" is new, keep it.
- Line 2 "banana" is new, keep it.
- Line 3 "apple" was already seen, drop it.
- Line 4 "cherry" is new, keep it.
- Line 5 "banana" was already seen, drop it.
Result: 3 unique lines (apple, banana, cherry), 2 removed.
Case-insensitive matching folds "Email" and "email" together.
- Input lines: Email, EMAIL, email, phone
- With "ignore case" on, all three email variants share the key "email".
- Line 1 "Email" is kept as the first occurrence.
- Lines 2 and 3 match an existing key, so they are dropped.
- Line 4 "phone" is new, keep it.
Result: 2 unique lines (Email, phone), 2 removed.
How each option changes what counts as a duplicate
| Option | Effect | Example treated as duplicates |
|---|---|---|
| Ignore case | Capitalisation is ignored when comparing lines. | "Apple" and "apple" |
| Trim whitespace | Leading and trailing spaces and tabs are stripped before comparing. | " cat" and "cat " |
| Drop empty lines | Blank or whitespace-only lines are removed entirely. | An empty line between entries |
| Keep order | First occurrence of each line stays in its original position. | Order preserved, repeats removed |
| Sort A to Z / Z to A | Unique lines are reordered alphabetically after deduping. | Order changed, repeats removed |
Common mistakes to avoid
- Expecting near-duplicates to be caught. This tool removes lines that are identical (after your chosen case and trim options). It will not merge "John Smith" and "John Smith" with different internal spacing, or fix typos. Clean those by hand first if needed.
- Forgetting that trailing spaces hide duplicates. Two lines that look identical may differ by an invisible trailing space, so they survive as separate lines. Keep "Trim whitespace" on to fold them together.
- Sorting when order was meaningful. If your list is a sequence (steps, a playlist, a timeline), choosing a sort option will scramble it. Use "Keep original order" to remove repeats without reshuffling.
- Assuming case is ignored by default. By default the match is case-sensitive, so "USA" and "usa" are treated as two different lines. Turn on "Ignore case" if they should count as the same.
Glossary
- Duplicate line
- A line whose text matches a line that already appeared earlier in the list, under the current matching options.
- Deduplicate (dedupe)
- To remove repeated entries from a list so that each distinct value appears only once.
- Case-insensitive
- A comparison mode that treats uppercase and lowercase letters as equal, so "A" and "a" match.
- Trim
- To remove whitespace (spaces, tabs) from the start and end of a line before comparing or output.
- Whitespace
- Invisible characters such as spaces and tabs that take up space but show no mark.
Frequently asked questions
How do I remove duplicate lines from a list?
Paste your list into the box above with one item per line. The tool keeps the first copy of each line and removes the rest automatically, then shows the unique result you can copy.
Does it keep the first or last copy of a duplicate?
It keeps the first occurrence of each line and removes every later copy. When you choose "Keep original order", that first copy also stays in its original position.
Can it ignore capitalisation when finding duplicates?
Yes. Turn on "Ignore case" and lines like "Apple" and "apple" are treated as the same line, so only one is kept. With it off, the comparison is case-sensitive.
Will it sort my lines or keep the original order?
You choose. "Keep original order" preserves the sequence and just drops repeats. "Sort A to Z" or "Sort Z to A" reorder the unique lines alphabetically after deduping.
Is my text uploaded anywhere?
No. The deduplication runs entirely in your browser using JavaScript. Your text is never sent to a server, so it stays private on your own device.
How many lines can it handle?
It comfortably handles thousands of lines because the work happens locally and uses an efficient lookup. Extremely large pastes (hundreds of thousands of lines) may feel slow depending on your device.