How line deduplication works
The input is split on \n or \r\n into a flat array. Each line is mapped to a comparison key, which is the line itself by default, or a trimmed and lowercased copy if you flip those toggles. The dedupe mode then walks the array once, deciding which lines survive. Output is rejoined with \n.
Three modes cover the common shapes. All Duplicates keeps the first occurrence of each key and drops every later copy, regardless of position. Consecutive only collapses adjacent repeats: a, a, b, a becomes a, b, a, useful for log files where bursts of identical lines are noise but later repeats still mean something. Absolutely Unique drops every line that appears more than once anywhere in the input, leaving only the lines that occurred exactly once.
Two helper toggles tighten the match. Match Case off treats Apple and apple as the same line. Trim Compare on ignores leading and trailing whitespace, so "apple" and " apple " both hit the same key. The original line text is preserved in output; the comparison key is just for matching. Sort Output on alphabetises the surviving lines after dedupe.
How to use remove duplicate lines
- 1Paste lines into the input panel on the left.
- 2Pick a Mode: All Duplicates, Consecutive, or Absolutely Unique.
- 3Toggle Match Case off if
APPLEandappleshould match. - 4Toggle Trim Compare on if leading or trailing spaces should not affect matching.
- 5Turn on Sort Output if you want the surviving lines alphabetised.
Keyboard shortcuts
Drive TextResult without touching the mouse.
| Shortcut | Action |
|---|---|
| Ctrl F | Open the find & replace panel inside the input Plus |
| Ctrl Z | Undo the last input change |
| Ctrl Shift Z | Redo |
| Ctrl Shift Enter | Toggle fullscreen focus on the editor Plus |
| Esc | Close find & replace, or exit fullscreen |
| Ctrl K | Open the command palette to jump to any tool Plus |
| Ctrl S | Save current workflow draft Plus |
| Ctrl P | Run a saved workflow Plus |
What this tool actually does
Three dedupe modes for different jobs
All Duplicates keeps the first copy of each line. Consecutive only collapses adjacent repeats, so non-adjacent dupes survive. Absolutely Unique filters down to lines that appear exactly once across the whole input. Pick the one that matches your data shape.
Match-case toggle
When Match Case is on (default), Apple and apple are different lines. Turn it off and the comparison runs on a lowercased key, so any combination of letter case maps to the same line. The original casing of the surviving line is kept.
Trim-compare for whitespace tolerance
Flip Trim Compare on to call String.prototype.trim() on each line before keying. "apple" and " apple " then collide on apple. The surviving line keeps its original whitespace, only the comparison ignores it.
Optional alphabetic sort
Flip Sort Output on to run Array.prototype.sort() on the survivors. The default JavaScript sort is lexicographic on UTF-16 code units, which matches ASCII alphabetical order for plain English text.
Linear single pass with a hash map
Each mode runs in a single pass over the input array, with a hash map keyed by the comparison key. So a 100k-line input dedupes in one walk, no quadratic blowup. Output is computed on every keystroke in your browser, with no upload.
Worked example
Mode: All Duplicates, Match Case off, Trim Compare on. The trailing banana and the indented apple collide with the earlier copies.
apple banana apple cherry Banana apple banana
apple banana cherry
Settings reference
| Setting or behaviour | Effect on output |
|---|---|
| Mode: All Duplicates | Keeps the first occurrence of each line, drops every later copy. |
| Mode: Consecutive | Collapses only adjacent repeats. Non-adjacent dupes survive. |
| Mode: Absolutely Unique | Drops every line that appears more than once. Only lines with frequency 1 survive. |
| Match Case (default on) | When off, comparison runs on lowercased keys. |
| Trim Compare (default off) | When on, leading and trailing whitespace is ignored when matching, but kept on the surviving line. |
| Sort Output (default off) | When on, surviving lines are sorted lexicographically before output. |
| Line endings | Input is split on \n or \r\n; output is joined with \n. |
FAQ
How do I keep only lines that appear exactly once?
What is the difference between Consecutive and All Duplicates?
a, a, b, a becomes a, b, a. All Duplicates keeps the first occurrence and drops every later copy regardless of position, so the same input becomes a, b. Pick Consecutive when later repeats still mean something (event logs), All Duplicates when they do not (mailing lists).Can I dedupe case-insensitively?
Apple, APPLE, and apple all collide. The first occurrence is kept with its original casing intact in the output.Will whitespace differences cause spurious dupes?
"apple" and " apple " are different strings. Turn Trim Compare on to ignore leading and trailing whitespace when matching. The original line keeps its whitespace; only the comparison key is trimmed.