How word deduplication works
The text is split on whitespace into words, with the runs of spaces, tabs, and newlines between them kept as separators. Each word is mapped to a comparison key: the word itself, lowercased unless Match case is on. Walking the words left to right, the first time a key is seen the word is kept; a later word with the same key is dropped along with the single space that preceded it, so the surrounding spacing stays clean.
Two modes cover the common shapes. All Duplicates keeps the first occurrence of each word and drops every later copy anywhere in the text, so the cat and the dog becomes the cat and dog. Consecutive only collapses back-to-back repeats, which fixes stutters like the the cat or a doubled is is without touching words that legitimately recur later.
Two toggles tune the match. Ignore punctuation strips leading and trailing punctuation from the comparison key, so cat,, cat!, and cat all count as the same word while the surviving copy keeps its own punctuation. Per line restarts the seen-word set on every line, so a word repeated on two different lines survives once per line rather than once for the whole text.
How to use remove duplicate words
- 1Paste or type your text into the input panel on the left.
- 2Pick a Mode: All Duplicates to drop every later repeat, or Consecutive to collapse only adjacent repeats.
- 3Leave Match case off to treat
Theandtheas one word, or turn it on to keep them separate. - 4Turn on Ignore punctuation if
cat,andcatshould match. - 5Turn on Per line to dedupe each line independently instead of across the whole text.
Keyboard shortcuts
Drive TextResult without touching the mouse.
| Shortcut | Action |
|---|---|
| Ctrl F | Open the find & replace panel inside the input Plus |
| Ctrl Z | Undo the last input change |
| Ctrl Shift Z | Redo |
| Ctrl Shift Enter | Toggle fullscreen focus on the editor Plus |
| Esc | Close find & replace, or exit fullscreen |
| Ctrl K | Open the command palette to jump to any tool Plus |
| Ctrl S | Save current workflow draft Plus |
| Ctrl P | Run a saved workflow Plus |
What this tool actually does
First occurrence wins
In All Duplicates mode the earliest copy of each word is kept and every later copy is removed, however far apart they are. Word order is otherwise preserved: nothing is reordered, only later repeats are deleted.
Consecutive mode for stutters
Consecutive only collapses adjacent repeats, so the the cat is is here becomes the cat is here but a word that comes back later in the sentence is left in place. Useful for cleaning dictation or transcription doubles.
Case-insensitive by default
With Match case off, the comparison runs on a lowercased key, so The, THE, and the collapse to the first spelling seen. Turn it on and each casing is treated as a distinct word.
Punctuation-aware matching
By default a trailing comma or period makes a word distinct, because the punctuation is part of the token. Turn Ignore punctuation on to strip leading and trailing punctuation from the key, so cat, and cat match. The kept word keeps whatever punctuation it had.
Whitespace stays clean
Removing a word also removes the single space that preceded it, so you never get double spaces where a duplicate used to be. Newlines are preserved as separators, and Per line keeps each line deduped on its own.
Worked example
Mode: All Duplicates, Match case off. THE and the collide with The, and Cat collides with cat, so only the first spelling of each word survives.
The THE the cat Cat dog
The cat dog
Settings reference
| Setting or behaviour | Effect on output |
|---|---|
| Mode: All Duplicates | Keeps the first occurrence of each word, drops every later copy. |
| Mode: Consecutive | Collapses only adjacent repeats. Non-adjacent repeats survive. |
| Match case (default off) | When off, comparison runs on lowercased words. When on, casing matters. |
| Ignore punctuation (default off) | When on, leading and trailing punctuation is stripped from the comparison key. |
| Per line (default off) | When on, each line is deduped separately, so a word can survive once per line. |
| Word boundary | Words are whitespace-separated tokens; runs of spaces, tabs, and newlines are the separators. |
FAQ
Does it remove every repeat or only doubled words?
the the, leaving a word that legitimately appears again later untouched.Are Cat and cat treated as the same word?
Cat and cat as two different words.What about a word followed by a comma or period?
cat, and cat are different, because the punctuation is part of the token. Turn Ignore punctuation on to strip leading and trailing punctuation before comparing, so they match. The surviving word keeps its own punctuation.