Formula
kept lines = lines whose position is the first (or last) position of their textduplicates removed = lines checked − different lines
- Two lines are the same when their text is identical after the options you choose (case, trimming)
- Lines keep their original order
Each distinct line is remembered the first time it is seen; later copies are dropped.
Worked examples
A list with repeats, ignoring case
- Your lines
- apple Banana apple cherry banana cherry date
- What to output
- Each line once (remove duplicates)
- Treat upper and lower case as
- The same (Apple = apple)
- Trim spaces at the start and end of each line
- Yes
- Remove blank lines
- Yes
- When a line repeats, keep the
- first occurrence
Lines kept
4
Lines checked: 7
Different lines: 4
Duplicate lines removed: 3
apple Banana cherry date
Only the repeated lines
- Your lines
- a b a c b b
- What to output
- Only lines that appear more than once
- Treat upper and lower case as
- Different (Apple ≠ apple)
- Trim spaces at the start and end of each line
- No
- Remove blank lines
- No
- When a line repeats, keep the
- first occurrence
Lines that repeat
2
Lines checked: 6
Different lines: 3
Duplicate lines removed: 0
a b
How it works
The text is split at every line break (Windows, Mac and Linux endings are all understood) and each line is compared with the ones before it. By default two lines count as duplicates only when they match exactly, so Apple and apple stay separate; switch the case option to treat them as the same. Trimming ignores spaces at the start and end of a line, which is the usual reason two lines that look identical are not removed.
Besides removing duplicates, the output can be switched to show only the lines that appear exactly once (useful for finding what is unique to a list) or only the lines that repeat (useful for finding what to clean up). The order of the surviving lines never changes, and the summary shows how many lines were checked, how many different lines there are and how many were removed. If you need the result alphabetical as well, run it through Sort Lines Alphabetically.
Comparison is exact string matching, not fuzzy matching: John Smith and Smith, John are different lines. Very large lists work, up to about two million characters, and everything happens on your device. To compare two separate lists instead of cleaning one, use the Text Diff Checker, and to tidy stray spaces first, use Remove Extra Spaces.
Assumptions and limits
- Lines are separated by line breaks; a trailing empty line is ignored.
- Matching is exact apart from the case and trim options; there is no fuzzy matching.
- Text above about 2 million characters is not accepted.