How to remove duplicate lines in a text file

Removing a duplicate line sounds trivial until you have to decide what “duplicate” means. IsAcme the same as acme? Does a trailing space count? Should the first copy survive, or the last one? And if the list is ranked, does removing a line change the order that matters?

The methods below answer those questions in different ways. Start with the one that matches the tool you already have, then use the browser tool when you want to see the removed lines before you replace the file.

First, decide what counts as a duplicate

Most tools compare whole lines as strings. That means apple, Apple, and apple are three different lines unless the tool has an option to ignore case or trim the edges. Invisible characters — a non-breaking space, a zero-width space, or a carriage return left over from a Windows file — can also make two visually identical lines compare as different.

Decide three things before you start:

• Comparison. Exact line, case-insensitive, or trimmed at the edges? If the list came from several sources, case and edge spaces are the usual culprits.
• Survivor. Keep the first copy to preserve the original order, or keep the last copy when the newest entry should win.
• Order. If the sequence carries meaning — priorities, chronology, a ranking — do not sort just to deduplicate. Use an order-preserving method.

If you are deduplicating a CSV, remember that a simple row is one line; a field that contains a line break is a different problem. Two rows with the same values in a different column order are not identical strings. For column-level deduplication, use a spreadsheet rather than a line-based tool.

Method 1: a text editor

On Windows, Notepad++ has a built-in command that removes duplicate lines without sorting. Open the file, then choose Edit → Line Operations → Remove Duplicate Lines. Review the result before saving, and save to a new file rather than overwriting the original. Notepad++ does not offer the ignore-case or trim options described above, so it is best for lists that are already consistently formatted.

In VS Code, there is no built-in dedupe command. The built-in Sort Lines command changes the order, which is exactly what you do not want for a ranked list. You can install an extension, but check what it does with case and whitespace before trusting it on a client file.

Text editors are a good fit when the file is small enough to open comfortably and you want to see the result in place. They are a poor fit when you need an audit trail of which lines were removed, or when the file is too large for the editor to handle smoothly.

Method 2: a spreadsheet

A spreadsheet is the right tool when the duplicate is a value in one column and you need to keep the rest of the row. In Excel, select the range, go to Data → Remove Duplicates, choose the column or columns that define a duplicate, and confirm whether the range has headers. Excel keeps the first occurrence and leaves the remaining rows in their original order.

Google Sheets has the same idea under Data → Data cleanup → Remove duplicates. You can also use =UNIQUE(A:A) in a spare column; it returns the unique values in the order they first appear. Both routes are convenient, but they can change how values are interpreted: long IDs may lose precision, leading zeros may disappear, and dates may be normalized. Format the column as text first if the exact string matters.

Work on a copy. “Remove Duplicates” deletes rows from the sheet, and undo is not a substitute for a backup when the file is important.

Method 3: the command line

For a plain text file, an order-preserving one-liner is often the fastest and most predictable route. The following awk command prints the first occurrence of each line and ignores later copies:

awk '!seen[$0]++' input.txt > output.txt

The seen array is keyed by the whole line. The first time a line appears, the expression is false, so awk prints it; later appearances are skipped. Nothing is sorted. If you need a case-insensitive comparison, lowercase the key:

awk '!seen[tolower($0)]++' input.txt > output.txt

To trim leading and trailing spaces and tabs before comparing, use:

awk '{ key=$0; gsub(/^[ \t]+|[ \t]+$/, "", key); if (!seen[key]++) print }' input.txt > output.txt

If you actually want a sorted unique list, sort -u is shorter — but it changes the order, so do not use it on a ranked list:

sort -u input.txt > output.txt

On Windows PowerShell, a similar order-preserving command is:

Get-Content .\input.txt | Select-Object -Unique | Set-Content .\output.txt

PowerShell’s default string comparisons are case-insensitive. If case matters, use a case-sensitive method, WSL, or the browser tool with “Ignore case” turned off. Always redirect to a new file, as shown above; redirecting to the same path truncates the input before the command can read it.

Method 4: the browser tool

The Remove Duplicate Lines tool is the option to use when you want a preview, a removed-lines list, or a case/space comparison without writing a command. It runs in your browser, so the file never leaves your computer.

1. Open the file and copy its contents into the input box. For a very large file, use the command-line method instead.
2. Choose whether to ignore case and surrounding spaces. Leave both off for an exact match.
3. Choose whether the first or last copy should survive. First is the default and preserves the original order.
4. Check the unique and duplicate counts, then open “Show removed lines” to audit what was dropped.
5. Copy or download the result, and compare it with the original before replacing anything.

red
blue
red
Red

red
blue

With “Ignore case” off, the last Red would survive as a separate line. That is not a bug; it is the difference between an exact string comparison and a case-insensitive one. Decide which behavior you need before you copy the output.

If the lines contain invisible characters, run them through Clean Text first. If you need a sorted unique list after deduping, use Sort Text Lines as a separate step so the order change is deliberate.

Check the result before you replace the file

A duplicate count is not a proof that the output is correct. Compare line counts before and after, spot-check the beginning and end of the list, and keep a copy of the original until you have opened the new file in the app that will consume it.

• On macOS or Linux, wc -l input.txt output.txt gives a quick line count. The output should have fewer lines if duplicates were removed.
• On Windows PowerShell, (Get-Content .\input.txt).Count does the same job for each file.
• Check the encoding and line endings if the file is going into a build system, a database import, or a Windows-only tool. A UTF-8 file with LF endings is not always interchangeable with a UTF-8 file that uses CRLF.
• If the list is ranked, confirm that the surviving entries are still in the intended order. Deduplication should remove lines, not reorder them.

When the file is a one-column list and you want the browser to do the whole job, the tool is the shortest path. When the duplicate is a value inside a row, or the file is too large to paste, stay with the spreadsheet or command-line method. The next guide in this series covers the other common office problem: reversing text in Word.

Page updated 29 October 2026