Duplicate Text Remover Tool

The tool cuts your text into units, builds a comparison key for every unit using the rules you set, groups the units whose keys are identical, and rebuilds the text with one copy from each group. Everything after that — which copy survives, what the output shows, how it is sorted, how the pieces are joined back together — is a decision you make about the same index, which is why changing those settings is instant even on a very large list.

Step by step

1. 1

   Add your text

   Paste into the box, or drop a .txt, .csv, .tsv, .log or .md file. Nothing is uploaded — the file is read on your device.

2. 2

   Choose what a duplicate is

   Pick the unit: lines, words, paragraphs, or a delimiter such as a comma. Then set the matching rules — case, trimming, spacing, punctuation and accents.

3. 3

   Decide which copy survives

   Keep first preserves the earliest occurrence; keep last preserves the final one. Either way the surviving entries stay in their original order.

4. 4

   Review, then take the result

   Check the counters and the occurrence table, switch between unique, duplicates-only and removed to see exactly what changed, then copy or download.

Removing duplicates is one operation and several questions. Most online deduplicators answer only the first of them.

• Counts every repeat — the occurrence table lists each distinct entry with how many times it appeared, and exports as CSV.
• Shows the duplicates themselves — switch the output to list only the entries that repeat, rather than the ones that do not.
• Shows what was removed — a third view lists exactly what deduplication deleted, so the operation is reviewable before it is trusted.
• Reads files, not just pastes — drop a .txt, .csv, .tsv, .log, .md or .list file; encodings and byte-order marks are handled.
• Splits on any delimiter — a comma, a semicolon, a pipe or a word — not only on line breaks.
• No hourly limit — no task counter, no queue. The work happens on your device.

The single most important setting is what counts as one unit, because everything else compares those units. Four choices cover the shapes text actually arrives in.

Lines

Each row is a unit. This is the mode for lists, logs, exported columns, URL sets, email lists and anything one-per-row. Blank lines are structure rather than content and are never compared, so a document's spacing survives.

Words

Units are runs of non-whitespace, split on any whitespace including line breaks. Use it for vocabulary extraction, tag clouds, or stripping a repeated token out of a blob of text. The output separator setting decides whether the survivors come back as a sentence, one per line, or with their original spacing preserved.

Paragraphs

A unit is a block of consecutive non-blank lines, separated from the next by one or more blank ones. This is the mode for repeated sections in notes, duplicated boilerplate in drafts, and merged documents that acquired the same passage twice. A line holding only spaces counts as blank, because to every human reading the document it is a break.

A delimiter you choose

Type any string and the text splits on it: a comma for a keyword list, a semicolon for a European CSV cell, a pipe for a log field, or a whole word if that is what separates your entries. The survivors are rejoined with the same delimiter, so the result drops straight back where it came from.

Two entries are duplicates when their comparison keys are identical. Each rule below changes how that key is built, and they combine freely — which is the difference between a set of switches and a fixed list of preset strategies.

Case

Matching ignores case by default, so 'Apple' and 'apple' are one entry. Turn case sensitivity on when the difference is meaningful — identifiers, hashes, code, anything where capitalisation carries information.

Trimming

Leading and trailing whitespace is ignored by default. You can trim only the start, only the end, or nothing at all. Trimming only the end is the right setting for an indented list, where the leading spaces are the structure you are keeping and the trailing ones are noise from an export.

Repeated spaces inside an entry

'John Smith' and 'John Smith' differ by one space. Collapsing runs of whitespace to a single space before comparing makes them one entry — useful for names, addresses and anything a human typed twice.

Punctuation

Ignoring punctuation drops the printable ASCII symbols and the four curly quotation marks a word processor substitutes, so 'St. John's' and 'St Johns' match. It deliberately leaves other scripts' punctuation alone, because a Japanese middle dot and an Arabic comma are word-forming characters this tool has no business reasoning about.

Composed and decomposed Unicode

'café' typed on a Mac and 'café' typed on Windows can be different byte sequences that render identically — one stores the accented letter as a single character, the other as a letter followed by a combining mark. Normalising composition makes them match. This is the duplicate nobody can see and almost no deduplicator catches.

Accents

Going one step further and stripping the marks entirely makes 'resume' match 'résumé' and 'Munchen' match 'München'. Use it when a list was typed by several people, or partly by someone without the right keyboard layout.

When a group of duplicates is found, exactly one member of it is kept. Which one is your choice, and the two answers are genuinely different.

• Keep first — the earliest occurrence survives. Right when the first entry is canonical, or when the order of a list was deliberate.
• Keep last — the final occurrence survives. Right for append-only logs and exports where later rows supersede earlier ones.
• Order is preserved either way — the surviving entries stay where they were in the document. Only which copy survives changes, never the sequence of the ones that do.

The index the tool builds answers three questions at once, so all three are one control rather than three tools.

Unique

Every entry once — the classic deduplication. This is what you take when you want the cleaned list itself.

Only the duplicates

Just the entries that appeared more than once, listed once each. This inverts the operation: instead of the list with the repeats removed, you get the repeats. It is the fastest way to answer 'what is doubled up in here?' without reading the whole file.

Only what was removed

Exactly the occurrences deduplication deleted, in document order. Check this before you overwrite anything — it is the difference between trusting a tool and verifying it.

Beside the result sits a table of every distinct entry and its occurrence count, sorted by count so the worst offenders are at the top. It answers a different question from deduplication: not 'give me the list without repeats' but 'which entries repeat, and how badly'.

• Sort the result itself by occurrence count to put the most repeated entries first.
• Export the complete table as a CSV with a byte-order mark, so it opens correctly in Excel with accents intact.
• The table on screen is capped for the browser's sake; the exported CSV always contains every row, on every plan.
• Blank entries are shown as such rather than as an empty row, so a count of 40 blanks is visible rather than mysterious.

Once the survivors are chosen, three settings decide what the output actually looks like. All three re-emit from the existing index, so they are instant even on a very large document.

Sorting

Leave it off to keep document order, or sort A-Z, in natural order (so 'item2' comes before 'item10'), by length, or by occurrence count. Every sort breaks ties on the original position, so the same input and settings always produce the same file.

How the pieces are joined

Lines rejoin with line breaks, words with spaces and paragraphs with blank lines by default — but you can force any of them to newlines, spaces, commas or tabs, or ask for the original spacing between each surviving unit to be preserved exactly.

Line endings

The output keeps whichever line ending dominated your input, or can be forced to Unix (LF) or Windows (CRLF). A file that ended with a line break still ends with one, because every downstream shell loop and spreadsheet import expects that and losing it breaks pipelines quietly.

Free handles up to 25 MB of text at once — roughly 700,000 lines of a typical export — and every mode, matching rule, output view and export works on it. Nothing about the result is held back by plan; Pro raises the ceiling to what your device can handle and adds multi-file batches with one archive download.

• Everything runs on your device, not our servers. Nothing is uploaded, and there is no queue and no hourly task limit.
• Files are read as UTF-8, and UTF-16 or legacy Western encodings are detected rather than turned into replacement characters.
• A byte-order mark is stripped, so your first entry is not silently exempted from matching by an invisible character.
• Accepted files: .txt, .text, .csv, .tsv, .log, .md and .list. Anything else is refused at the point you pick it, with a link to the tool that can open it.
• Very large inputs are bounded by your device's memory. The preview on screen is windowed; copy and download always carry the whole result.

Deduplication runs on your device, off the main thread, so a 25 MB list stays responsive while it works.

Deduplication deletes data. A few habits make the difference between a clean list and a quietly damaged one.

• Look at the 'removed' view before you overwrite the source. It is one click and it shows exactly what would be lost.
• If intentional repetition matters — poetry, code, test fixtures, a chorus — check the result rather than assuming.
• Structured data deserves a column-aware tool. Deduplicating a CSV by whole lines ignores which column actually identifies a row.
• Paragraph mode needs clear blank-line boundaries; a document using indentation instead will read as one block.
• Trim-aware matching treats spaced variants as the same entry, which is usually right and occasionally not. Turn it off when whitespace is data.
• On a very large paste, try the settings on a sample first — the answer arrives faster and the decision is the same.

Structural deduplication compares keys exactly under the rules you set. Three cases need something else.

• Near-duplicates and typos. 'Jon Smith' and 'John Smith' are different entries here and always will be — that is fuzzy matching, a different problem with different failure modes.
• Rows where duplicates carry different data you need to merge rather than discard. A spreadsheet or a database can combine them; this tool can only keep one.
• Uniqueness defined by a column or a key rather than by the whole line. Use the CSV duplicate remover, which matches on the columns you choose.

For everything else, deduplicate first to cut the noise, then sort the unique output for review, trim stray whitespace, or run a quick analysis over what is left.

• Text sorter
• Text merger
• Text trimmer
• Text splitter
• Text analyzer
• Remove duplicate CSV rows