Linux

The uniq Command

uniq removes adjacent duplicate lines. That's the key word: adjacent.

Basic Usage

Terminal
$cat file.txt
apple apple banana apple orange orange
$uniq file.txt
apple banana apple orange

Notice: the third "apple" wasn't removed because it's not adjacent to the first two.

Sort First!

uniq only removes adjacent duplicates. To remove ALL duplicates, sort first:

hljs bash
sort file.txt | uniq

Or just use sort -u.

The Sort + Uniq Pattern

Terminal
$cat file.txt
apple apple banana apple orange orange
$sort file.txt | uniq
apple banana orange

This is the standard pattern: sort | uniq.

Count Occurrences

Terminal
$sort file.txt | uniq -c
3 apple 1 banana 2 orange

-c counts how many times each line appears.

Most Common Pattern

Find the most common items:

hljs bash
sort data.txt | uniq -c | sort -rn | head

This is incredibly useful for log analysis.

Show Only Duplicates

Terminal
$sort file.txt | uniq -d
apple orange

-d shows only lines that appear more than once.

Show Only Unique Lines

Terminal
$sort file.txt | uniq -u
banana

-u shows only lines that appear exactly once.

Case-Insensitive

Terminal
$echo -e 'Apple\napple\nAPPLE' | sort | uniq -i
Apple

-i ignores case differences.

Real-World Examples

Top 10 IP Addresses

Terminal
$cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -10
1523 192.168.1.1 892 10.0.0.5 ...

Find Duplicate Lines

Terminal
$sort file.txt | uniq -d
(lines that appear more than once)

Count Unique Errors

Terminal
$grep 'ERROR' app.log | sort | uniq -c | sort -rn
45 ERROR: Connection timeout 23 ERROR: Invalid input 5 ERROR: Out of memory

Files Modified by User

Terminal
$ls -l | awk '{print $3}' | sort | uniq -c
15 root 23 user 2 www-data

Uniq vs Sort -u

These are almost the same:

hljs bash
sort file.txt | uniq      # Classic way
sort -u file.txt          # Shorter

Use sort -u when you just want unique lines.

Use sort | uniq -c when you need counts.

Knowledge Check

Why doesn't `uniq file.txt` always remove all duplicates?

Quick Reference

FlagEffect
-cCount occurrences
-dOnly show duplicates
-uOnly show unique (non-duplicates)
-iCase-insensitive

Key Takeaways

  • uniq removes adjacent duplicates only
  • Always sort first: sort | uniq
  • -c counts occurrences - incredibly useful
  • sort file | uniq -c | sort -rn | head = top N most common
  • -d finds duplicates, -u finds unique lines
  • sort -u is shorthand for sort | uniq

Next: counting with wc.