How to print only the duplicate values from a text file?

Suppose there is a column of numeric values like following:

File1:

1 
2
3
3
3
4
4
4
5
6

I want the output:

3  
4

That is, only the repeated lines. Are there any command line tools to find this out in Linux? (NB: The values are numerically sorted).

Asked By: N. F.

||

You can use uniq(1) for this if the file is sorted:

uniq -d file.txt

If the file is not sorted, run it through sort(1) first:

sort file.txt | uniq -d

This will print out the duplicates only.

Technically the input does not need to be in sorted order, but the duplicates in the file need to be consecutive. The usual way to achieve that is to sort the file.

Answered By: camh

Using uniq and awk:

cat File1  | uniq -c | awk '$1 > 1 { print $2 }'
Answered By: Ricardo Reyes

Execute this: perl -ne 'print if $a{$_}++' filename.txt

Answered By: Sanjay Nair

uniq requires your list to be ordered, sort defaults to alphabetical

sort path/to/your/filename | uniq -d

or

cat fileName | sort | uniq -d

Answered By: jasonleonhard
Categories: Answers Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.