Delete line if next line is the same

What sed/awk command can I use? Just sort -u will remove all instances

Input:

abc
abc
def
abc
abc
def

Expected output:

abc
def
abc
def
Asked By: 干猕猴桃

||

That’s what the uniq standard command is for.

uniq your-file

Note that some uniq implementations like GNU uniq will give you the first of a sequence of lines that sort the same (where strcoll() returns 0) as opposed to are byte-to-byte identical (where memcmp() or strcmp() returns 0). To force a byte to byte comparison regardless of the uniq implementation, you can force the locale to C with:

LC_ALL=C uniq your-file
Answered By: Stéphane Chazelas

Vim can achieve this nicely:

:g/v^(.*n)1/d

Or if you’d rather use vim as a command line tool, you could do this as

vim file -c "g/v^(.*n)1/d" -c "wq"

This way you don’t have to wrestle with exiting vim later 😉

Explanation:

:g/

On all lines that match this regex…

v^(.*n)1

Any line followed by itself…

/d

run the delete command (delete the current line). The -c "wq" is to save the changes and exit.

Answered By: DJMcMayhem

Here is an awk equivalent:

awk 'p!=$0; {p=$0}' file

p!=$0; print line if it is not identical to previous one, ; is necessary here.
{p=$0} set p to current line.

An obvious bug is missing the first line if it’s empty. A simple fix is:

awk 'NR==1 || p!=$0; {p=$0}' file
Answered By: thanasisp
Categories: Answers Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.