How do I delete the first n lines of an ascii file using shell commands?
I have multiple files that contain ascii text information in the first 5-10 lines, followed by well-tabulated matrix information. In a shell script, I want to remove these first few lines of text so that I can use the pure matrix information in another program. How can I use bash shell commands to do this?
If it’s any help, I’m using RedHat and an Ubuntu linux systems.
As long as the file is not a symlink or hardlink, you can use sed, tail, or awk. Example below.
$ cat t.txt
12
34
56
78
90
sed
$ sed -e '1,3d' < t.txt
78
90
You can also use sed in-place without a temp file: sed -i -e 1,3d yourfile
. This won’t echo anything, it will just modify the file in-place. If you don’t need to pipe the result to another command, this is easier.
tail
$ tail -n +4 t.txt
78
90
awk
$ awk 'NR > 3 { print }' < t.txt
78
90
If the tabulated lines are the ones that have a tab character:
grep '␉' <input_file >output_file
(␉
being a literal tab character) or equivalently
sed -n '/␉/p' <input_file >output_file
In a bash/ksh/zsh script, you can write $'t'
for a tab, e.g. grep $'t'
or sed -n $'/t/p'
.
If you want to eliminate 10 lines at the beginning of the file:
tail -n +11 <input_file >output_file
(note that it’s +11
to eliminate 10 lines, because +11
means “start from line 11” and tail numbers lines from 1) or
sed '1,10d' <input_file >output_file
On Linux, you can take advantage of GNU sed’s -i
option to modify files in place:
sed -i -n '/t/p' *.txt
Or you can use a shell loop and temporary files:
for x in *.txt; do
tail -n +11 <"$x" >"$x.tmp"
mv "$x.tmp" "$x"
done
Or if you don’t want to modify the files in place, but instead give them a different name:
for x in *.txt; do
tail -n +11 <"$x" >"${x%.txt}.data"
done
You can use Vim in Ex mode:
ex -s -c '1d5|x' file
-
1
move to first line -
5
select 5 lines -
d
delete -
x
save and close
sed -i '1,3d' file.txt
This deletes first 3 line from file.txt.
# deletes first line
echo "anb" | sed '1d'
# read list.txt and write list.csv without first line
cat list.txt | sed '1d' > list.csv
Other useful commands:
# finds first character (pipe|)
grep '^|'
# deletes pipe
sed 's/|//g'
# deletes space
sed 's/ //g'
By percentage
Using bash
, to clean up a file using a percentage number instead of an absolute number of lines:
sed -i -e 1,$( printf "$((`cat php_errors.log | wc -l` * 75 /100 ))" )d php_errors.log
Watch out because that command can be destructive since it deletes content in-place, without creating a copy.
It deletes the first 75% of lines from the mentioned file.
i created a small script which will delete everything from the /var/log/messages except last 4 lines.
# cat remove-range-of-lines.sh
#!/usr/bin/bash #print total line number line_count=`awk 'END{print NR}' /var/log/messages` #exclude last 4 lines remove_line=`expr $line_count - 4` #remove everything except last 4 lines sed -i '1,'$remove_line'd' /var/log/messages
using this script we can always keep latest entry in the /var/log/messages based on our requirement
if you want to keep last 10000 entries in the /var/log/messages below is the modified script.
#!/usr/bin/bash #print total line number line_count=`awk 'END{print NR}' /var/log/messages` #exclude last 10000 lines remove_line=`expr $line_count - 10000` #remove everything except last 10000 lines sed -i '1,'$remove_line'd' /var/log/messages