Redirecting output depending on grep regex

I am using gradle run to start a REST server. The output of the REST server looks like this:

XXX.XXX.XX.XXX - <moreinfo>
XXX.XXX.XX.XXX - <moreinfo>
XXX.XXX.XX.XXX - <moreinfo>
XXX.XXX.XX.XXX - <moreinfo>

XXX.XXX.XX.XXX here is an IP address, randomtext are error messages. All output is directed to stdout, sadly.

How can I direct all lines starting with an IP address to a file called err.log and every other line to all.log?

Unfortunately, gradle run can only be started once and doesn’t stop, since it is a REST server.

Maybe use a tee , grep combination?

Asked By: polym


In Bash, you can use process substitution with tee:

tee >(grep XXX > err.log) | grep -v XXX > all.log

This will put all lines matching XXX into err.log, and all lines into all.log. >( ... ) creates the process in the parentheses and connects its standard output to a pipe. This works in zsh and other modern shells too.

You can also use the pee command from moreutils:

pee "grep XXX > err.log" "grep -v XXX > all.log"

pee redirects standard input to multiple commands (“tee for pipes”).

A further alternative is with awk:

awk '{ if (/^([0-9]{1,3}.){3}[0-9]{1,3}/) { print > "err.log" } else { print > "all.log" } }'

That just tests every line against the expression and writes the whole thing into err.log if it matches and all.log if it doesn’t.

The awk regular expression is suitable for grep -E too (although it does match some bad addresses — 999.0.0.0 and so on — but that probably isn’t a problem).

Answered By: Michael Homer

So, it looks like gradle run doesn’t comply with tee, pee, grep and io-redirection. It always stops reading after 4096 bytes.

To circumvent this issue, I read each line of gradle run. I didn’t test it yet, but I guess that reading a line that is over 4k characters long will also fail.

Anyway, here is the code to solve my question specifically:

while read -r line; do
    [[ $line =~ ^[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.* ]] && printf '%sn' "$line" >> "$STDERRLOG" && continue
    printf '%sn' "$line" >> "$STDOUTLOG"
done < <(gradle run)
Answered By: polym
Categories: Answers Tags: , , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.