merge some line start with special character

Question

merge some line start with special character

I have some log like this:

2023-11-15T08:59:28.000000+00:00 database-1 # Time: 231115  8:59:28
# User@Host: rdsadmin[rdsadmin] @ localhost []
# Thread_id: 3  Schema:   QC_hit: No
# Query_time: 0.000123  Lock_time: 0.000000  Rows_sent: 1  Rows_examined: 0
# Rows_affected: 0  Bytes_sent: 49
SET timestamp=1700038768;
SET STATEMENT max_statement_time=60 FOR SELECT 1;

Can you help me to merge some line start with # character to one line like this:

2023-11-15T08:59:28.000000+00:00 database-1 # Time: 231115  8:59:28# User@Host: rdsadmin[rdsadmin] @ localhost []# Thread_id: 3  Schema:   QC_hit: No# Query_time: 0.000123  Lock_time: 0.000000  Rows_sent: 1  Rows_examined: 0# Rows_affected: 0  Bytes_sent: 49
SET timestamp=1700038768;
SET STATEMENT max_statement_time=60 FOR SELECT 1;

I using awscli to tail and want to show default log from aws rds.

I have try with this command but not work

aws logs tail /aws/rds/instance/database-1/slowquery --log-stream-names database-1 --follow |  sed ':a;N;$!ba;s/n#/ #/g'

Asked By: dvthanh

||

Source

Answer 1

Using awk:

$ awk '
  {
    printf "%s", ((/^#/ || NR==1) ? "" : ORS )$0
  }
  END {
    if (NR) print ""
  }'

Answered By: Prabhjot Singh

Answer 2

aws logs tail /aws/rds/instance/database-1/slowquery --log-stream-names database-1 --follow | sed ':a;N;$!ba;s/n#/#/g'

Answered By: Kristtof V88

Answer 3

Using any awk:

$ awk '
    /^#/ { rec = rec OFS $0; next }
    NR>1 { print rec }
    { rec = $0 }
    END { print rec }
' file
2023-11-15T08:59:28.000000+00:00 database-1 # Time: 231115  8:59:28 # User@Host: rdsadmin[rdsadmin] @ localhost [] # Thread_id: 3  Schema:   QC_hit: No # Query_time: 0.000123  Lock_time: 0.000000  Rows_sent: 1  Rows_examined: 0 # Rows_affected: 0  Bytes_sent: 49
SET timestamp=1700038768;
SET STATEMENT max_statement_time=60 FOR SELECT 1;

With the above we build up line by line and store the multi-line record in rec and print it when the next record begins. That way if you have anything you need to do with a record before printing it, you have it in rec. If you need to retain the individual lines to do something before printing rec then just tweak the above to store a record separator between lines:

awk '
    /^#/ { rec = rec RS $0; next }
    NR>1 { prt() }
    { rec = $0 }
    END { prt() }
    function prt() { gsub(RS,OFS,rec); print rec }
' file
2023-11-15T08:59:28.000000+00:00 database-1 # Time: 231115  8:59:28 # User@Host: rdsadmin[rdsadmin] @ localhost [] # Thread_id: 3  Schema:   QC_hit: No # Query_time: 0.000123  Lock_time: 0.000000  Rows_sent: 1  Rows_examined: 0 # Rows_affected: 0  Bytes_sent: 49
SET timestamp=1700038768;
SET STATEMENT max_statement_time=60 FOR SELECT 1;

and then in the prt() function you have all of the information about the separate lines from which rec was constructed.

Answered By: Ed Morton

Answer 4

Using the ed command below, we join any line starting with a # character with the previous line:

g/^#/ -,. j

What that command does is to apply the command -,. j to each line matching the regular expression ^#. The j command joins the previous line (-) and the current line (.).

Adding ,p to print the whole buffer and Q to quit:

$ printf '%sn' 'g/^#/ -,. j' ,p Q | ed -s file
2023-11-15T08:59:28.000000+00:00 database-1 # Time: 231115  8:59:28# User@Host: rdsadmin[rdsadmin] @ localhost []# Thread_id: 3  Schema:   QC_hit: No# Query_time: 0.000123  Lock_time: 0.000000  Rows_sent: 1  Rows_examined: 0# Rows_affected: 0  Bytes_sent: 49
SET timestamp=1700038768;
SET STATEMENT max_statement_time=60 FOR SELECT 1;

Change the two commands ,p and Q to the single command wq (or to w and q separately) to write the edited buffer back to the original file and quit.

Answered By: Kusalananda