text-processing

perl: replace a string with variable containing new lines (line breaks) I want to make edits containing meta characters. Here https://stackoverflow.com/questions/29613304/is-it-possible-to-escape-regex-metacharacters-reliably-with-sed one of solutions is via perl with sQ … E (near end of accepted answer). It is mentioned it handles multi-liners, but: echo ‘a-b’ > a.txt;from=’-‘;to=’n’;perl -s -0777 -pe ‘s/Q$fromE/$to/’ — -from="$from" -to="$to" a.txt …

How to find a string in files, said string including line breaks, ", @ I know grep allows me to search for a string in files, however I did not manage to find the right method to search for the following string: "snapshot3" : { "@custom_name" : true So the string I am looking for …

Can I extract complete dates from file with grep command? I need help using grep to extract a zoned date time from a file on a Linux system. Source file is a XML with the data below: <item start="20231010073000 +0100" stop="20231010100000 +0100">…</item> And I need to extract the complete start date, but with grep I …

Delete a whole word from a CSV file that is not part of another word using SED I’ve searched for an answer to this and have come close, but not close enough. We receive a CSV file that contains the text "NULL" which needs to be replaced by nothing. Example: Input 12345,George,MCNULLMAN,NULL,green,NULL Replacement should result …

How to list process with their depth? I have a slightly different process tree question. I need to list the process and their depth in scalar value. pstree gives the result in a "graphic" manner. root 100930 0.0 0.3 839396 49020 ? Ssl Aug15 38:20 /usr/sbin/rsyslogd -n root 847414 0.0 0.0 76640 7788 ? Ss …

Remove duplicates by adding numerical suffix How do I append a numerical suffix to lines to remove duplicates? Pseudo code: if currLine.startsWith("tag:") x = numFutureLinesMatching(currLine) if (x > 0) currLine = currLine + ${x:01} Input file tag:20230901-FAT val:1034 tag:20230901-FAT val:1500 tag:20230901-LAX val:8934 tag:20230901-SMF val:2954 tag:20230901-LAX val:1000 tag:20230901-FAT val:1500 Desired output tag:20230901-FAT-02 val:1034 tag:20230901-FAT-01 val:1500 tag:20230901-LAX-01 …

Making single digit numbers between 1 and 9 into double digits, inside CSV file I have a CSV file with thousands of lines like these 1664;4;5;35;37;43;5;6 1663;21;23;32;40;49;8;11 1662;16;17;34;35;44;5;10 1661;2;9;23;32;40;6;7 1660;23;25;30;44;47;9;12 1659;3;5;9;32;43;6;10 1658;4;6;10;13;34;3;5 1657;8;9;33;35;40;3;6 1656;15;20;31;44;48;1;3 1655;25;27;35;40;45;7;11 1654;7;32;33;34;38;6;9 1653;5;7;11;27;37;6;12 1652;7;31;33;35;36;7;10 1651;4;12;34;35;45;1;9 1650;5;8;29;35;48;5;6 1649;2;11;28;42;48;4;9 1648;2;11;12;19;38;4;8 You can see that the numbers between 1 and 9 are single digit. …

Use find, cat, etc… to generate an M3U playlist from album playlists I have a music folder with selected playlists in each folder titled "playlist.m3u" (should also match *.m3u8). What I would like to do is have a shell script generate a playlist concatenated with all these selected tracks. But in order for the playlist …

Multiply found text in sed – is it possible? This is what I have: echo -e "at4nbt7nct2ndt12n" | sed -rn ‘s/(.*)t([0-9]*)$/2t111t/p’ 4 aaa 7 bbb 2 ccc 12 ddd This I want: 4 aaaa 7 bbbbbbb 2 cc 12 dddddddddddd Is this possible in sed only or I have to use some other tools? Asked …

Why does 'tr' with '-c' option and set 2 extension add an unwarranted character to the end? I wanted to use tr to substitute "illegal" characters in a string with a replacement character, where "illegal" characters are all outside of a set of "allowed" characters (i.e. they are the complement of the set of allowed …

Is it now safe to parse the output of GNU ls? The accepted wisdom for the past few decades has been that it is never a good idea to parse the output of ls ([1],[2]). For example, if I want to save a file’s modification date along with its name into a shell variable, this …

How to know a JSON array is an empty array in bash? I use this code to consume GitHub API and automate some tasks: curl –silent -H ‘Authorization: token github_access_token’ ‘https://api.github.com/orgs/OrganizationName/repos?per_page=100’ Sometimes I get this as the response: [ ] I want to know if the response is an empty array or not. I thought …

Merge successive lines into one line In a text file, if the first 25 characters in a line are a space, how can I append that line to the previous line until another line comes along that starts with an ASCII character in column one. Since it can’t be displayed like that here, I’ve added …

Need help with a sed script for specific word replacements in a text file I’m trying to automate a specific text replacement task for Romanian text using sed or any other suitable Unix tool. Here are my requirements: Replace all instances of the letter â with î inside words. Words that have in their composition …

Using the command line, multiply all entries within a column to a fixed number, but only if the line is not a comment I am trying to: Always add 0 0 0 to the first row of the file. Multiply 2*pi or 6.2832 to the first column of a three-column file formatted similar below, but …

Process CSV file through command line: Remove only middle rows between consecutive row entries if consecutive entries have the same 2nd column value I have a CSV file with only 2 columns (but lots of rows) and the occasional irregular row which always starts with an asterisk (*) character and may span more than two …

Can some command or script subtract lines in one file from another faster than grep? I have a shell script that runs regularly and the following part of it causes a slowdown. grep -v -f RemoveTheseGoodIPs.txt FromTheseShadyIPs.txt > RemainingBadIPs.txt It works. It just takes 156 seconds to give me an output. I’m hoping to figure …

Get value after a keyword that is in the middle of the line only from given line Hey so I have the following line in the middle of a file and I only need to get the value following "energy=". The line number is stored in a variable called "lineNumber". There are other lines in …

Replace the first column of a text file to be 6 minus the current value I have the following file, space separated 2.5 235215 3.1925 3.7 125255 1.3125 I need to change the first column to be 6 minus the current value. The desired result is 3.5 235215 3.1925 2.3 125255 1.3125 How to do …

Transforming a filename with 'tr' using RegEx doesn't work I want to use the tr command to rename something like filename.ext to someName.ext. To do that I’ve tried echo "filename.ext" | tr -c ".a-z" "someName"` to replace the complement (-c) of the filename extension (specified as period, followed by lower-case letters) with the new basename. …