Process the same stdin two time and append the outputs

Question

Process the same stdin two time and append the outputs

I have a json file that looks like:

[
    {
        "key": "alt+down",
        "command": "-editor.action.moveLinesDownAction",
        "when": "editorTextFocus && !editorReadonly"
    },
    {
        "key": "alt+f12",
        "command": "editor.action.peekDefinition",
        "when": "editorHasDefinitionProvider && editorTextFocus && !inReferenceSearchEditor && !isInEmbeddedEditor"
    }
]
//  {
//      "key": "ctrl+shift+d",
//      "command": "workbench.action.toggleMaximizedPanel"
//  },
//  {
//      "key": "ctrl+shift+d",
//      "command": "-workbench.view.debug",
//      "when": "viewContainer.workbench.view.debug.enabled"
//  }

I want to sort this file.

jq give error if there is // at the beginning of line as this is not a valid json.

So to sort this file, the command I came up with is:

grep -v '^[ ]*//' keybindings.json | jq 'sort_by(.key)'

But I do not want to discard the commented lines. So, to get the commented lines, the command I came up with is:

grep '^[ ]*//' keybindings.json

Now to solve my problem, what I can simply do is:

#!/bin/bash

SORTED_JSON=$(grep -v '^[ ]*//' keybindings.json | jq 'sort_by(.key)')
COMMENTED_JSON=$(grep '^[ ]*//' keybindings.json)

echo "$SORTED_JSON" >keybindings.json
echo "$COMMENTED_JSON" >>keybindings.json

But there is a catch.

I have to do this in one command. This is because, I am doing this via a vscode settings.

"filterText.commandList": [
    {
        "name": "Sort VSCode Keybindings",
        "description": "Sorts keybindings.json by keys. Select everything except the comment in fist line. Then run this command",
        "command": "jq 'sort_by(.key)'"
    }
]

The command take the selected text as stdin, process it, then output the processed text.

So, as far i understand, i have to read the stdin two times (once with grep -v '^[ ]*//' | jq 'sort_by(.key)' and the second time with grep '^[ ]*//'). And append the two command output in stdout.

How can I solve this problem?

Update 1:

I have tried both

cat keybindings.json| {grep -v '^[ ]*//' | jq 'sort_by(.key)' ; grep '^[ ]*//'}

and

cat keybindings.json| (grep -v '^[ ]*//' | jq 'sort_by(.key)' ; grep '^[ ]*//')

These does not show the commented lines.

Update 2:

The following seems to be close to what I was expecting. But here commented lines come before the uncommented lines.

$ cat keybindings.json| tee >(grep -v '^[ ]*//' | jq 'sort_by(.key)') >(grep '^[ ]*//') > /dev/null 2>&1
    //  {
    //      "key": "ctrl+shift+d",
    //      "command": "workbench.action.toggleMaximizedPanel"
    //  },
    //  {
    //      "key": "ctrl+shift+d",
    //      "command": "-workbench.view.debug",
    //      "when": "viewContainer.workbench.view.debug.enabled"
    //  }
[
  {
    "key": "alt+down",
    "command": "-editor.action.moveLinesDownAction",
    "when": "editorTextFocus && !editorReadonly"
  },
  {
    "key": "alt+f12",
    "command": "editor.action.peekDefinition",
    "when": "editorHasDefinitionProvider && editorTextFocus && !inReferenceSearchEditor && !isInEmbeddedEditor"
  }
]

Update 3:

cat keybindings.json| (tee >(grep '^[ ]*//'); tee >(grep -v '^[ ]*//' | jq 'sort_by(.key)'))

or,

cat keybindings.json| {tee >(grep '^[ ]*//'); tee >(grep -v '^[ ]*//' | jq 'sort_by(.key)')}

also seems to give the same output as Update 3 (commented lines come before the uncommented lines).

Asked By: Ahmad Ismail

||

Source

Answer 1

I know of no way to interpolate mixed comment lines and non-comment lines; you have to treat them as separate blocks and process them separately.

If you didn’t mind the commented lines being output first you could use awk like this:

awk '{ if ($0 ~ /^ *///) { print } else { print | "jq "sort_by(.key)"" } }' keybindings.json

But since you want the comment lines to come at the end you need to store the comment lines and output them later:

awk '
    # Define a convenience variable for the jq process
    BEGIN {
        jq = "jq "sort_by(.key)""
    }

    # Each line hits this. Either we save the comment or we feed it to jq
    {
        if ($0 ~ /^ *///) { c[++i] = $0 }
        else { print | jq }
    }
    
    # Close the pipe to get its output and then print the saved comment lines
    END {
        close (jq);
        for (i in c) { print c[i] }
    }
' keybindings.json

Now, regarding your "I have to do this in one command". Remember that there is nothing stopping you creating your own commands (programs, scripts). Put the necessary set of commands into a file, make the file executable, and put it into a directory that’s in your $PATH. I have always used $HOME/bin and I have the equivalent of export PATH="$PATH:$HOME/bin" in my ~/.bash_profile and ~/.profile.

Answered By: roaima

Answer 2

Turn your comments into strings, sort your arrays, and then turn the comment strings into comments again.

Turning the comments into strings is done by

Detect lines starting with // (with optional indentation with spaces).
Replace any existing " on those lines with ", and
Double quote the line.

sed ':^ *//: { s/"/\"/g; s/.*/"&"/; }'

Then, sort the arrays:

jq 'if type == "array" then sort_by(.key) else . end'

At the end, turn back the comment strings into comments:

Detect lines starting with "// (with optional spaces between " and //) and ending with ".
Remove the first " and the " at the end of those lines.
Replace each " by ".

sed ':^"( *//.*)"$: { s//1/; s/\"/"/g; }'

The pipeline in full reading from standard input:

sed ':^ *//: { s/"/\"/g; s/.*/"&"/; }' |
jq 'if type == "array" then sort_by(.key) else . end' |
sed ':^"( *//.*)"$: { s//1/; s/\"/"/g; }'

This assumes that your data does not already contain escaped double-quotes.

Answered By: Kusalananda

Answer 3

Since I have done some research, would like to answer my question as well. For single line command:

cat keybindings.json | (tee /tmp/program-code-binding.json | grep -v '^[ ]*//' | jq 'sort_by(.key)'; cat /tmp/program-code-binding.json | grep '^[ ]*//')

If you want to use script then:

#!/bin/bash

THESTDIN=$(cat)

SORTED_JSON=$(echo "$THESTDIN" | grep -v '^[ ]*//' | jq 'sort_by(.key)')
COMMENTED_JSON=$(echo "$THESTDIN" | grep '^[ ]*//')

echo "$SORTED_JSON"
echo "$COMMENTED_JSON"

There is an edge case for the script. THESTDIN=$(cat) will hang indefinitely if there is nothing in the pipe. To deal with this issue, the script will actually look like:

#!/bin/bash

__=""
THESTDIN=""

read -N1 -t1 __  && {
    (( $? <= 128 ))  && {
        IFS= read -rd '' _stdin
        THESTDIN="$__$_stdin"
    }
}

SORTED_JSON=$(echo "$THESTDIN" | grep -v '^[ ]*//' | jq 'sort_by(.key)')
COMMENTED_JSON=$(echo "$THESTDIN" | grep '^[ ]*//')

echo "$SORTED_JSON"
echo "$COMMENTED_JSON"

Answered By: Ahmad Ismail