Combine two files as part of a command line parameter

I am trying to use csvgrep (part of csvkit) to extract rows matching one of two files. Unfortunately, there is no command line option that allows two matching files (like grep does). Only,

csvgrep -f list1.txt -c 2 myfile.csv

Is there a way to concatenate list1.txt with list2.txt as part of the command?

csvgrep -f (list1.txt+list2.txt) -c 2 myfile.csv

The only way I can think of doing this is to use a temporary intermediate file, but I would rather avoid that if possible.

Asked By: Robert Altman


Assuming you want the command to see a "filename" that gives the contents of both files when read, then process substitution and cat are what you want:

csvgrep -f <(cat list1.txt list2.txt) -c 2 myfile.csv

Basically, process substitution is a generalized pipeline, it runs the commands inside, and arranges for their output to be available via magic file that can be opened like any regular file. The expansion itself expands to the name of that magic file. (i.e. echo <(echo foo) prints something like /dev/fd/63, and cat <(echo foo) prints foo.)

Answered By: ilkkachu

Many programs allow - to be used as a filename to represent standard input or output (depending on whether it’s an input or output file). So you can try piping to csvgrep and using -f -.

cat list1.txt list2.txt | csvgrep -f - -c 2 myfile.csv

If it works, it has an advantage over the process substitition method because it doesn’t depend on a shell-specific extension.

If that doesn’t work, you could use /dev/stdin. This is a standard Linux device that automatically refers to the process’s standard input.

Note that all these solutions depend on csvgrep not needing to seek in the -f file, because the output of cat is being sent through a pipe. But this seems like a safe assumption.

Answered By: Barmar
Categories: Answers Tags: , , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.