Strip filesize from rclone file list

I use rclone and I want to have the file list without the size of the files mention. I can’t find how to do this in rclone so I thought of stripping it with awk or something like that. My output looks like this

 59183070 fileserver/transfer_kimberly_2022-12-18_0558 (1).zip
     3690 fileserver/transfer_kimberly_2022-12-18_0558 (1).zip - Shortcut.lnk
 35961190 fileserver/transfer_2023-06-27_0814.zip
  7803667 fileserver/woodproject.zip
7437905920 Them/Data/Before_20230526132130642.FDB
1064525824 Them/Data/Updating_20220705231152059.FDB
1064525824 Them backup/Data/Updating_20220706231124156.FDB
1064525824 Them backup/Data/Updating_20220705231152059.FDB
1064525824 Them backup/Data/Updating_20220706231124156.FDB
  7004362 test.zip
  7004362 test (1).zip
  7803667 37939 37/Data/Updating_20220706231124156.FDB
  7803667 37939/Data/Updating_20220706231124156.FDB

The first number left is the filesize. I want this output

fileserver/transfer_kimberly_2022-12-18_0558 (1).zip
fileserver/transfer_kimberly_2022-12-18_0558 (1).zip - Shortcut.lnk
fileserver/transfer_2023-06-27_0814.zip
fileserver/woodproject.zip
Them/Data/Before_20230526132130642.FDB
Them/Data/Updating_20220705231152059.FDB
Them backup/Data/Updating_20220706231124156.FDB
Them backup/Data/Updating_20220705231152059.FDB
Them backup/Data/Updating_20220706231124156.FDB
test.zip
test (1).zip
37939 37/Data/Updating_20220706231124156.FDB
37939/Data/Updating_20220706231124156.FDB

I thought about stripping everything left of the last space that comes before the first /. But the space in the first directory or file names makes it complicated. Help is appreciated

Asked By: unixcandles

||

It looks like your data is line based. Furthermore it looks like the pattern is "zero or more spaces, one or more digits, one space". One tool to edit lines like this is sed:

sed -E 's/^[[:space:]]*[[:digit:]]+[[:space:]]//'

I’ve used character classes ([[...]]) but you could also use literal character ranges:

sed -E 's/^ *[0-9]+ //'

You can use sed in a pipe or have it process a file:

cmd_a | sed... | cmd_b
sed... {file}
Answered By: Chris Davies

You may use the lsf sub-command of rclone to list files and directories without listing their file sizes. Since lsf does not recurse by default, you may want to use lsf -R if you need to list directories recursively.

If you want only the paths of the files (no directories), use lsf -R —-files-only. You may also tweak the output format using --format, and several options exist for more advanced data filtering (see rclone lsf —-help).


For easy parsing by further downstream utilities, I would suggest using the JSON output of the rclone sub-command lsjson.

If using lsjson rather than ls or lsf, then use jq (a well-known JSON processing tool) to pull out the Path data from each element of the returned JSON array:

rclone lsjson -R someremote:somepath | jq -r '.[].Path'

Note that lsjson does not recurse by default, so I use it with its -R option here.

If you want to avoid listing directories, then select only array entries whose IsDir entry is false:

rclone lsjson -R someremote:somepath | jq -r 'map(select(.IsDir|not))[].Path'

(… or use lsjson -R --files-only, as with lsf.)


For reference, lsjson will return a JSON array of objects that look like the equivalent of this example (using your data as a mock-up):

{
  "Path": "Them/Data/Before_20230526132130642.FDB",
  "Name": "Before_20230526132130642.FDB",
  "Size": 7437905920,
  "MimeType": "application/octet-stream",
  "ModTime": "2023-09-04T20:10:54.367101898+02:00",
  "IsDir": false
}
Answered By: Kusalananda

Using Perl

~$ perl -pe 's/^s*d+s//;'  file

Using Raku (formerly known as Perl_6)

~$ raku -pe 's/^ s* d+ s //;'  file

Above are answers written in Perl and Raku. The Perl/Raku -pe flags denote execution linewise with autoprinting (-e for code alone, -p for linewise-autoprinting, i.e. the code gets applied to each line of input and no explicit print or put is necessary). Note in Raku d includes both ASCII-plus-Unicode digits. If you want ASCII-only in Raku use <+[0..9]>.


Sample Input:

59183070 fileserver/transfer_kimberly_2022-12-18_0558 (1).zip
     3690 fileserver/transfer_kimberly_2022-12-18_0558 (1).zip - Shortcut.lnk
 35961190 fileserver/transfer_2023-06-27_0814.zip
  7803667 fileserver/woodproject.zip
7437905920 Them/Data/Before_20230526132130642.FDB
1064525824 Them/Data/Updating_20220705231152059.FDB
1064525824 Them backup/Data/Updating_20220706231124156.FDB
1064525824 Them backup/Data/Updating_20220705231152059.FDB
1064525824 Them backup/Data/Updating_20220706231124156.FDB
  7004362 test.zip
  7004362 test (1).zip
  7803667 37939 37/Data/Updating_20220706231124156.FDB
  7803667 37939/Data/Updating_20220706231124156.FDB

Sample Output:

fileserver/transfer_kimberly_2022-12-18_0558 (1).zip
fileserver/transfer_kimberly_2022-12-18_0558 (1).zip - Shortcut.lnk
fileserver/transfer_2023-06-27_0814.zip
fileserver/woodproject.zip
Them/Data/Before_20230526132130642.FDB
Them/Data/Updating_20220705231152059.FDB
Them backup/Data/Updating_20220706231124156.FDB
Them backup/Data/Updating_20220705231152059.FDB
Them backup/Data/Updating_20220706231124156.FDB
test.zip
test (1).zip
37939 37/Data/Updating_20220706231124156.FDB
37939/Data/Updating_20220706231124156.FDB

https://perldoc.perl.org/perlre
https://docs.raku.org/language/regexes

Answered By: jubilatious1
Categories: Answers Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.