Why is one process ps command showing spaces around a directory instead of slashes?
I’m trying to debug a failing legacy process monitor that searches processes that are running. I determined it is being caused by some mysterious linux behavior. The process being ran is /opt/my/path/directoryname/daemonname but for some odd reason, it shows up with spaces, not /, around directoryname. Many daemons are running from the same directory and show up with the correct path fine. It’s just this one particular daemon on every server.
ps -ef
shows the following for this daemon:
/opt/my/path directoryname daemonname --arg1 VALUE1 --arg-two VALUE.TWO --arg-three ARG.UMENT.THREE.ERR --worker daemonname --pid-role daemonname
This daemon is being launched by a perl script. Comparing it to others, it looks like the directory path is being provided using the exact same env variable as all the other scripts, but those other scripts result in correct path like /opt/my/path/directoryname/daemonname
. The relevant perl lines look something like exec => catfile( $settings->config("/directories/executables"), $daemon_name ),
and I’ve confirmed the value of config("/directories/executables") is the correct value /opt/my/path/directoryname
Searching for this question has been a nightmare due to all of the questions about directory names having spaces. This is not that. None of the path should contain a space. ps shows my mkdir command with a space where I had a slash is the closest I could find, which was unanswered.
Among the fields returned by ps -f
is the argument list passed to the command execution that the process (or one of its ancestor) last made. You can get just that with ps -o args
.
When a process executes a file, that’s with the execve()
system call:
execve("/path/to/executable", [arg0, arg1..., 0], [env1, env2... 0])
arg0
by convention is the name of the executable. When a shell interprets a:
cmd arg
command line, it calls execve("/path/to/cmd", ["cmd", "arg", 0], ...)
. When that’s:
/path/to/cmd arg
It calls execve("/path/to/cmd", ["/path/to/cmd", "arg", 0], ...)
.
Those strings (cmd
, arg
) end up at the bottom of the stack of the process, NUL-delimited. Inside the process, that’s what argv[0]
, argv[1]
… point to.
On Linux, that area is exposed in /proc/<pid>/cmdline
, and that’s where ps
gets it from for the args
field.
Technically, ps
takes that list of strings and prints it joined with spaces so that in the end the args
fields looks like the shell command line that would be used to execute the command.
Here, if ps
shows:
/opt/my/path directoryname daemonname --arg1...
Those spaces may indicate either a real space in those strings, or separate strings or the fact that spaces have been replaced by NULs in those strings.
The latter is plausible if the process happened to use the standard strtok()
function or a similar approach on its argv[0]
to split it on /
s possibly to find out the base name of the executable it was run as or the different directory components of its path.
For instance, if I create:
#include <string.h>
#include <unistd.h>
int main(int argc, char *argv[])
{
strtok(argv[0], "/");
while (strtok(NULL, "/"));
pause();
}
And compile and run it:
$ cc a.c
$ "$PWD/a.out" &
[1] 67758
$ ps -fp "$!"
UID PID PPID C STIME TTY TIME CMD
chazelas 67758 60043 0 19:19 pts/4 00:00:00 /home chazelas a.out
You’ll see the /
s have been replaced with spaces.
Actually, they have been replaced with NULs, as can be verified with:
$ sed -n l "/proc/$!/cmdline"
/home 00chazelas 00a.out 00$
But ps
saw those NULs and interpreted them as the delimiters for 3 different arguments which it joined with spaces.
As you later noted yourself in comment, dirname()
is another standard function that may replace /
s with 0 in its argument.
$ cat b.c
#include <libgen.h>
#include <unistd.h>
int main(int argc, char *argv[]) {dirname(dirname(argv[0])); pause();}
$ cc b.c
$ "$PWD/a.out" &
[1] 42128
$ ps -fp "$!"
UID PID PPID C STIME TTY TIME CMD
chazelas 42128 39744 0 20:18 pts/6 00:00:00 /home chazelas a.out
Then assuming that’s the correct explanation, it’s not a weird Linux behaviour it’s just the application itself modifying its argv[0]
after it has started (probably by replacing /
s with NULs within, probably to extract some path components).
On Linux, a more reliable way to find which executable a process is currently running is to do a readlink()
on /proc/<pid>/exe
, for instance with the realpath
or readlink
utility, or the stat
builtin of zsh
:
$ zmodload zsh/stat
$ stat +link /proc/$!/exe
/home/chazelas/a.out
$ readlink "/proc/$!/exe"
/home/chazelas/a.out
$ realpath "/proc/$!/exe"
/home/chazelas/a.out
And the procps implementation of ps
on Linux returns it with ps -o exe
.
There is no foolproof way to know for sure what arguments were passed to the command as what’s shown in /proc/<pid>/cmdline
can be modified by the process.
Audit logs may have the information though.