Why is one process ps command showing spaces around a directory instead of slashes?

I’m trying to debug a failing legacy process monitor that searches processes that are running. I determined it is being caused by some mysterious linux behavior. The process being ran is /opt/my/path/directoryname/daemonname but for some odd reason, it shows up with spaces, not /, around directoryname. Many daemons are running from the same directory and show up with the correct path fine. It’s just this one particular daemon on every server.

ps -ef shows the following for this daemon:

/opt/my/path directoryname daemonname --arg1 VALUE1 --arg-two VALUE.TWO --arg-three ARG.UMENT.THREE.ERR --worker daemonname --pid-role daemonname

This daemon is being launched by a perl script. Comparing it to others, it looks like the directory path is being provided using the exact same env variable as all the other scripts, but those other scripts result in correct path like /opt/my/path/directoryname/daemonname. The relevant perl lines look something like exec => catfile( $settings->config("/directories/executables"), $daemon_name ), and I’ve confirmed the value of config("/directories/executables") is the correct value /opt/my/path/directoryname

Searching for this question has been a nightmare due to all of the questions about directory names having spaces. This is not that. None of the path should contain a space. ps shows my mkdir command with a space where I had a slash is the closest I could find, which was unanswered.

Asked By: k.schroeder31

||

Among the fields returned by ps -f is the argument list passed to the command execution that the process (or one of its ancestor) last made. You can get just that with ps -o args.

When a process executes a file, that’s with the execve() system call:

execve("/path/to/executable", [arg0, arg1..., 0], [env1, env2... 0])

arg0 by convention is the name of the executable. When a shell interprets a:

cmd arg

command line, it calls execve("/path/to/cmd", ["cmd", "arg", 0], ...). When that’s:

/path/to/cmd arg

It calls execve("/path/to/cmd", ["/path/to/cmd", "arg", 0], ...).

Those strings (cmd, arg) end up at the bottom of the stack of the process, NUL-delimited. Inside the process, that’s what argv[0], argv[1]… point to.

On Linux, that area is exposed in /proc/<pid>/cmdline, and that’s where ps gets it from for the args field.

Technically, ps takes that list of strings and prints it joined with spaces so that in the end the args fields looks like the shell command line that would be used to execute the command.

Here, if ps shows:

/opt/my/path directoryname daemonname --arg1...

Those spaces may indicate either a real space in those strings, or separate strings or the fact that spaces have been replaced by NULs in those strings.

The latter is plausible if the process happened to use the standard strtok() function or a similar approach on its argv[0] to split it on /s possibly to find out the base name of the executable it was run as or the different directory components of its path.

For instance, if I create:

#include <string.h>
#include <unistd.h>
int main(int argc, char *argv[])
{
  strtok(argv[0], "/");
  while (strtok(NULL, "/"));
  pause();
}

And compile and run it:

$ cc a.c
$ "$PWD/a.out" &
[1] 67758
$ ps -fp "$!"
UID          PID    PPID  C STIME TTY          TIME CMD
chazelas   67758   60043  0 19:19 pts/4    00:00:00 /home chazelas a.out

You’ll see the /s have been replaced with spaces.

Actually, they have been replaced with NULs, as can be verified with:

$ sed -n l "/proc/$!/cmdline"
/home00chazelas00a.out00$

But ps saw those NULs and interpreted them as the delimiters for 3 different arguments which it joined with spaces.

As you later noted yourself in comment, dirname() is another standard function that may replace /s with 0 in its argument.

$ cat b.c
#include <libgen.h>
#include <unistd.h>
int main(int argc, char *argv[]) {dirname(dirname(argv[0])); pause();}
$ cc b.c
$ "$PWD/a.out" &
[1] 42128
$ ps -fp "$!"
UID          PID    PPID  C STIME TTY          TIME CMD
chazelas   42128   39744  0 20:18 pts/6    00:00:00 /home chazelas a.out

Then assuming that’s the correct explanation, it’s not a weird Linux behaviour it’s just the application itself modifying its argv[0] after it has started (probably by replacing /s with NULs within, probably to extract some path components).

On Linux, a more reliable way to find which executable a process is currently running is to do a readlink() on /proc/<pid>/exe, for instance with the realpath or readlink utility, or the stat builtin of zsh:

$ zmodload zsh/stat
$ stat +link /proc/$!/exe
/home/chazelas/a.out
$ readlink "/proc/$!/exe"
/home/chazelas/a.out
$ realpath "/proc/$!/exe"
/home/chazelas/a.out

And the procps implementation of ps on Linux returns it with ps -o exe.

There is no foolproof way to know for sure what arguments were passed to the command as what’s shown in /proc/<pid>/cmdline can be modified by the process.

Audit logs may have the information though.

Answered By: Stéphane Chazelas
Categories: Answers Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.