How to logrotate official nginx docker image running by podman

I started official nginx image in podman under root with command:

sudo podman run --name nginx 
 ... 
 -v /var/log/nginx:/var/log/nginx 
 docker.io/library/nginx:latest

Logging working ok, but when try to logrotate on the host machine with all configurations taken from internet it fails, for example this one:

/var/log/nginx/*.log {
 hourly
 missingok
 rotate 24
 compress
 delaycompress
 notifempty
 su root root
 create 0644
 sharedscripts
 postrotate
    podman exec nginx /bin/bash reset_log.sh
 endscript
}

reset_log.sh script contains well known command to start new log, and it works as expected if log into container and run manually:

 kill -USR1 `cat /var/run/nginx.pid`

But all this stuff does not work together as expected, logging continues to access.log.1 file or no logging at all.
Tried variants without ‘su root root’, without ‘create ..’ – no success.

Update: it seems there is a permission issue, need more investigation. manual enforced command logrotate -f -v /etc/logrotate.conf works as expected. No selinux or apparmor installed on host (arch linux)

systemd[1]: Starting Rotate log files...
conmon[4388]: conmon 32834b35446220b4e6d4 <nwarn>: runtime stderr: setns `mnt`: Operation not permitted
                                fail startup
conmon[4388]: conmon 32834b35446220b4e6d4 <error>: Failed to create container: exit status 1
logrotate[4377]: Error: crun: setns `mnt`: Operation not permitted: OCI permission denied
logrotate[4374]: error: error running shared postrotate script for '/var/log/nginx/*.log '
systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: logrotate.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Rotate log files.
Asked By: Almaz

||

The easiest solution is probably to replace your use of podman exec nginx ... with podman kill:

/var/log/nginx/*.log {
 hourly
 missingok
 rotate 24
 compress
 delaycompress
 notifempty
 su root root
 create 0644
 sharedscripts
 postrotate
    podman kill --signal USR1 nginx
 endscript
}

This works because podman kill sends the specified signal to PID 1 in the container, which we know to be nginx. We don’t need /var/run/nginx.pid, and we don’t need to enter the container at all, thus avoiding the issues causing the "OCI permission denied" error.


The underlying issue is that the logrotate service runs with a variety of protections enabled; looking at /lib/systemd/system/logrotate.service (on Rocky 9), we see:

[Service]
Type=oneshot
ExecStart=/usr/sbin/logrotate /etc/logrotate.conf

# performance options
Nice=19
IOSchedulingClass=best-effort
IOSchedulingPriority=7

# hardening options
#  details: https://www.freedesktop.org/software/systemd/man/systemd.exec.html
#  no ProtectHome for userdir logs
#  no PrivateNetwork for mail deliviery
#  no NoNewPrivileges for third party rotate scripts
#  no RestrictSUIDSGID for creating setgid directories
LockPersonality=true
MemoryDenyWriteExecute=true
PrivateDevices=true
PrivateTmp=true
ProtectClock=true
ProtectControlGroups=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectSystem=full
RestrictNamespaces=true
RestrictRealtime=true

The key restriction is the RestrictNamespaces=true option; from the documentation:

Restricts access to Linux namespace functionality for the processes of this unit…If true, access to any kind of namespacing is prohibited.

Among other things, this prevents access to alternate mount namespaces, so podman exec is unable to work. You could remove the restriction with a systemd override unit, but using podman kill is cleaner because it requires fewer changes.

Answered By: larsks
Categories: Answers Tags: , , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.