Reorder of launching Systemd services

OS: Debian 11 Bullseye

Context:

  • The Zerotier application adds the zerotier-one.service system service and creates a virtual network interface (when it works).
  • The sshd server default listens to all addresses 0.0.0.0

Until then, everything is fine with me

Now I am introducing custom config in /etc/ssh/sshd_config.d/my-sshd.conf add ListenAddress 192.168.10.10 that my sshd server accepts calls only at the Zerotier interface address.

Now I suspect that sshd.service starts before zerotier-one.service because after restarting the computer:

$ sudo systemctl status sshd.service
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2023-09-14 17:21:27 CEST; 28s ago
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 524 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
    Process: 551 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255/EXCEPTION)
   Main PID: 551 (code=exited, status=255/EXCEPTION)
        CPU: 21ms

systemd[1]: Starting OpenBSD Secure Shell server...
sshd[551]: error: Bind to port 22 on 192.168.10.10 failed: Cannot assign requested address.
sshd[551]: fatal: Cannot bind any address.
systemd[1]: ssh.service: Main process exited, code=exited, status=255/EXCEPTION
systemd[1]: ssh.service: Failed with result 'exit-code'.
systemd[1]: Failed to start OpenBSD Secure Shell server

So I added the After= option to /etc/systemd/system/ssh.service.d/override.conf changing using the command sudo systemctl edit sshd.service:

[Unit]
After=network.target auditd.service

to:

[Unit]
After=network.target auditd.service network-online.target zerotier-one.service

It looks like this now:

$ sudo systemctl cat sshd.service
# /lib/systemd/system/ssh.service
[Unit]
Description=OpenBSD Secure Shell server
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target
Alias=sshd.service

# /etc/systemd/system/ssh.service.d/override.conf
[Unit]
After=network.target auditd.service network-online.target zerotier-one.service

But after restarting the computer, the error still occurs

When I do a sudo systemctl restart sshd.service now I get:

$ sudo systemctl status sshd.service
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/ssh.service.d
             └─override.conf
     Active: active (running) since Thu 2023-09-14 17:40:43 CEST; 2s ago
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 3065 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 3066 (sshd)
      Tasks: 1 (limit: 9423)
     Memory: 1.0M
        CPU: 21ms
     CGroup: /system.slice/ssh.service
             └─3066 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

systemd[1]: Starting OpenBSD Secure Shell server...
sshd[3066]: Server listening on 192.168.10.10 port 22.
systemd[1]: Started OpenBSD Secure Shell server.

I have the impression that the sshd.service is still starting before zerotier-one.service

Is something missing or can it be checked differently?

Should I do something else in addition to adding zerotier-one.service to After=?

EDIT (Information for other users):

In addition to the solution proposed by @telkoM (for which I thank you), another trick solved the problem in my case:

Just add the directive ExecStartPost=sleep 10 to zerotier-one.service or ExecStartPre=sleep 10 to sshd.service

Asked By: DarekH

||
Source

With your configuration, sshd.service will certainly start only after zerotier-one.service starts. But that is not enough. The sshd.service would need to wait until Zerotier has actually connected successfully, which can happen quite a bit later (in computer timescales, at least). And the current zerotier-one.service is not even trying to provide that information to systemd:

[Unit]
Description=ZeroTier One
After=network-online.target network.target
Wants=network-online.target

[Service]
ExecStart=/usr/sbin/zerotier-one
Restart=always
KillMode=process

[Install]
WantedBy=multi-user.target

You would probably have to create a Type=oneshot service (it could be called zerotier-wait-online.service) that would run a script that includes a loop that calls e.g. zerotier-cli listnetworks or just ip addr show and looks for the IP address 192.168.10.10. If it is not available, the script would sleep a few seconds and try again.

When the script would see the address has appeared, the script would exit – and that would tell systemd that any service configured to run After=zerotier-wait-online.service can now proceed. (Unlike the default Type=simple and several other service types, services of Type=oneshot are only considered "started" after their main ExecStart process has successfully exited – and that’s exactly what you need.

Once you have that service working, you can change your sshd.service override to After=zerotier-wait-online.service, and then it should work as you wanted.

Note that you cannot simply require that zerotier-wait-online.service runs Before=network-online.target, because zerotier-one.service itself runs After=network-online.target. Trying to set up such a requirement would create an impossible situation.

The root of the problem is that the use of ListenAddress brings with it the requirement that the specified address must already be up when sshd starts.

If you need sshd to listen in the Zerotier IP address only, but don’t specifically have to use ListenAddress to implement it, you could use alternative ways to implement the restriction.

In /etc/ssh/sshd_config, you could add a Match block like this, to deny access on any local IP address except the Zerotier one:

Match LocalAddress *,!192.168.10.10
    DenyUsers *

Or you could use iptables to drop/reject incoming connections if the destination address is anything except 192.168.10.10:

iptables -I INPUT 1 -p tcp --dport 22 ! -d 192.168.10.10/32 -j DROP

DROP makes blocked connection attempts hang until they time out; if you want the blocked connections to fail quickly, use a rule like this instead:

iptables -I INPUT 1 -p tcp --dport 22 ! -d 192.168.10.10/32 -j REJECT --reject-with tcp-reset

If you use ufw or some other firewall management system, there is probably a way to configure an equivalent rule to it.

Answered By: telcoM

Any workaround will keep breaking because this is the incorrect layer to solve the problem. It is not on systemd or zerotier.. it is sshd that needs to allow setting the IP_FREEBIND socket option to allow it to listen on addresses that are "not yet" or "ever" configured.
}

here is the patch to do it https://bugzilla.mindrot.org/attachment.cgi?id=2763 , it is correct but unfortunately openSSH developers did not accept it.

Answered By: Cristian Rodríguez