Remote SMTP traffic appears to come from LXC Host to container

Summary: I have a mail server (exim 4, Debian 10) in an LXC container. The host is running Debian 11. Since yesterday evening spam traffic has been coming in that appears to come from the LXC Host. However, tcpdump logs show that it is actually remote traffic. What is going on?

This is an example of an exim4 log entry on the mail server, for a spam mail seemingly coming from the lxc host:

2023-07-23 11:15:51 1qNX42-009wSW-VR <= spammers@emailaddress.com H=LXCHOST (prvzvtrfnh) [LXCHOSTIPV4] P=esmtp S=615 id=64BCA975.1317B1FC@gmail.com

Yet on the tcpdump logs on the host I see corresponding entries like this:

14:06:07.165374 IP 39.170.36.149.34307 > MAILSERVERCONTAINER.smtp: Flags [P.], seq 5672:5702, ack 1397, win 27, options [nop,nop,TS val 1151815058 ecr 475541370], length 30: SMTP: MAIL FROM:<spammers@emailaddress.com>

So the traffic appears to come from the (Chinese) IP 39.170.36.149. (This IP does not appear at all in the container logs.) So why does this traffic appear as coming from the host to the mail server?

The relevant network interfaces on the host are:

  • eno1, the physical interface
  • br0, a bridge connecting the phyiscal interface with several lxc containers

The tcpdump command on the host that shows the spammy traffic is:

tcpdump -i br0 port 25 and dst host [MAILSERVERIPV4]

The bridge interface is setup like this in /etc/network/interfaces:

auto br0                        
iface br0 inet static            
        bridge_ports regex eth.* regex eno.*
        bridge_fd 0          
        address HOSTADDRES
        netmask 255.255.255.192 
        gateway HOSTGATEWAY

Both container and host are up to date with security updates. But the host’s uptime is 248 days, so it is possible that it is running outdated binaries.

UPDATE
I think the problem was caused by an iptables rule on the host, -t nat -A POSTROUTING -o br0 -j MASQUERADE. This rule is intended for containers without an external IP to reach the internet. I have apparently misunderstood what this does. Shouldn’t it only masquerade traffic that is routed from internal IPs to the internet? As I understand it, external traffic to the mail server is bridged and not routed at all. Also, it’s only one particular spammer that was able to exploit my setup. The normal traffic to my mail server shows external IPs. How did the spammer do this?

UPDATE 2: The problems started after installing docker on the host. Could it be that docker and lxc interact in a way to create these problems?

Asked By: Lennart

||

I think the problem was caused by an iptables rule on the host

iptables -t nat -A POSTROUTING -o br0 -j MASQUERADE

This rule is intended for containers without an external IP to reach the internet.

What this rule does is masquerade any traffic going out through br0. It could be traffic going out from the host to a container, or it could (as intended) be traffic leaving the host and heading off to the wider Internet.

The problems started after installing docker on the host. Could it be that docker and lxc interact in a way to create these problems?

Yes, I would say that’s quite likely. You will need to modify the rule to avoid masquerading local traffic.

As an example, let’s assume your host is 192.168.1.1 (and maybe also has a public IPv4 address), and you have a hidden container subnet of 192.168.1.0/24. Docker has come along and grabbed 172.17.0.0/16.

We might suppose that this rule is intended to masquerade anything leaving the Docker subnet,

iptables -t nat -A POSTROUTING -o br0 --src 172.17.0.0/24 -j MASQUERADE
Answered By: roaima
Categories: Answers Tags: , , , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.