Reply on same interface as incoming?
I have a system with two interfaces. Both interfaces are connected to the internet. One of them is set as the default route; a side effect of this is that if a packet comes in on the non-default-route interface, the reply is sent back through the default route interface. Is there a way to use iptables (or something else) to track the connection and send the reply back through the interface it came from?
The following commands create an alternate routing table via eth1 for packets that have the mark 1 (except packets to localhost). The ip command is from the iproute2 suite (Ubuntu: iproute Install iproute http://bit.ly/software-small, iproute-doc Install iproute-doc http://bit.ly/software-small).
ip rule add fwmark 1 table 1
ip route add 127.0.0.0/0 table 1 dev lo
ip route add 0.0.0.0/0 table 1 dev eth1
The other half of the job is recognizing packets that must get the mark 1; then use iptables -t mangle -A OUTPUT … -j MARK --set-mark 1 on these packets to have them routed through routing table 1. I think the following should do it (replace 1.2.3.4 by the address of the non-default-route interface):
iptables -t mangle -A OUTPUT -m conntrack --ctorigdst 1.2.3.4 -j MARK --set-mark 1
I’m not sure if that’s enough, maybe another rule is needed on the incoming packets to tell the conntrack module to track them.
echo 200 isp2 >> /etc/iproute2/rt_tables
ip rule add from <interface_IP> table isp2 prio 1
ip route add default via <gateway_IP> dev <interface> table isp2
The above doesn’t require any packet marking with ipfilter. It works because the outgoing (reply) packets will have the IP address that was originally used to connect to the 2nd interface as the source (from) address on the outgoing packet.
I had issues with the locally generated packets with the solution suggested by Peter, I’ve found that the following corrects that:
echo 200 isp2 >> /etc/iproute2/rt_tables
ip rule add from <interface_IP> table isp2 priority 900
ip rule add from dev <interface> table isp2 priority 1000
ip route add default via <gateway_IP> dev <interface> table isp2
ip route add <interface_prefix> dev <interface> proto static scope link src <interface_IP> table isp2
NOTE: You may run into syntax issues with the 4th line above. In such cases the syntax for the 4th command may be this now:
ip rule add iif <interface> table isp2 priority 1000
I’m assuming you are running Linux and, further, that you are utilising a RedHat/CentOS-based distribution. Other Unix’s and distributions will require similar steps – but the details will be different.
Start by testing (note that this is very similar to @Peter’s answer. I am assuming the following:
- eno0 is isp0 and has the overall default gateway
- eno1 is isp1 and has the IP/range 192.168.1.2/24 with gateway 192.168.1.1
The commands are as follows:
$ echo 200 isp1 >> /etc/iproute2/rt_tables
$ ip rule add from eno1 table isp1
$ ip route add default via 192.168.1.1 dev eno1 table isp1
The firewall is not involved in any way. Reply packets would always have been sent from the correct IP – but previously were being sent out via the wrong interface. Now these packets from the correct IP will be sent via the correct interface.
Assuming the above worked, you can now make the rule and route changes permanent. This depends on what version of Unix you are using. As before, I’m assuming a RH/CentOS-based Linux distribution.
$ echo "from eno1 table isp1" > /etc/sysconfig/network-scripts/rule-eno1
$ echo "default via 192.168.1.1 dev eno1 table isp1" > /etc/sysconfig/network-scripts/route-eno1
Test that the network change is permanent:
$ ifdown eno1 ; ifup eno1
If that didn’t work, on the later versions of RH/CentOS you also need to go with one of two options:
- Don’t use the default NetworkManager.service; Use network.service instead. I haven’t explored the exact steps needed for this. I would imagine it involves the standard chkconfig or systemctl commands to enable/disable services.
or
- Install the NetworkManager-dispatcher-routing-rules package
Personally I prefer installing the rules package as it is the simpler more supported approach:
$ yum install NetworkManager-dispatcher-routing-rules
Another strong recommendation is to enable arp filtering as this prevents other related issues with dual network configurations. With RH/CentOS, add the following content to the /etc/sysctl.conf file:
net.ipv4.conf.default.arp_filter=1
net.ipv4.conf.all.arp_filter=1