Source NAT changes IP, but doesn't route to correct uplink

[Guest]

[jvaughn]

We have several uplinks. By default, all outgoing traffic goes to main uplink. We can use SNAT to set the outgoing IP of traffic for a given internal IP, and this works fine for IPs on the main uplink, but when doing so for traffic that should go out the other uplinks, the SNAT changes the IP but it still tries to exit on eth1 (main uplink), where it dies as the other end of main uplink throws away the packet.

You would think that the routing table would handle this, but apparently not. Even though appropriate routes exist for the uplinks, the packets originating from IPs on those other uplinks end up on the main uplink interface.

We can use policy routes to make anything from those IPs go to a given uplink, but then we can't access our own sites from within the network because the return traffic from the servers gets shoved out onto the uplink rather than coming back to the internal IP of the user that requested it. Since Endian doesn't allow us to create rules that use the '!' negation operator, we can't simply exclude our internal network from the policy route.

For example:
10.10.104.x > SNAT to main uplink 97.x.x.y
10.10.102.x > SNAT to uplink2 50.x.x.y

Ping from 10.10.102.200 to 8.8.8.8 with SNAT, SNAT turns it into 50.x.x.200, but then the packet is placed on main uplink (eth1) and is nothing routes it further on the other end; In this configuration we CAN access the external IP from another machine internally
Ping from 10.10.102.200 to 8.8.8.8  with SNAT off, it gets NAT'd to 97.x.x.2, gets placed on main uplink (eth1) and routes fine; In this configuration we CAN access the external IP from another machine internally however for DNS and other UDP purposes this won't work properly as the replies from the destination have the wrong source IP
Ping from 10.10.102.200 to 8.8.8.8  with SNAT on and a policy route to force source 10.10.102.200 to uplink2, SNAT makes it 50.x.x.200 and it routes over eth4 (uplink2) correctly, and works as it should; However in this configuration we CANNOT access the external IP from another machine internally

Ping from 10.10.104.8 to 8.8.8.8 with SNAT, SNAT turns it into 97.x.x.8, is placed correctly on eth1, and routes; we CAN reach it internally
Ping from 10.10.104.8 to 8.8.8.8 without SNAT, NAT turns it into 97.x.x.2, is placed correctly on eth1, and routes; we CAN reach it internally however for DNS and other UDP purposes this won't work properly as the replies from the destination have the wrong source IP

[jvaughn]

Maybe this is the problem: Endian is placing rules for the source NAT to go to all interfaces, not just the correct one (eth4 in this case):
-A SOURCENAT -s 10.10.102.200/32 -o eth0 -j SNAT --to-source 50.x.x.200
-A SOURCENAT -s 10.10.102.200/32 -o eth1 -j SNAT --to-source 50.x.x.200
-A SOURCENAT -s 10.10.102.200/32 -o eth2 -j SNAT --to-source 50.x.x.200
-A SOURCENAT -s 10.10.102.200/32 -o eth4 -j SNAT --to-source 50.x.x.200

I'm trying to locate the script that generates these rules, to see if I can figure out how to fix it. Though, Perl is not my strong suit ...

[jvaughn]

Well it appears to call a python script to actually update the rules, and that calls some compiled python code ... so I'm stuck.

[jvaughn]

Well, nevermind.. it appears to have fixed itself