Problem: When there is additional public subnet ip address static nat configured then sending traffic from vm to public is not going via additional public subnet. It is going via the default source nat interface.
Root cause: The problem is in the iptables mangle rule where packet got accepted by ACL_OUTBOUND_eth2 before marking got applied.
Moving the rule ACL_OUTBOUND_eth2 to the end will solve the problem.
root@r-189-JAY:~# iptables -t mangle -L -nv
Chain PREROUTING (policy ACCEPT 498 packets, 29013 bytes)
pkts bytes target prot opt in out source destination
36 3024 CONNMARK all -- eth2 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED CONNMARK restore
1 84 ACL_OUTBOUND_eth2 all -- eth2 * 10.1.1.0/24 !10.1.1.1 state NEW
0 0 CONNMARK all -- eth1 * 0.0.0.0/0 0.0.0.0/0 state NEW CONNMARK set 0x1
0 0 MARK all -- * * 10.1.1.60 0.0.0.0/0 state NEW MARK set 0x1
0 0 CONNMARK all -- * * 10.1.1.60 0.0.0.0/0 state NEW CONNMARK save
0 0 CONNMARK all -- eth3 * 0.0.0.0/0 0.0.0.0/0 state NEW CONNMARK set 0x3
2 129 MARK all -- * * 10.1.1.36 0.0.0.0/0 state NEW MARK set 0x3
2 129 CONNMARK all -- * * 10.1.1.36 0.0.0.0/0 state NEW CONNMARK save
Chain INPUT (policy ACCEPT 425 packets, 22881 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 74 packets, 6216 bytes)
pkts bytes target prot opt in out source destination
74 6216 VPN_STATS_eth3 all -- * * 0.0.0.0/0 0.0.0.0/0
74 6216 VPN_STATS_eth1 all -- * * 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 390 packets, 49102 bytes)
pkts bytes target prot opt in out source destination
0 0 CHECKSUM udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:68 CHECKSUM fill
Chain POSTROUTING (policy ACCEPT 464 packets, 55318 bytes)
pkts bytes target prot opt in out source destination
0 0 CHECKSUM udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:68 CHECKSUM fill
Chain ACL_OUTBOUND_eth2 (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 224.0.0.18
0 0 ACCEPT all -- * * 0.0.0.0/0 225.0.0.50
1 84 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain VPN_STATS_eth1 (1 references)
pkts bytes target prot opt in out source destination
0 0 all -- * eth1 0.0.0.0/0 0.0.0.0/0 mark match 0x525
0 0 all -- eth1 * 0.0.0.0/0 0.0.0.0/0 mark match 0x524
Chain VPN_STATS_eth3 (1 references)
pkts bytes target prot opt in out source destination
0 0 all -- * eth3 0.0.0.0/0 0.0.0.0/0 mark match 0x525
0 0 all -- eth3 * 0.0.0.0/0 0.0.0.0/0 mark match 0x524
root@r-189-JAY:~# iptables -t mangle -L -nv^C
root@r-189-JAY:~# iptables-save -t mangle
root@r-189-JAY:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 0e:00:a9:fe:01:5b brd ff:ff:ff:ff:ff:ff
inet 169.254.1.91/16 brd 169.254.255.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 1e:00:9d:00:00:0f brd ff:ff:ff:ff:ff:ff
inet 10.147.46.103/24 brd 10.147.46.255 scope global eth1
inet 10.147.46.106/24 brd 10.147.46.255 scope global secondary eth1
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:00:42:73:00:02 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.1/24 brd 10.1.1.255 scope global eth2
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 1e:00:a4:00:00:28 brd ff:ff:ff:ff:ff:ff
inet 10.147.55.101/24 brd 10.147.55.255 scope global eth3
root@r-189-JAY:~#
