Thursday, February 14, 2019

[rp_filter] Packet request and replies are are seen on interface but not forwarded


Did you observe the following problems  on router ?

1. Router with two or more interfaces forwarding the packets has problem of  'reply packet is seen on the interface but not forwarded to another interface', given iptables rules are correct.

2.  Reply packet is seen on iptables mangle PREROUTING chain after that it is disappeared.
     Packet is not sent to iptables FORWARD chain after routing.

3. packet source address is not directly routable via the interface it is not reachable, packet received on the interface but sent out or forwarded.

4.  A non default route public interface traffic received that packet but not forwarded, given iptables rules are correct.

If you observe the above problems then it is the problem of rp_filter setting on the interface.


What is linux rp_filter (reverse path forwarding)
/proc/sys/net/ipv4/conf/<interface>/rp_filter

rp_filter - 0    disable or

0. no source validation

rp_filter - 1    strict

1. When a machine with interface rp_filter enabled receives a packet on the interface, if this received packet source is not reachable (routable) via this interface (the interface on which the packet received)  then the packet is dropped.

  If it is routable through the interface which it came, then the machine will accept the packet
  If it is not routable through the interface, which it came, then the machine will drop that packet.


rp_filter - 2   loose

if the received packet source address is routable with any of the routes on the any of the interface on the router then it accepts the packet.

Example:
Router has two public interfaces eth1, eth2.
eth1 is default route interface
eth2 is non default route interface but it has policy based routes.

rp_filter on eth2 is set to 1, its ip is 10.147.52.100 .

A packet with (src, dst) 10.233.70.100 ---- 10.147.52.100 is received on eth2.  Now eth2 will check whether 10.233.70.100 is reachable with its routes. If the packet is not reachable then it will drop the packet. That means its reverse path is not reachable.

conf/{all,interface}/rp_filter 

rp_filter takes the max of {all,interface} value and then takes the action. if all->1 and interface value is 2 then interface rp_filter will take effect.

/proc/sys/net/ipv4/conf/eth5/log_martians
This will log the dropped packets of the rp_filter in dmesg.

#dmesg
[61162.827562] ll header: 00000000: 1e 00 18 00 00 9b 38 90 a5 27 4b d4 08 00        ......8..'K...
[61163.842472] IPv4: martian source 10.1.3.145 from 8.8.8.8, on dev eth2


Destination mac: 1e 00 18 00 00 9b
Source mac: 38 90 a5 27 4b d4
Type: 08 is ip header 



When rp_filter takes the action:
1. rp_filter checks the packet after the SNAT source nat get applied on the packet  post routing

2. On return traffic after prerouting, conntrack changes the destination ip address of the NAT packet. Before forwarding the packet rp_filter checks  whether source ip reachable from this interface then takes the action.


Let me show the problem in the setup.

Setup image:




Video explanation:







 




Thursday, October 11, 2018

Adding static ip route when interface is down or Network is unreachable

Adding static route on interface fails when interface is down


root@r-118-JAY:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 0e:00:a9:fe:00:84 brd ff:ff:ff:ff:ff:ff
    inet 169.254.0.132/16 brd 169.254.255.255 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 1e:00:7c:00:00:11 brd ff:ff:ff:ff:ff:ff
    inet 10.147.46.105/24 brd 10.147.46.255 scope global eth1
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 1e:00:7c:00:00:22 brd ff:ff:ff:ff:ff:ff
    inet 10.147.52.202/24 brd 10.147.52.255 scope global eth2
       valid_lft forever preferred_lft forever
root@r-118-JAY:~#
root@r-118-JAY:~# ip route show table Table_eth2
throw 10.147.46.0/24  proto static
throw 10.147.70.0/24  proto static
throw 10.147.80.0/24
throw 10.147.81.0/24
root@r-118-JAY:~# ip route show table static_route
root@r-118-JAY:~#
root@r-118-JAY:~#  ip route add throw 10.147.81.0/24 table static_route
ip route throw on static_route table is success but below route add fails.
root@r-118-JAY:~# ip route add 10.147.70.0/24 via 10.147.52.1 table static_route
RTNETLINK answers: Network is unreachable


This route add is failing because ip interface eth2 (10.147.52.0/24) subnet is down. Here the route using via 10.147.52.1 which is not reachable now. Due to this the route add is failing.
 
root@r-118-JAY:~#
root@r-118-JAY:~# ip route show table static_route
throw 10.147.81.0/24
root@r-118-JAY:~#
 

Wednesday, October 10, 2018

keepalived configuration


global_defs {
    router_id r-17-VM
}

vrrp_script heartbeat {
    script "/ramdisk/rrouter/heartbeat.sh"
    interval 5
}

vrrp_instance inside_network {
    state BACKUP
    interface eth0
    virtual_router_id 51
    nopreempt

    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 9YasrrhZ
    }

    virtual_ipaddress {
        10.1.1.1 brd 10.1.1.255 dev eth0
    }

    track_script {
        heartbeat
    }

    !That's the correct path of the master.py file.
    notify_backup "/opt/cloud/bin/master.py --backup"
    notify_master "/opt/cloud/bin/master.py --master"
    notify_fault "/opt/cloud/bin/master.py --fault"
}root@r-17-VM:~#




conntrackd.conf:

Sync {
    Mode FTFW {
        #
        # Size of the resend queue (in objects). This is the maximum
        # number of objects that can be stored waiting to be confirmed
        # via acknoledgment. If you keep this value low, the daemon
        # will have less chances to recover state-changes under message
        # omission. On the other hand, if you keep this value high,
        # the daemon will consume more memory to store dead objects.
        # Default is 131072 objects.
        #
        # ResendQueueSize 131072

        #
        # This parameter allows you to set an initial fixed timeout
        # for the committed entries when this node goes from backup
        # to primary. This mechanism provides a way to purge entries
        # that were not recovered appropriately after the specified
        # fixed timeout. If you set a low value, TCP entries in
        # Established states with no traffic may hang. For example,
        # an SSH connection without KeepAlive enabled. If not set,
        # the daemon uses an approximate timeout value calculation
        # mechanism. By default, this option is not set.
        #
        # CommitTimeout 180

        #
        # If the firewall replica goes from primary to backup,
        # the conntrackd -t command is invoked in the script.
        # This command schedules a flush of the table in N seconds.
        # This is useful to purge the connection tracking table of
        # zombie entries and avoid clashes with old entries if you
        # trigger several consecutive hand-overs. Default is 60 seconds.
        #
        # PurgeTimeout 60

        # Set the acknowledgement window size. If you decrease this
        # value, the number of acknowlegdments increases. More
        # acknowledgments means more overhead as conntrackd has to
        # handle more control messages. On the other hand, if you
        # increase this value, the resend queue gets more populated.
        # This results in more overhead in the queue releasing.
        # The following value is based on some practical experiments
        # measuring the cycles spent by the acknowledgment handling
        # with oprofile. If not set, default window size is 300.
        #
        # ACKWindowSize 300

        #
        # This clause allows you to disable the external cache. Thus,
        # the state entries are directly injected into the kernel
        # conntrack table. As a result, you save memory in user-space
        # but you consume slots in the kernel conntrack table for
        # backup state entries. Moreover, disabling the external cache
        # means more CPU consumption. You need a Linux kernel
        # >= 2.6.29 to use this feature. By default, this clause is
        # set off. If you are installing conntrackd for first time,
        # please read the user manual and I encourage you to consider
        # using the fail-over scripts instead of enabling this option!
        #
        # DisableExternalCache Off
    }

    #
    # Multicast IP and interface where messages are
    # broadcasted (dedicated link). IMPORTANT: Make sure
    # that iptables accepts traffic for destination
    # 225.0.0.50, eg:
    #
    #    iptables -I INPUT -d 225.0.0.50 -j ACCEPT
    #    iptables -I OUTPUT -d 225.0.0.50 -j ACCEPT
    #
    Multicast {
IPv4_address 225.0.0.50
Group 3780
IPv4_interface 10.1.1.197
Interface eth0
SndSocketBuffer 1249280
RcvSocketBuffer 1249280
Checksum on
    }
    #
    # You can specify more than one dedicated link. Thus, if one dedicated
    # link fails, conntrackd can fail-over to another. Note that adding
    # more than one dedicated link does not mean that state-updates will
    # be sent to all of them. There is only one active dedicated link at
    # a given moment. The `Default' keyword indicates that this interface
    # will be selected as the initial dedicated link. You can have
    # up to 4 redundant dedicated links. Note: Use different multicast
    # groups for every redundant link.
    #
    # Multicast Default {
    #    IPv4_address 225.0.0.51
    #    Group 3781
    #    IPv4_interface 192.168.100.101
    #    Interface eth3
    #    # SndSocketBuffer 1249280
    #    # RcvSocketBuffer 1249280
    #    Checksum on
    # }

    #
    # You can use Unicast UDP instead of Multicast to propagate events.
    # Note that you cannot use unicast UDP and Multicast at the same
    # time, you can only select one.
    #
    # UDP {
        #
        # UDP address that this firewall uses to listen to events.
        #
        # IPv4_address 192.168.2.100
        #
        # or you may want to use an IPv6 address:
        #
        # IPv6_address fe80::215:58ff:fe28:5a27

        #
        # Destination UDP address that receives events, ie. the other
        # firewall's dedicated link address.
        #
        # IPv4_Destination_Address 192.168.2.101
        #
        # or you may want to use an IPv6 address:
        #
        # IPv6_Destination_Address fe80::2d0:59ff:fe2a:775c

        #
        # UDP port used
        #
        # Port 3780

        #
        # The name of the interface that you are going to use to
        # send the synchronization messages.
        #
        # Interface eth2

        #
        # The sender socket buffer size
        #
        # SndSocketBuffer 1249280

        #
        # The receiver socket buffer size
        #
        # RcvSocketBuffer 1249280

        #
        # Enable/Disable message checksumming.
        #
        # Checksum on
    # }

}

#
# General settings
#
General {
    #
    # Set the nice value of the daemon, this value goes from -20
    # (most favorable scheduling) to 19 (least favorable). Using a
    # very low value reduces the chances to lose state-change events.
    # Default is 0 but this example file sets it to most favourable
    # scheduling as this is generally a good idea. See man nice(1) for
    # more information.
    #
    Nice -20

    #
    # Select a different scheduler for the daemon, you can select between
    # RR and FIFO and the process priority (minimum is 0, maximum is 99).
    # See man sched_setscheduler(2) for more information. Using a RT
    # scheduler reduces the chances to overrun the Netlink buffer.
    #
    # Scheduler {
    #    Type FIFO
    #    Priority 99
    # }

    #
    # Number of buckets in the cache hashtable. The bigger it is,
    # the closer it gets to O(1) at the cost of consuming more memory.
    # Read some documents about tuning hashtables for further reference.
    #
    HashSize 32768

    #
    # Maximum number of conntracks, it should be double of:
    # $ cat /proc/sys/net/netfilter/nf_conntrack_max
    # since the daemon may keep some dead entries cached for possible
    # retransmission during state synchronization.
    #
    HashLimit 131072

    #
    # Logfile: on (/var/log/conntrackd.log), off, or a filename
    # Default: off
    #
    LogFile on

    #
    # Syslog: on, off or a facility name (daemon (default) or local0..7)
    # Default: off
    #
    #Syslog on

    #
    # Lockfile
    #
    LockFile /var/lock/conntrack.lock

    #
    # Unix socket configuration
    #
    UNIX {
        Path /var/run/conntrackd.ctl
        Backlog 20
    }

    #
    # Netlink event socket buffer size. If you do not specify this clause,
    # the default buffer size value in /proc/net/core/rmem_default is
    # used. This default value is usually around 100 Kbytes which is
    # fairly small for busy firewalls. This leads to event message dropping
    # and high CPU consumption. This example configuration file sets the
    # size to 2 MBytes to avoid this sort of problems.
    #
    NetlinkBufferSize 2097152

    #
    # The daemon doubles the size of the netlink event socket buffer size
    # if it detects netlink event message dropping. This clause sets the
    # maximum buffer size growth that can be reached. This example file
    # sets the size to 8 MBytes.
    #
    NetlinkBufferSizeMaxGrowth 8388608

    #
    # If the daemon detects that Netlink is dropping state-change events,
    # it automatically schedules a resynchronization against the Kernel
    # after 30 seconds (default value). Resynchronizations are expensive
    # in terms of CPU consumption since the daemon has to get the full
    # kernel state-table and purge state-entries that do not exist anymore.
    # Be careful of setting a very small value here. You have the following
    # choices: On (enabled, use default 30 seconds value), Off (disabled)
    # or Value (in seconds, to set a specific amount of time). If not
    # specified, the daemon assumes that this option is enabled.
    #
    # NetlinkOverrunResync On

    #
    # If you want reliable event reporting over Netlink, set on this
    # option. If you set on this clause, it is a good idea to set off
    # NetlinkOverrunResync. This option is off by default and you need
    # a Linux kernel >= 2.6.31.
    #
    # NetlinkEventsReliable Off

    #
    # By default, the daemon receives state updates following an
    # event-driven model. You can modify this behaviour by switching to
    # polling mode with the PollSecs clause. This clause tells conntrackd
    # to dump the states in the kernel every N seconds. With regards to
    # synchronization mode, the polling mode can only guarantee that
    # long-lifetime states are recovered. The main advantage of this method
    # is the reduction in the state replication at the cost of reducing the
    # chances of recovering connections.
    #
    # PollSecs 15

    #
    # The daemon prioritizes the handling of state-change events coming
    # from the core. With this clause, you can set the maximum number of
    # state-change events (those coming from kernel-space) that the daemon
    # will handle after which it will handle other events coming from the
    # network or userspace. A low value improves interactivity (in terms of
    # real-time behaviour) at the cost of extra CPU consumption.
    # Default (if not set) is 100.
    #
    # EventIterationLimit 100

    #
    # Event filtering: This clause allows you to filter certain traffic,
    # There are currently three filter-sets: Protocol, Address and
    # State. The filter is attached to an action that can be: Accept or
    # Ignore. Thus, you can define the event filtering policy of the
    # filter-sets in positive or negative logic depending on your needs.
    # You can select if conntrackd filters the event messages from
    # user-space or kernel-space. The kernel-space event filtering
    # saves some CPU cycles by avoiding the copy of the event message
    # from kernel-space to user-space. The kernel-space event filtering
    # is prefered, however, you require a Linux kernel >= 2.6.29 to
    # filter from kernel-space. If you want to select kernel-space
    # event filtering, use the keyword 'Kernelspace' instead of
    # 'Userspace'.
    #
    Filter From Userspace {
        #
        # Accept only certain protocols: You may want to replicate
        # the state of flows depending on their layer 4 protocol.
        #
        Protocol Accept {
            TCP
            SCTP
            DCCP
            # UDP
            # ICMP # This requires a Linux kernel >= 2.6.31
        }

        #
        # Ignore traffic for a certain set of IP's: Usually all the
        # IP assigned to the firewall since local traffic must be
        # ignored, only forwarded connections are worth to replicate.
        # Note that these values depends on the local IPs that are
        # assigned to the firewall.
        #
        Address Ignore {
            IPv4_address 127.0.0.1
            IPv4_address 169.254.0.4
        }

        #
        # Uncomment this line below if you want to filter by flow state.
        # This option introduces a trade-off in the replication: it
        # reduces CPU consumption at the cost of having lazy backup
        # firewall replicas. The existing TCP states are: SYN_SENT,
        # SYN_RECV, ESTABLISHED, FIN_WAIT, CLOSE_WAIT, LAST_ACK,
        # TIME_WAIT, CLOSED, LISTEN.
        #
        # State Accept {
        #    ESTABLISHED CLOSED TIME_WAIT CLOSE_WAIT for TCP
        # }
    }
}
root@r-17-VM:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 02:00:63:d3:00:05 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.197/24 brd 10.1.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.1.1.1/32 brd 10.1.1.255 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 0e:00:a9:fe:00:04 brd ff:ff:ff:ff:ff:ff
    inet 169.254.0.4/16 brd 169.254.255.255 scope global eth1
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 1e:00:3d:00:00:10 brd ff:ff:ff:ff:ff:ff
    inet 10.147.48.42/24 brd 10.147.48.255 scope global eth2
       valid_lft forever preferred_lft forever
    inet 10.147.48.45/24 brd 10.147.48.255 scope global secondary eth2
       valid_lft forever preferred_lft forever
root@r-17-VM:~#

Why both VRRP instances both become backup or master

1. When both Master/primay and Backup become master

One of the reason for both VRRP instances become master is
A. There is no reachability (connection issue) between the two. When there is no reachability both instances send VRRP packets and assume master as themselves.


B. The auth type protocol mismatch in VRRP instances in keepalived.conf

When there is auth_type mismatch between the two then both VRRP instances does not understand VRRP packets themselves and both become master.




2. When VRRP instance become fault
If keepavlied service stop on the instance.

Stateful firewall conntrack states transistion

This blog shows how to check the conntrack state changes when a connection is made or connection got dropped.

Using conntrack command we can states update information.

Example conntrack command to show connection states.
Conntrack command example filter:
# conntrack -E -p tcp --dport 33

1. First ssh connection request sent to firewall public ip address.

    [NEW] tcp      6 120 SYN_SENT src=10.233.89.39 dst=10.147.46.107 sport=64887 dport=33 [UNREPLIED] src=10.1.1.44 dst=10.233.89.39 sport=22 dport=64887 mark=2
 [UPDATE] tcp      6 60 SYN_RECV src=10.233.89.39 dst=10.147.46.107 sport=64887 dport=33 src=10.1.1.44 dst=10.233.89.39 sport=22 dport=64887 mark=2
 [UPDATE] tcp      6 432000 ESTABLISHED src=10.233.89.39 dst=10.147.46.107 sport=64887 dport=33 src=10.1.1.44 dst=10.233.89.39 sport=22 dport=64887 [ASSURED] mark=2


 2. ssh session got closed.
[UPDATE] tcp      6 120 FIN_WAIT src=10.233.89.39 dst=10.147.46.107 sport=64887 dport=33 src=10.1.1.44 dst=10.233.89.39 sport=22 dport=64887 [ASSURED] mark=2
 [UPDATE] tcp      6 30 LAST_ACK src=10.233.89.39 dst=10.147.46.107 sport=64887 dport=33 src=10.1.1.44 dst=10.233.89.39 sport=22 dport=64887 [ASSURED] mark=2
 [UPDATE] tcp      6 120 TIME_WAIT src=10.233.89.39 dst=10.147.46.107 sport=64887 dport=33 src=10.1.1.44 dst=10.233.89.39 sport=22 dport=64887 [ASSURED] mark=2




Tuesday, July 3, 2018

Conntrackd connection sync checking.

Setup:
Setup is a redundant router setup with one master router and one backup router.


Before making connection  (ssh to VM which is behind this nat router) the conntrackd states:
1. -i shows the internal connection states
2. -e shows the external connection states

For master the connection will be internal. For backup the connection entries will be external because it is synced from master. 

root@r-15-JAY:~# checkrouter.sh
Status: MASTER
root@r-15-JAY:~# conntrackd -i
root@r-15-JAY:~# conntrackd -e
root@r-15-JAY:~# conntrackd -s
cache internal:
current active connections:               0
connections created:                  15    failed:               0
connections updated:                  39    failed:               0
connections destroyed:                  15    failed:               0

cache external:
current active connections:               0
connections created:                   3    failed:               0
connections updated:                   2    failed:               0
connections destroyed:                   3    failed:               0

traffic processed:
                   0 Bytes                         0 Pckts

multicast traffic (active device=eth0):
               86876 Bytes sent                26508 Bytes recv
                8749 Pckts sent                 1638 Pckts recv
                   0 Error send                    0 Error recv

message tracking:
                   0 Malformed msgs                    0 Lost msgs


root@r-16-JAY:~# checkrouter.sh
Status: BACKUP
root@r-16-JAY:~# conntrackd -i
root@r-16-JAY:~# conntrackd -e
root@r-16-JAY:~# conntrackd -s
cache internal:
current active connections:               0
connections created:                   0    failed:               0
connections updated:                   0    failed:               0
connections destroyed:                   0    failed:               0

cache external:
current active connections:               0
connections created:                  11    failed:               0
connections updated:                  38    failed:               0
connections destroyed:                  11    failed:               0

traffic processed:
                   0 Bytes                         0 Pckts

multicast traffic (active device=eth0):
               21576 Bytes sent                25324 Bytes recv
                1354 Pckts sent                 1398 Pckts recv
                   0 Error send                    0 Error recv

message tracking:
                   0 Malformed msgs                    0 Lost msgs

After ssh connection is made to VM which is behind the router:

root@r-15-JAY:~# conntrackd -i
tcp      6 ESTABLISHED src=10.233.88.78 dst=10.147.52.202 sport=63913 dport=22 src=10.1.1.218 dst=10.233.88.78 sport=22 dport=63913 [ASSURED] mark=2 [active since 5s]
root@r-15-JAY:~# conntrackd -e
root@r-15-JAY:~# conntrackd -s
cache internal:
current active connections:               1
connections created:                  16    failed:               0
connections updated:                  41    failed:               0
connections destroyed:                  15    failed:               0

cache external:
current active connections:               0
connections created:                   3    failed:               0
connections updated:                   2    failed:               0
connections destroyed:                   3    failed:               0

traffic processed:
                   0 Bytes                         0 Pckts

multicast traffic (active device=eth0):
               88040 Bytes sent                27452 Bytes recv
                8811 Pckts sent                 1697 Pckts recv
                   0 Error send                    0 Error recv

message tracking:
                   0 Malformed msgs                    0 Lost msgs
root@r-16-JAY:~# conntrackd -e
tcp      6 ESTABLISHED src=10.233.88.78 dst=10.147.52.202 sport=63913 dport=22 [ASSURED] mark=2 [active since 11s]
root@r-16-JAY:~# conntrackd -i
root@r-16-JAY:~# conntrackd -s
cache internal:
current active connections:               0   ------------------- Here connection is not active.
connections created:                   0    failed:               0
connections updated:                   0    failed:               0
connections destroyed:                   0    failed:               0

cache external:
current active connections:               1
connections created:                  12    failed:               0
connections updated:                  40    failed:               0
connections destroyed:                  11    failed:               0

traffic processed:
                   0 Bytes                         0 Pckts

multicast traffic (active device=eth0):
               22152 Bytes sent                26096 Bytes recv
                1390 Pckts sent                 1436 Pckts recv
                   0 Error send                    0 Error recv

message tracking:
                   0 Malformed msgs                    0 Lost msgs

root@r-16-JAY:~#

When there is switch (Backup becomes Master) the conntrack active since will be reset in the new Master.

Tuesday, October 31, 2017

A greedy grep


While using the grep for pattern matching keep in mind that grep is greedy on matching.
grep gives the results when the part of the string matches.

Your pattern string should be  made to match the lines uniquely.
In the below example instead of hcptest it can be matched with ',hcptest1,'.

A good QA will includes the test cases of below kind when grep command is seen.

$ cat /tmp/test.txt
02:03:01:1a:00:73,set:10_31_0_31,10.31.0.31,dhcptest1,infinite
02:03:01:1a:00:74,set:10_31_0_14,10.31.0.14,dhcptest2,infinite
02:03:01:1a:00:74,set:10_31_0_14,10.31.0.14,hcptest1,infinite

$cat /tmp/test.txt | grep -i 'hcptest1' --color
02:03:01:1a:00:73,set:10_31_0_31,10.31.0.31,dhcptest1,infinite
02:03:01:1a:00:74,set:10_31_0_14,10.31.0.14,hcptest1,infinite

Wednesday, September 6, 2017

Additional public subnet traffic routing


Problem: When there is additional public subnet ip address static nat configured then sending traffic from vm to public is not going via additional public subnet. It is going via the default source nat interface.

Root cause:  The problem is in the iptables mangle rule where packet got accepted by ACL_OUTBOUND_eth2  before marking got applied.

Moving the rule  ACL_OUTBOUND_eth2 to the end will solve the problem.


root@r-189-JAY:~# iptables -t mangle -L -nv
Chain PREROUTING (policy ACCEPT 498 packets, 29013 bytes)
 pkts bytes target     prot opt in     out     source               destination        
   36  3024 CONNMARK   all  --  eth2   *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED CONNMARK restore
    1    84 ACL_OUTBOUND_eth2  all  --  eth2   *       10.1.1.0/24         !10.1.1.1             state NEW
    0     0 CONNMARK   all  --  eth1   *       0.0.0.0/0            0.0.0.0/0            state NEW CONNMARK set 0x1
    0     0 MARK       all  --  *      *       10.1.1.60            0.0.0.0/0            state NEW MARK set 0x1
    0     0 CONNMARK   all  --  *      *       10.1.1.60            0.0.0.0/0            state NEW CONNMARK save
    0     0 CONNMARK   all  --  eth3   *       0.0.0.0/0            0.0.0.0/0            state NEW CONNMARK set 0x3
    2   129 MARK       all  --  *      *       10.1.1.36            0.0.0.0/0            state NEW MARK set 0x3
    2   129 CONNMARK   all  --  *      *       10.1.1.36            0.0.0.0/0            state NEW CONNMARK save

Chain INPUT (policy ACCEPT 425 packets, 22881 bytes)
 pkts bytes target     prot opt in     out     source               destination        

Chain FORWARD (policy ACCEPT 74 packets, 6216 bytes)
 pkts bytes target     prot opt in     out     source               destination        
   74  6216 VPN_STATS_eth3  all  --  *      *       0.0.0.0/0            0.0.0.0/0          
   74  6216 VPN_STATS_eth1  all  --  *      *       0.0.0.0/0            0.0.0.0/0          

Chain OUTPUT (policy ACCEPT 390 packets, 49102 bytes)
 pkts bytes target     prot opt in     out     source               destination        
    0     0 CHECKSUM   udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:68 CHECKSUM fill

Chain POSTROUTING (policy ACCEPT 464 packets, 55318 bytes)
 pkts bytes target     prot opt in     out     source               destination        
    0     0 CHECKSUM   udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:68 CHECKSUM fill

Chain ACL_OUTBOUND_eth2 (1 references)
 pkts bytes target     prot opt in     out     source               destination        
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            224.0.0.18         
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            225.0.0.50         
    1    84 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0          

Chain VPN_STATS_eth1 (1 references)
 pkts bytes target     prot opt in     out     source               destination        
    0     0            all  --  *      eth1    0.0.0.0/0            0.0.0.0/0            mark match 0x525
    0     0            all  --  eth1   *       0.0.0.0/0            0.0.0.0/0            mark match 0x524

Chain VPN_STATS_eth3 (1 references)
 pkts bytes target     prot opt in     out     source               destination        
    0     0            all  --  *      eth3    0.0.0.0/0            0.0.0.0/0            mark match 0x525
    0     0            all  --  eth3   *       0.0.0.0/0            0.0.0.0/0            mark match 0x524
root@r-189-JAY:~# iptables -t mangle -L -nv^C
root@r-189-JAY:~# iptables-save -t mangle

root@r-189-JAY:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 0e:00:a9:fe:01:5b brd ff:ff:ff:ff:ff:ff
    inet 169.254.1.91/16 brd 169.254.255.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 1e:00:9d:00:00:0f brd ff:ff:ff:ff:ff:ff
    inet 10.147.46.103/24 brd 10.147.46.255 scope global eth1
    inet 10.147.46.106/24 brd 10.147.46.255 scope global secondary eth1
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 02:00:42:73:00:02 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.1/24 brd 10.1.1.255 scope global eth2
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 1e:00:a4:00:00:28 brd ff:ff:ff:ff:ff:ff
    inet 10.147.55.101/24 brd 10.147.55.255 scope global eth3

root@r-189-JAY:~#

Wednesday, August 9, 2017

How to setup VPN server PPTP on CentOS



             How to setup PPTP server on CentOS

 

My setup is cloudstack  as  below:

VM --------------------------VR (Virtual Router) -----------------------Public network (windows)
v1n1(pptp server)               public interface 10.147.46.102


  Configuring pptp server in the Centos VM (v1n1)

 1. Install the pptp server in cents
      #rpm -i http://poptop.sourceforge.net/yum/stable/rhel6/pptp-release-current.noarch.rpm
          warning: /var/tmp/rpm-tmp.zgYGDb: Header V3 DSA/SHA1 Signature, key ID 862acc42: NOKEY
      # yum -y install pptpd

2. Before updating the configuration take backup of the configuration.
 Copy the below below configuration files into your pptp server.

[root@v1n1 ~]# cat /etc/*-release
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)

[root@v1n1 ~]# cat /etc/pptpd.conf
#start of custom file
#logwtmp
option /etc/ppp/options.pptpd
localip 192.168.0.1   # local vpn IP
remoteip 192.168.0.234-238,192.168.0.245[root@v1n1 ~]# cat /etc/ppp/chap-secrets
# Secrets for authentication using CHAP
# client    server    secret            IP addresses


test1 * test1 *
test2 * test2 *
[root@v1n1 ~]#
#remoteip 192.168.0.100-200  # ip range for connections
listen 10.1.1.242 # eth0 my example public IP and network interface
#end of custom file
debug
[root@v1n1 ~]#
[root@v1n1 ~]# cat /etc/ppp/options.pptpd
#custom settings for a simple fast pptp server
ms-dns 8.8.8.8
ms-dns 4.2.2.2
lock
name pptpd
#require-mschap-v2
require-chap
# Require MPPE 128-bit encryption
# (note that MPPE requires the use of MSCHAP-V2 during authentication)
# require-mppe-128
[root@v1n1 ~]# 

3. Add the below config in sysctl.conf and run using sysctl -p
net.ipv4.ip_forward = 1


4. Start your pptp server using below command
/etc/init.d/pptpd restart-kill && /etc/init.d/pptpd start

5. make sure you have the below iptables configuration to allow the pptp and gre

[root@v1n1 ~]# iptables -L -nv
Chain INPUT (policy ACCEPT 604 packets, 57798 bytes)
 pkts bytes target     prot opt in     out     source               destination        
 496K   47M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
    2   120 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0          
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0          
    3   192 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22
   39  2028 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:1723
    2   109 ACCEPT     47   --  eth0   *       0.0.0.0/0            0.0.0.0/0 
         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination        
  199 12913 ACCEPT     all  --  ppp+   eth0    0.0.0.0/0            0.0.0.0/0          
  217 24050 ACCEPT     all  --  eth0   ppp+    0.0.0.0/0            0.0.0.0/0          

Chain OUTPUT (policy ACCEPT 4117 packets, 527K bytes)
 pkts bytes target     prot opt in     out     source               destination        
[root@v1n1 ~]#
[root@v1n1 ~]# iptables-save
# Generated by iptables-save v1.4.7 on Wed Aug  9 09:19:50 2017
*nat
:PREROUTING ACCEPT [190:16147]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [13:1119]
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT
# Completed on Wed Aug  9 09:19:50 2017
# Generated by iptables-save v1.4.7 on Wed Aug  9 09:19:50 2017
*filter
:INPUT ACCEPT [604:57798]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [4130:529629]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 1723 -j ACCEPT
-A INPUT -i eth0 -p gre -j ACCEPT
-A FORWARD -i ppp+ -o eth0 -j ACCEPT
-A FORWARD -i eth0 -o ppp+ -j ACCEPT
COMMIT
# Completed on Wed Aug  9 09:19:50 2017
[root@v1n1 ~]#


Configurations with in the VR:

1. Configure port forwarding to VM v1n1 on the VR and also configure firewall on 10.147.46.102 (FIREWALL_10.147.46.102 chain) to allow the pptp port TCP 1723. Here the public interface is eth3.  

2. Insert the pptp modules in VR.
root@r-123-JAY:~# modprobe nf_conntrack_pptp
root@r-123-JAY:~# modprobe nf_nat_pptp 



root@r-123-JAY:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 02:00:1a:98:00:09 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.1/24 brd 10.1.1.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 0e:00:a9:fe:00:b6 brd ff:ff:ff:ff:ff:ff
    inet 169.254.0.182/16 brd 169.254.255.255 scope global eth1
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 1e:00:7f:00:00:27 brd ff:ff:ff:ff:ff:ff
    inet 10.147.55.100/24 brd 10.147.55.255 scope global eth2
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 1e:00:01:00:00:0e brd ff:ff:ff:ff:ff:ff
    inet 10.147.46.102/24 brd 10.147.46.255 scope global eth3

root@r-123-JAY:~#

 
root@r-123-JAY:~# iptables -t mangle -L -nv
Chain PREROUTING (policy ACCEPT 674 packets, 64448 bytes)
 pkts bytes target     prot opt in     out     source               destination        
  507 36204 FIREWALL_10.147.46.102  all  --  *      *       0.0.0.0/0            10.147.46.102      
    0     0 VPN_10.147.46.102  all  --  *      *       0.0.0.0/0            10.147.46.102      
    4   304 VPN_10.147.55.100  all  --  *      *       0.0.0.0/0            10.147.55.100      
    0     0 FIREWALL_10.147.55.100  all  --  *      *       0.0.0.0/0            10.147.55.100      
    0     0 VPN_10.147.55.100  all  --  *      *       0.0.0.0/0            10.147.55.100      
  662 63576 CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED CONNMARK restore
    0     0 CONNMARK   all  --  eth2   *       0.0.0.0/0            0.0.0.0/0            state NEW CONNMARK set 0x2
    0     0 CONNMARK   all  --  eth3   *       0.0.0.0/0            0.0.0.0/0            state NEW CONNMARK set 0x3
    0     0 MARK       all  --  *      *       10.1.1.242           0.0.0.0/0            state NEW MARK set 0x3
    0     0 CONNMARK   all  --  *      *       10.1.1.242           0.0.0.0/0            state NEW CONNMARK save

Chain INPUT (policy ACCEPT 408 packets, 30668 bytes)
 pkts bytes target     prot opt in     out     source               destination        

Chain FORWARD (policy ACCEPT 777 packets, 70288 bytes)
 pkts bytes target     prot opt in     out     source               destination        

Chain OUTPUT (policy ACCEPT 318 packets, 40691 bytes)
 pkts bytes target     prot opt in     out     source               destination        

Chain POSTROUTING (policy ACCEPT 1095 packets, 111K bytes)
 pkts bytes target     prot opt in     out     source               destination        
    0     0 CHECKSUM   udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:68 CHECKSUM fill

Chain FIREWALL_10.147.46.102 (1 references)
 pkts bytes target     prot opt in     out     source               destination        
  507 36204 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            LOG flags 0 level 4 prefix "JAY"
    0     0 RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:1723
    0     0 RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0 
         

         

Chain FIREWALL_10.147.55.100 (1 references)
 pkts bytes target     prot opt in     out     source               destination        
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 255
    0     0 RETURN     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:4500
    0     0 RETURN     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:1701
    0     0 RETURN     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:500
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0          

Chain VPN_10.147.46.102 (1 references)
 pkts bytes target     prot opt in     out     source               destination        
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0          

Chain VPN_10.147.55.100 (2 references)
 pkts bytes target     prot opt in     out     source               destination        
    0     0 ACCEPT     esp  --  *      *       0.0.0.0/0            0.0.0.0/0          
    0     0 ACCEPT     ah   --  *      *       0.0.0.0/0            0.0.0.0/0          
    4   304 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0          
root@r-123-JAY:~#
root@r-123-JAY:~#
root@r-123-JAY:~# iptables -t nat -L -nv
Chain PREROUTING (policy ACCEPT 173 packets, 11501 bytes)
 pkts bytes target     prot opt in     out     source               destination        
    0     0 DNAT       all  --  eth0   *       0.0.0.0/0            10.147.46.102        to:10.1.1.242
   29  1526 DNAT       all  --  *      *       0.0.0.0/0            10.147.46.102        to:10.1.1.242

Chain INPUT (policy ACCEPT 42 packets, 3024 bytes)
 pkts bytes target     prot opt in     out     source               destination        

Chain OUTPUT (policy ACCEPT 5 packets, 380 bytes)
 pkts bytes target     prot opt in     out     source               destination        
    0     0 DNAT       all  --  *      *       0.0.0.0/0            10.147.46.102        to:10.1.1.242

Chain POSTROUTING (policy ACCEPT 29 packets, 1526 bytes)
 pkts bytes target     prot opt in     out     source               destination        
    0     0 SNAT       all  --  *      eth0    10.1.1.0/24          10.1.1.242           to:10.1.1.1
  126  8097 SNAT       all  --  *      eth3    10.1.1.242           0.0.0.0/0            to:10.147.46.102
   10   760 SNAT       all  --  *      eth2    0.0.0.0/0            0.0.0.0/0            to:10.147.55.100
    0     0 SNAT       all  --  *      eth2    0.0.0.0/0            0.0.0.0/0            to:10.147.46.102
root@r-123-JAY:~#
 

Configuration on the windows client:


 

ppp tunnel interface on pptp server after the client connection from the windows

[root@v1n1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 02:00:54:41:00:08 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.242/24 brd 10.1.1.255 scope global eth0
    inet6 fe80::54ff:fe41:8/64 scope link
       valid_lft forever preferred_lft forever
42: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1400 qdisc pfifo_fast state UNKNOWN qlen 3
    link/ppp
    inet 192.168.0.1 peer 192.168.0.234/32 scope global ppp0
[root@v1n1 ~]#


Common errors:
 1. peer refused to authenticate: terminating link *
 This error means your options.pptpd has configured for the mschap-v2 (require-mschap-v2) and user,password are configured in /etc/ppp/chap-secrets. But the client is trying to connect with chap.

Update your /etc/ppp/options.pptpd to require-chap and comment require-mppe-128


*Aug  9 09:38:49 v1n1 pptpd[20130]: CTRL: Client 10.147.38.153 control connection started
Aug  9 09:38:49 v1n1 pptpd[20130]: CTRL: Starting call (launching pppd, opening GRE)
Aug  9 09:38:49 v1n1 pppd[20131]: pppd 2.4.5 started by root, uid 0
Aug  9 09:38:49 v1n1 pppd[20131]: Using interface ppp0
Aug  9 09:38:49 v1n1 pppd[20131]: Connect: ppp0 <--> /dev/pts/1


Aug  9 09:38:52 v1n1 pppd[20131]: peer refused to authenticate: terminating link
Aug  9 09:38:52 v1n1 pppd[20131]: Connection terminated.
Aug  9 09:38:52 v1n1 pppd[20131]: Exit.
Aug  9 09:38:52 v1n1 pptpd[20130]: GRE: read(fd=6,buffer=611860,len=8196) from PTY failed: status = -1 error = Input/output error, usually caused by unexpected termination of pppd, check option syntax and pppd logs
Aug  9 09:38:52 v1n1 pptpd[20130]: CTRL: PTY read or GRE write failed (pty,gre)=(6,7)
Aug  9 09:38:52 v1n1 pptpd[20130]: CTRL: Client 10.147.38.153 control connection finished