pfSense Support Subscription

Author Topic: kernel routing table grows and finaly crash on PFSense 2.3.2 with OSPF 0.99.24.1  (Read 107 times)

0 Members and 1 Guest are viewing this topic.

Offline Green IT

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Hi all,

We installed three months ago 2 PFSense (Hardware, Supermicro MB, Xeon E5, RAM 16 Go, SSD) as Core Firewalls in DC.

In Master / Slave mode, all works fine.
But in Master / Master mode (asymetric full BGP routing), we have routing holes in and out approx. each hour with kernel routing tables flushs and / or growing inexpectidly.

Hope that some one in this forum can help us to diagnose and help resolving our problem :

- an BGP / OSPF misconfiguration in Full BGP routing mode ?
- a Quagga misconfiguration on PFSense ?
- a bug ?

1) Topologywith OSPF Areas

  +-------------+                                          +-------------+
  |             |                                          |             |
  | ISP1 ROUTER |                                          | ISP2 ROUTER |
  |             |                                          |             |
  +------+------+                                          +------+------+
         |                                                        |
         |                                                        |
         |                                                        |
         |                                                        |
+--------+--------+                                      +--------+--------+
|                 |                                      |                 |
|  RBGP1 (CISCO)  |                                      |  RBGP2 (CISCO)  |
|                 |                                      |                 |
+-----------------+                                      +-----------------+
         |Gi0/0/2                                                 |Gi0/0/2
         |                                                        |
         |                                                        |
         |                                                        |
 OSPF AREA 0.0.0.1                                        OSPF AREA 0.0.0.2
         |                                                        |
         |                                                        |
         |                                                        |
         |IGB0                                                    |IGB0
+-----------------+                                      +-----------------+
|                 |RE1                                   |                 |
| FWN11 (PFSENSE) +----------OSPF AREA 0.0.0.0-----------+ FWN12 (PFSENSE) |
|                 |                                   RE1|                 |
+-----------------+                                      +-----------------+
         |IGB1                                                    |IGB1
         |                                                        |
         |                                                        |
         |                                                        |
 OSPF AREA 0.0.0.3                                        OSPF AREA 0.0.0.3

2) Topology with IP Adresses (Not real @IPs)

  +-------------+                                          +-------------+
  |             |                                          |             |
  | ISP1 ROUTER |                                          | ISP2 ROUTER |
  |             |                                          |             |
  +------+------+                                          +------+------+
         |                                                        |
         |                                                        |
         |                                                        |
         |                                                        |
+--------+--------+                                      +--------+--------+
|                 |                                      |                 |
|  RBGP1 (CISCO)  |                                      |  RBGP2 (CISCO)  |
|                 |                                      |                 |
+--------+--------+                                      +--------+--------+
         |                                                        |
         |Gi0/0/2                                                 |Gi0/0/2
         |125.132.100.25/29                                       |125.132.100.33/29
         |                                                        |
         |                                                        |
         |                                                        |
         |IGB0                                                    |IGB0
         |125.132.100.26/29                                       |125.132.100.34/29
         |                                                        |
+--------+--------+RE1                                   +--------+--------+
|                 |125.132.100.17/30                     |                 |
| FWN11 (PFSENSE) +--------------------------------------+ FWN12 (PFSENSE) |
|                 |                                   RE1|                 |
+--------+--------+                     125.132.100.18/30+--------+--------+
         |                                                        |
         |IGB1                                                    |IGB1
         |125.132.100.252/25                                      |125.132.100.253/25
         |                                                        |
         |          +-----------------------------------+         |
         |          | CARP IP  : 125.132.100.254/25     |         |
         +----------+ IP ALIAS : 125.132.102.254/24     +---------+
                    | IP ALIAS : 125.132.103.254/24     |
                    +-+---------------+---------------+-+
                      |               |               |
                      |               |               |
                      |               |               |
                +-----+---+      +----+----+      +---+-----+
                |         |      |         |      |         |
                | HOST 01 |      | HOST 02 |      | HOST 02 |
                |         |      |         |      |         |
                +---------+      +---------+      +---------+

3) FWN11 PFSense Configuration - Quagga (version 0.99.24.1)
Code: [Select]
!
log file /var/log/quagga/ospfd.log informational
log syslog informational
log record-priority
!
interface enc0
 ipv6 nd suppress-ra
 no link-detect
!
interface igb0
 ip ospf hello-interval 5
 ipv6 nd suppress-ra
 ipv6 ospf6 network broadcast
 no link-detect
!
interface igb1
 ipv6 nd suppress-ra
 ipv6 ospf6 network broadcast
 ipv6 ospf6 passive
 no link-detect
!
interface lo0
 no link-detect
!
interface pflog0
 ipv6 nd suppress-ra
 no link-detect
!
interface pfsync0
 ipv6 nd suppress-ra
 no link-detect
!
interface re0
 ipv6 nd suppress-ra
 no link-detect
!
interface re1
 ip ospf hello-interval 5
 ipv6 nd suppress-ra
 ipv6 ospf6 network broadcast
 no link-detect
!
router ospf
 ospf router-id 125.132.100.17
 redistribute kernel
 network 125.132.100.16/30 area 0.0.0.0
 network 125.132.100.24/29 area 0.0.0.1
 network 125.132.100.128/25 area 0.0.0.3
 network 125.132.102.0/27 area 0.0.0.3
 network 125.132.103.0/24 area 0.0.0.3
 area 0.0.0.3 stub no-summary
!
router ospf6
 router-id 125.132.100.17
 redistribute kernel
 interface re1 area 0.0.0.0
 interface igb0 area 0.0.0.1
 interface igb1 area 0.0.0.3
!
ip forwarding
ipv6 forwarding
!
line vty
!
end

4) FWN12 Configuration
Code: [Select]
!
log file /var/log/quagga/ospfd.log informational
log syslog informational
log record-priority
!
interface enc0
 ipv6 nd suppress-ra
 no link-detect
!
interface igb0
 ip ospf hello-interval 5
 ipv6 nd suppress-ra
 no link-detect
!
interface igb1
 ipv6 nd suppress-ra
 no link-detect
!
interface lo0
 no link-detect
!
interface pflog0
 ipv6 nd suppress-ra
 no link-detect
!
interface pfsync0
 ipv6 nd suppress-ra
 no link-detect
!
interface re0
 ipv6 nd suppress-ra
 no link-detect
!
interface re1
 ip ospf hello-interval 5
 ipv6 nd suppress-ra
 no link-detect
!
router ospf
 ospf router-id 125.132.100.18
 redistribute kernel
 network 125.132.100.16/30 area 0.0.0.0
 network 125.132.100.32/29 area 0.0.0.2
 network 125.132.100.128/25 area 0.0.0.3
 network 125.132.102.0/27 area 0.0.0.3
 network 125.132.103.0/24 area 0.0.0.3
 area 0.0.0.3 stub no-summary
!
router ospf6
 router-id 125.132.100.18
 redistribute kernel
 interface re1 area 0.0.0.0
 interface igb0 area 0.0.0.2
 interface igb1 area 0.0.0.3
!
ip forwarding
ipv6 forwarding
!
line vty
!
end
5) RBGP1 Configuration
Code: [Select]
version 15.4
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
no platform punt-keepalive disable-kernel-core
!
hostname RBGP1
!
boot-start-marker
boot system flash bootflash:/asr1001x-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin
boot-end-marker
!
!
vrf definition Mgmt-intf
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
 exit-address-family
!
no aaa new-model
!
no ip domain lookup
ip domain name xxxxxx
!
ipv6 unicast-routing
!
subscriber templating
multilink bundle-name authenticated
!         
!
hw-module subslot 0/0 ethernet vlan unlimited
!
redundancy
 mode none
!
interface Loopback0
 no ip address
 ipv6 address 2006:DC80:0:100::1/128
!
interface Loopback1
 ip address 125.132.100.1 255.255.255.252
!
interface GigabitEthernet0/0/0
 description *** Connexion vers SFR ***
 no ip address
 negotiation auto
!
interface GigabitEthernet0/0/0.1200
 description ***Connexion vers SFR***
 encapsulation dot1Q 1200
 ip address 119.24.138.6 255.255.255.252
 ipv6 address 2002:8400:1:1::206/126
!
interface GigabitEthernet0/0/2
 description Connexion vers SWDN11
 ip address 125.132.100.25 255.255.255.248
 ip ospf dead-interval 40
 ip ospf hello-interval 5
 negotiation auto
 ipv6 address 2006:DC80:0:1100::1/56
 ipv6 ospf 1 area 0.0.0.1
!
interface GigabitEthernet0
 vrf forwarding Mgmt-intf
 ip address 10.19.68.171 255.255.255.0 secondary
 ip address 10.245.0.1 255.255.240.0
 negotiation auto
!
router ospfv3 1
 !
 address-family ipv6 unicast
  redistribute bgp 200098
  router-id 125.132.100.25
  area 0.0.0.1 normal
 exit-address-family
!
router ospf 1
 router-id 125.132.100.25
 redistribute bgp 200098 subnets
 network 125.132.100.24 0.0.0.7 area 0.0.0.1
 distribute-list OSPF_FILTER out
!
router bgp 200098
 bgp router-id 125.132.100.1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 2002:8400:1:1::205 remote-as 15557
 neighbor 2002:8400:1:1::205 description *** SESSION BGP 1 VERS SFR ***
 neighbor 2002:8400:1:1::205 ebgp-multihop 10
 neighbor 2002:8400:1:1::205 password 7 xxxxxxxx
 neighbor 119.24.138.5 remote-as 15557
 neighbor 119.24.138.5 description BGP-PEER1-SFR
 neighbor 119.24.138.5 ebgp-multihop 10
 neighbor 119.24.138.5 password 7 xxxxxxx
 !
 address-family ipv4
  aggregate-address 125.132.100.0 255.255.252.0 summary-only
  redistribute ospf 1
  neighbor 119.24.138.5 activate
  neighbor 119.24.138.5 advertise-map BGP_ADVERTISE exist-map BGP_CONDITION
  neighbor 119.24.138.5 soft-reconfiguration inbound
  neighbor 119.24.138.5 prefix-list DCN-IPV4->ISP out
 exit-address-family
 !
 address-family ipv6
  network 2006:DC80::/29
  neighbor 2002:8400:1:1::205 activate
  neighbor 2002:8400:1:1::205 soft-reconfiguration inbound
  neighbor 2002:8400:1:1::205 prefix-list DCN-IPV6->ISP out
 exit-address-family
!
ip forward-protocol nd
!
ip access-list standard CAP_SFR
 permit 125.132.100.54
ip access-list standard OSPF_FILTER
 deny   125.132.100.0 0.0.3.255
 permit any
!
!
ip prefix-list BGP_CONDITION seq 10 permit 125.132.100.16/30
!
ip prefix-list DCN-IPV4->ISP seq 5 permit 125.132.100.0/22
access-list 1 permit any
ipv6 route 2006:DC80::/29 GigabitEthernet0/0/2
!
!
ipv6 prefix-list DCN-IPV6->ISP seq 5 permit 2006:DC80::/29
route-map BGP_ADVERTISE permit 10
 match ip address prefix-list DCN-IPV4->ISP
!
route-map BGP_CONDITION permit 10
 match ip address prefix-list BGP_CONDITION
!
control-plane
!
end

6) RBGP2 Configuration
Code: [Select]
version 15.4
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
no platform punt-keepalive disable-kernel-core
!
hostname RBGP2
!
boot-start-marker
boot system flash bootflash:/asr1001x-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin
boot-end-marker
!
!
vrf definition Mgmt-intf
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
 exit-address-family
!
no aaa new-model
!
no ip domain lookup
ip domain name xxxxx
!
ipv6 unicast-routing
!
subscriber templating
multilink bundle-name authenticated
!
redundancy
 mode none
!
interface Loopback0
 no ip address
 ipv6 address 2006:DC80:0:200::1/128
!
interface Loopback1
 ip address 125.132.100.5 255.255.255.255
!
interface GigabitEthernet0/0/1
 description CONNEXION VERS OBS
 ip address 222.234.112.253 255.255.255.252
 negotiation auto
 ipv6 address 2001:C915:4000:300::1/64
!
interface GigabitEthernet0/0/3
 description Connexion vers SWDN12
 ip address 125.132.100.33 255.255.255.248
 ip ospf dead-interval 40
 ip ospf hello-interval 5
 negotiation auto
 ipv6 address 2006:DC80:0:1200::1/56
 ipv6 ospf 1 area 0.0.0.2
!
interface GigabitEthernet0
 vrf forwarding Mgmt-intf
 ip address 10.245.0.2 255.255.240.0
 negotiation auto
!
router ospfv3 1
 !
 address-family ipv6 unicast
  redistribute bgp 200098
  router-id 125.132.100.33
  area 0.0.0.2 normal
 exit-address-family
!
router ospf 1
 router-id 125.132.100.33
 redistribute bgp 200098 subnets
 network 125.132.100.32 0.0.0.7 area 0.0.0.2
 distribute-list OSPF_FILTER out
!
router bgp 200098
 bgp router-id 125.132.100.5
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 2001:C000:0:2010::1DF2:3 remote-as 3215
 neighbor 2001:C000:0:2010::1DF2:3 description *** SESSION BGP 1 VERS OBS ***
 neighbor 2001:C000:0:2010::1DF2:3 ebgp-multihop 64
 neighbor 2001:C000:0:2010::1DF2:3 password 7 xxxxxxxxx
 neighbor 2001:C000:0:2010::1DF2:4 remote-as 3215
 neighbor 2001:C000:0:2010::1DF2:4 description *** SESSION BGP 2 VERS OBS ***
 neighbor 2001:C000:0:2010::1DF2:4 ebgp-multihop 64
 neighbor 2001:C000:0:2010::1DF2:4 password 7 xxxxxxxxxx
 neighbor 183.253.157.241 remote-as 3215
 neighbor 183.253.157.241 description BGP-PEER1-OBS
 neighbor 183.253.157.241 ebgp-multihop 64
 neighbor 183.253.157.241 password 7 xxxxxxxxxx
 neighbor 183.253.157.242 remote-as 3215
 neighbor 183.253.157.242 description BGP-PEER2-OBS
 neighbor 183.253.157.242 ebgp-multihop 64
 neighbor 183.253.157.242 password 7 xxxxxxxxxx
 !
 address-family ipv4
  aggregate-address 125.132.100.0 255.255.252.0 summary-only
  redistribute ospf 1
  neighbor 183.253.157.241 activate
  neighbor 183.253.157.241 advertise-map BGP_ADVERTISE exist-map BGP_CONDITION
  neighbor 183.253.157.241 soft-reconfiguration inbound
  neighbor 183.253.157.241 prefix-list DCN-IPV4->ISP out
  neighbor 183.253.157.242 activate
  neighbor 183.253.157.242 advertise-map BGP_ADVERTISE exist-map BGP_CONDITION
  neighbor 183.253.157.242 soft-reconfiguration inbound
  neighbor 183.253.157.242 prefix-list DCN-IPV4->ISP out
 exit-address-family
 !
 address-family ipv6
  network 2006:DC80::/29
  neighbor 2001:C000:0:2010::1DF2:3 activate
  neighbor 2001:C000:0:2010::1DF2:3 soft-reconfiguration inbound
  neighbor 2001:C000:0:2010::1DF2:3 prefix-list DCN-IPV6->ISP out
  neighbor 2001:C000:0:2010::1DF2:4 activate
  neighbor 2001:C000:0:2010::1DF2:4 soft-reconfiguration inbound
  neighbor 2001:C000:0:2010::1DF2:4 prefix-list DCN-IPV6->ISP out
 exit-address-family
!
ip forward-protocol nd
!
ip route 183.253.157.241 255.255.255.255 222.234.112.254 name BGP-PEER1-OBS
ip route 183.253.157.242 255.255.255.255 222.234.112.254 name BGP-PEER2-OBS
!
ip access-list standard OSPF_FILTER
 deny   125.132.100.0 0.0.3.255
 permit any
!
ip prefix-list BGP_CONDITION seq 10 permit 125.132.100.16/30
!
ip prefix-list DCN-IPV4->ISP seq 5 permit 125.132.100.0/22
ipv6 route 2001:C000:0:2010::1DF2:3/128 2001:C915:4000:300::2 name BGP-PEER1-OBS
ipv6 route 2001:C000:0:2010::1DF2:4/128 2001:C915:4000:300::2 name BGP-PEER2-OBS
ipv6 route 2006:DC80::/29 GigabitEthernet0/0/3
!
!
ipv6 prefix-list DCN-IPV6->ISP seq 5 permit 2006:DC80::/29
route-map BGP_ADVERTISE permit 10
 match ip address prefix-list DCN-IPV4->ISP
!
route-map BGP_CONDITION permit 10
 match ip address prefix-list BGP_CONDITION
!
control-plane
!
end


7) Logs & Debug

At the moment of writing this post, i have no logs at the time of the last outage (clog and pfsense logs are not so easy to manipulate than classic syslog)
We are working to have a remote syslog server to catch all logs from Cisco and PFsense routers this weekend.
I post them here as soon as possible.

Offline quadrinary

  • Newbie
  • *
  • Posts: 23
  • Karma: +0/-0
    • View Profile
Interesting. In my scenario, I'm essentially scrapping the use of kernel routes entirely as well as abandoning interface-based gateways (this caused some really squirrly results when trying to use learned routes). I'd suggest starting by choosing to use Quagga for all routes (static and dynamic) and not use pfSense defined routes.

Offline Green IT

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Hi quadrinary,

Thanks for your reply.

Since my post, our firewalling solution is in master/slave mode without any outage.

Your reply seems to be a good idea. We will test in the last few nights and give you a reply.

Bertrand