pfSense Support Subscription

Author Topic: Need a way for stateless rules  (Read 228 times)

0 Members and 1 Guest are viewing this topic.

Offline ksteinb

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Need a way for stateless rules
« on: November 27, 2017, 04:44:39 am »
Hi,

we do have some problems with NFS via pfsense Firewalls
(no it is not possible for us to avoid NFS over firewalls, our infrastructure is too much spread over a large campus)

One of the  problems is that after a server crash NFS connections hung, because the NFS client is trying to reconnect from same socket (the SYN's are then dropped because there is already an open state).

So we hoped to avoid that with stateless rules, but we found no way to get them running.

We tryed both interface rules, as well as floating rules (with quick of course), but always the packets go out on the WAN side, the answer came back and come in on WAN (proved by tcpdump), but the answer never go out on the inside interface.

So how can we setup "stateless" rules on pfsense?

Offline johnpoz

  • Hero Member
  • *****
  • Posts: 14468
  • Karma: +1341/-200
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Need a way for stateless rules
« Reply #1 on: November 27, 2017, 04:59:31 am »
On the rule just go to advanced and pick the NONE for the state type.. When you say wan.. Why would there be traffic between your campus and WAN interfaces on your pfsense?  Do you have multiple pfsense with some of them being downstream of others?  Have you disabled NAT for these local networks?
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.3.4_p1 (work)
1x SG-4860 2.4.2-RELEASE-p1 (home)

Offline ksteinb

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Need a way for stateless rules
« Reply #2 on: November 28, 2017, 05:48:03 am »
WAN is not really WAN, in our case "WAN" is the interface to the campus backbone.

"NONE" for the state type just didn't help, what we see by tracing is the following:

-> packet goes in on the internal interface
-> goes out on so called WAN interface
-> reply packet goes in on so called WAN interface
-> never goes out on inside

We tried the following things:

-> interface rule with state type "NONE" on the inside interface
-> floating rule with state type "NONE" bound to inside and "WAN"

always the same.

the rules looks like this:

pass in quick on ix1.2474 inet proto tcp from 10.153.133.1 to 10.153.15.14 port = nfsd flags S/SA no state label "USER_RULE"


the complete rule set for ix1.2474:

2.4.1-RELEASE][root@sal-sv-fw02.server.physik.uni-muenchen.de]/root: pfctl -sr | grep ix1.2474
block drop in log on ! ix1.2474 inet6 from 2001:4ca0:4103:406::/64 to any
block drop in log on ix1.2474 inet6 from fe80::a236:9fff:fe40:97b2 to any
block drop in log on ! ix1.2474 inet from 10.153.133.0/24 to any
pass in quick on ix1.2474 inet proto udp from any port = bootpc to 255.255.255.255 port = bootps keep state label "allow access to DHCP relay"
pass quick on ix1.2474 inet6 proto udp from fe80::/10 to fe80::/10 port = dhcpv6-client keep state label "allow access to DHCPv6 server"
pass quick on ix1.2474 inet6 proto udp from fe80::/10 to ff02::/16 port = dhcpv6-client keep state label "allow access to DHCPv6 server"
pass quick on ix1.2474 inet6 proto udp from fe80::/10 to ff02::/16 port = dhcpv6-server keep state label "allow access to DHCPv6 server"
pass quick on ix1.2474 inet6 proto udp from ff02::/16 to fe80::/10 port = dhcpv6-server keep state label "allow access to DHCPv6 server"
pass in quick on ix1.2474 inet6 proto udp from fe80::/10 to 2001:4ca0:4103:406::1:2 port = dhcpv6-client keep state label "allow access to DHCPv6 server"
pass out quick on ix1.2474 inet6 proto udp from 2001:4ca0:4103:406::1:2 port = dhcpv6-server to fe80::/10 keep state label "allow access to DHCPv6 server"
pass log quick on ix1.2474 inet proto tcp from any to ! <DMZ_FAKULTAET> port = smtp flags S/SA keep state label "USER_RULE"
pass in quick on ix1.2474 inet proto tcp from 10.153.133.1 to 10.153.15.14 port = nfsd flags S/SA no state label "USER_RULE"
pass in quick on ix1.2474 inet from 10.153.133.0/24 to any flags S/SA keep state label "USER_RULE"
pass in quick on ix1.2474 inet6 from 2001:4ca0:4103:406::/64 to any flags S/SA keep state label "USER_RULE"

(from inside to out we normally allow anything)

i see thean a state created on outside:

WAN    tcp    10.153.133.1:39944 -> 10.153.15.14:2049    ESTABLISHED:SYN_SENT    3 / 4    180 B / 240 B

wow -> something is not really stateless



now I try it with a floating rule:

[2.4.1-RELEASE][root@sal-sv-fw02.server.physik.uni-muenchen.de]/root: pfctl -sr | grep nfsd
pass in quick on ix0 inet proto tcp from 10.153.133.1 to 10.153.15.14 port = nfsd no state label "USER_RULE"
pass in quick on ix1.2474 inet proto tcp from 10.153.133.1 to 10.153.15.14 port = nfsd no state label "USER_RULE"

again a state is created:


WAN    tcp    10.153.133.1:39974 -> 10.153.15.14:2049    ESTABLISHED:SYN_SENT    2 / 4    120 B / 240 B

(WAN == ix0,  ix1.2474 is the inside interface)

Offline johnpoz

  • Hero Member
  • *****
  • Posts: 14468
  • Karma: +1341/-200
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Need a way for stateless rules
« Reply #3 on: November 28, 2017, 07:25:16 am »
"WAN is not really WAN, in our case "WAN" is the interface to the campus backbone."

So you disabled nat? Why would you nat at pfsense if pfsense is just a downstream router to your larger campus network?  So wan is a transit network, or are there hosts on it that use some other gateway other than pfsense wan IP?

Also if your going to move nfs across pfsense - did you disable the PF scrubbing in the advanced section?

"Disables the PF scrubbing option which can sometimes interfere with NFS traffic."
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.3.4_p1 (work)
1x SG-4860 2.4.2-RELEASE-p1 (home)

Offline ksteinb

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Need a way for stateless rules
« Reply #4 on: November 28, 2017, 07:56:24 am »
Yes you can say it is a transit network. But we NAT in some cases, for private networks going outside the Campus network, (and only going out of the campus) because the Campus wide NAT is sometimes overburdened, sometimes in the google blacklist and tends to jump in with its security measures on video conferencing.

But back to the NFS problem:

This networks will not be NATed, they are just both inside the campus network but on different locations (up to 30 kilometers away). An yes we disable PF scrubbing.

Most times NFS works, we just run into trouble with hung NFS Clients on Server failovers (planned and sometimes unplanned)

So why does the stateless rule not work?

Offline ksteinb

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Need a way for stateless rules
« Reply #5 on: November 30, 2017, 12:12:09 am »
No idea why stateless does not work ?

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9244
  • Karma: +1052/-308
    • View Profile
Re: Need a way for stateless rules
« Reply #6 on: November 30, 2017, 12:24:38 am »
Quote
pass in quick on ix0 inet proto tcp from 10.153.133.1 to 10.153.15.14 port = nfsd no state label "USER_RULE"
pass in quick on ix1.2474 inet proto tcp from 10.153.133.1 to 10.153.15.14 port = nfsd no state label "USER_RULE"

Not sure why you have that set to pass traffic with the same source and destination on both interfaces. Seems it should be one way on one interface and the reciprocal on the other. You probably also need to figure out the port. I'd have to look up the NFS protocol to know.

Might try setting it to look like this:

pass in quick on ix0 inet proto tcp from 10.153.133.14 port = nfsd to 10.153.15.14 no state label "USER_RULE"
pass in quick on ix1.2474 inet proto tcp from 10.153.133.1 to 10.153.15.14 port = nfsd no state label "USER_RULE"

If it's a TCP connection it seems like you're barking up the wrong tree. A regular pass rule on ix1.2474 to 10.153.15.14 port nfsd should work fine.

But NFS is a strange beast.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline johnpoz

  • Hero Member
  • *****
  • Posts: 14468
  • Karma: +1341/-200
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Need a way for stateless rules
« Reply #7 on: November 30, 2017, 04:00:16 am »
NFS is normally UDP... Are you sure your using TCP?  Yeah I know NFS can use tcp, but doesn't it default to udp?
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.3.4_p1 (work)
1x SG-4860 2.4.2-RELEASE-p1 (home)

Offline ksteinb

  • Newbie
  • *
  • Posts: 6
  • Karma: +0/-0
    • View Profile
Re: Need a way for stateless rules
« Reply #8 on: November 30, 2017, 10:22:20 am »
1) the two rules are because I tried to bind the floating rule to both the inside and the outside net, because I hoped that this helps.
    I tried it also diffently.

2) nfs is Port 2049 (especially with NFSV4 anything is running over Port 2049)

3) the seond poster:  Yes NFS is TCP (UDP may work but is not recommended since long long times, and nobody uses UDP with NFS nowadays)

4) A stateful rule with NFS works as long as nothing goes wrong with the server.

Why do we need the stateless rule (or some other method to avoid dropped SYN's):
 
-> if a server fails ( broken failover on a Highavailble Server, crashed single server, or an overloaded server because of too many clients
                            on too few NFSD threads)
           we get clients which try to reconnect, but as NFS is a beast, they try for some reason from the _same_ client socket, as before, so
           pfsense believes that the connection is already open (state ESTABLISHED), and drops the newly coming in SYN Packets.

          The client will never be able to reconnect to the server until somebody deletes the state at the pfsense Firewall

So we hoped to get around this (and maybe better throughput because of avoiding the state engine) by stateless rulles.

But for reasons I do not understand "state=NONE" does not work. So I'm in need of help.

Sincerly,
Klaus