Netgate SG-1000 microFirewall

Author Topic: pfSense freezing with CBQ-shapers  (Read 2327 times)

0 Members and 1 Guest are viewing this topic.

Offline Birke

  • Jr. Member
  • **
  • Posts: 76
  • Karma: +13/-0
    • View Profile
pfSense freezing with CBQ-shapers
« on: April 20, 2017, 04:13:27 am »
Hi,

we have a problem with our pfSense: it freezes (no ping and no reaction on console) after a while or when we change something on some interfaces. It is the first of 4 pfSenses we want to go live with.

We use HP Proliant DL20 Gen9 servers with 2*4port network cards (means each server has 10 network ports).
The two 4 port network cards are HPE 366T and 366FLR with Intel I350 chips (means the igb-drivers are used). We use the tweaks mentioned here (with and without  msi and msix enabled).
The two onboard ports use a 332i chip (means bge-drivers are used), but they aren't used in pfSense.
The network setup is (all with fixed ip):
igb3 = wan
igb4 = lan1
igb5 = lan2
igb6 = lan3
igb7 = lan4
vlan3 on igb4 = lan5
vlan4 on igb4 = lan6
bge0 = HP ILO (not used in pfSense)
All other ports are not used atm.

We tested with pfSense version 2.3.2, 2.3.3, 2.3.3_1 and 2.4beta (all amd64). We also tested with and without additional packages (Cron, mailreport, pfBlockerNG, snort).

Our pf-setup includes
- CBQ-traffic shapers for each interface
- IPsec-vpns between the pfSenses and ZyXel ZyWalls USG300, IKE V1
- virtual ips on wan1 and lan1
- dhcp server on all lan interfaces
- dns resolver for all lan interfaces
- snort enabled without blocking on all interfaces

On our first try, it freezed after 2 days. And after that it keept crashing/freezing whenever we tried to use them as our live-gateway. It didn't even matter if the vpns or the internet connection were active or if we did anything on the firewall.

On our last go-live-try two weeks ago we had crashes when we changed IPs or other things on the interfaces (i attached 2 crashlogs).
After reading some threads about problems with CBQ-shapers, i deleted the shapers and made new ones (without wizard). After that, no more crashes but pfSense still freezes and the server needs a hard restart. And since that is not a crash, there are no newer crashlogs.

As long as the 4 pfSenses run in our test-lab they run stable and are connected to each other with an IPsec IKE v2 VPN. Routing and everything works without problem, only when we make them the live-firewall again (by switching the ips to the network gateways and enabling the dhcp server) it crashes/freezes again.

Maybe someone can see in the crashlog what happened or has some ideas how to avoid the freezes.

Offline Birke

  • Jr. Member
  • **
  • Posts: 76
  • Karma: +13/-0
    • View Profile
Re: pfSense freezing with CBQ-shapers
« Reply #1 on: April 27, 2017, 03:26:49 am »
Seems we found the problem:
whenever we set a traffic shaper on the vlan-interfaces (even if its only a default queue) the freezing/crashes happen. It doesnt even matter if there is traffic on that vlan.
We deleted the shapers on these 2 interfaces and so far no freeze or crash happened. Lets hope, it stays that way :)

Btw: we moved from CBQ to HSFC and the freezes and crashes still happened.

Offline moikerz

  • Full Member
  • ***
  • Posts: 139
  • Karma: +7/-0
    • View Profile
Re: pfSense freezing with CBQ-shapers
« Reply #2 on: April 27, 2017, 05:06:34 pm »
I had similar issues on an SG-1000 with HFSC applied to a fresh install using 2.4 firmware. Using the console reported a kernel panic and a reboot of the device shortly thereafter, occurring in a cycle/loop. Would be interested in a solution to this too..

Offline moscato359

  • Jr. Member
  • **
  • Posts: 91
  • Karma: +10/-6
    • View Profile
Re: pfSense freezing with CBQ-shapers
« Reply #3 on: April 28, 2017, 08:36:59 am »
Do you get crashes with dummynet limiters?

Offline Birke

  • Jr. Member
  • **
  • Posts: 76
  • Karma: +13/-0
    • View Profile
Re: pfSense freezing with CBQ-shapers
« Reply #4 on: April 28, 2017, 08:59:52 am »
We don't use limiters atm, but when i had two limiters active, there were no problems with them. But the limiters were only active on the "normal" interface igb4, not on a vlan.

We are using the pfSense atm without shaper/limiter on the vlans and it still works without problems.
I think we will keep it this way as long as the users on the vlans don't generate too much traffic. If the traffic gets too high i will try with limiters.

Offline moscato359

  • Jr. Member
  • **
  • Posts: 91
  • Karma: +10/-6
    • View Profile
Re: pfSense freezing with CBQ-shapers
« Reply #5 on: April 28, 2017, 10:45:01 am »
I have *also* had a box freeze up when adjusting altq, on 2.3.3

There clearly has to be some kind of bug here

Offline Nullity

  • Hero Member
  • *****
  • Posts: 977
  • Karma: +97/-9
    • View Profile
Re: pfSense freezing with CBQ-shapers
« Reply #6 on: April 28, 2017, 01:38:40 pm »
I have *also* had a box freeze up when adjusting altq, on 2.3.3

There clearly has to be some kind of bug here

Back when I was frequently tweaking my queues (~2.2.x or 2.1.x ?) I'd occasionally have a freeze. I think it happened when enabling or disabling ALTQ on an interface. I forget whether the entire system froze or if it was just the GUI. I think it was the whole system, which froze for 2-3 minutes or sometimes forever.

Maybe it was fixed? Dunno.
Please correct any obvious misinformation in my posts.
-Not a professional; an arrogant ignoramous.

Offline Chrismallia

  • Full Member
  • ***
  • Posts: 279
  • Karma: +23/-4
    • View Profile
Re: pfSense freezing with CBQ-shapers
« Reply #7 on: May 01, 2017, 11:45:56 am »
Same problem here enabled  codel on vlan and had to re install pfsense as it froze and never came out of it. 

Offline Nullity

  • Hero Member
  • *****
  • Posts: 977
  • Karma: +97/-9
    • View Profile
Re: pfSense freezing with CBQ-shapers
« Reply #8 on: May 01, 2017, 01:18:34 pm »
Same problem here enabled  codel on vlan and had to re install pfsense as it froze and never came out of it.

In my case a restart was the the most impactful thing I had to do. No re-install needed.
Please correct any obvious misinformation in my posts.
-Not a professional; an arrogant ignoramous.

Offline Chrismallia

  • Full Member
  • ***
  • Posts: 279
  • Karma: +23/-4
    • View Profile
Re: pfSense freezing with CBQ-shapers
« Reply #9 on: May 02, 2017, 12:40:59 am »
Rebooting did not work for me had no choice looking at  console  it kept freezing while booting. Also setting up  traffic shaper wizard sometimes freezes pfsense but that  is solved with a reboot

Offline Birke

  • Jr. Member
  • **
  • Posts: 76
  • Karma: +13/-0
    • View Profile
Re: pfSense freezing with CBQ-shapers
« Reply #10 on: May 02, 2017, 07:03:44 am »
Rebooting did not work for me had no choice looking at  console  it kept freezing while booting. Also setting up  traffic shaper wizard sometimes freezes pfsense but that  is solved with a reboot
had that problem too: just unplug the interface with the vlans and it booted again. if the vlan-interface is the lan-interface: then unplug it, let it start, replug it and be fast with deactivating the shapers


i was also able to reproduce the error on a new installed vmware:
i gave the vm 3 adapters, 1 wan, 1 lan and 1 opt. i set up vlans and shapers on lan and opt and i could let it crash on every ip-change.

Offline moscato359

  • Jr. Member
  • **
  • Posts: 91
  • Karma: +10/-6
    • View Profile
Re: pfSense freezing with CBQ-shapers
« Reply #11 on: May 03, 2017, 10:42:03 am »
The real question is why does the thing freeze at all.

It's happened to me too.

I run a business network with 100+ users, so a firewall reboot is a bit of a pain. Have like 20 people come to me and be like WHYYY DID MY STUFF GO DOWN, WAAH

It's not fun.

I'm actually afraid of updating my QoS settings.

I set fairq, with codel, and stopped.

Offline Chrismallia

  • Full Member
  • ***
  • Posts: 279
  • Karma: +23/-4
    • View Profile
Re: pfSense freezing with CBQ-shapers
« Reply #12 on: May 04, 2017, 02:21:48 am »
The real question is why does the thing freeze at all.

It's happened to me too.

I run a business network with 100+ users, so a firewall reboot is a bit of a pain. Have like 20 people come to me and be like WHYYY DID MY STUFF GO DOWN, WAAH

It's not fun.

I'm actually afraid of updating my QoS settings.

I set fairq, with codel, and stopped.

I hear you. I ended up swapping pfsense at a location with a different product that have also added fq_codel  and what can I say clients  are much  happier  no freezing no things that breake  much better reporting and as I say the clients them selves felt a better quality connection, sorry for saying this but this is true for me, infacct clients are opening a new location and want this new firewall in the second location

Offline Birke

  • Jr. Member
  • **
  • Posts: 76
  • Karma: +13/-0
    • View Profile
Re: pfSense freezing with CBQ-shapers
« Reply #13 on: May 04, 2017, 02:29:39 am »
just tried around a little bit and found out that it seems codel is my problem:
if i only activate red and ecn on the vlan-interfaces there are at least no instant freezes/crashes anymore when i change the ips.

maybe you can try that too. that way we could narrow down the reason for the freezes/crashes.

Offline moscato359

  • Jr. Member
  • **
  • Posts: 91
  • Karma: +10/-6
    • View Profile
Re: pfSense freezing with CBQ-shapers
« Reply #14 on: May 04, 2017, 08:19:47 am »
I can set codel sometimes, and have zero issues. I can also set hfsc sometimes and just have a total lockup.

I've also had a crash setting ipv6 before.

If I don't touch firewall or interface settings, the firewall runs forever without issues.