Netgate SG-1000 microFirewall

Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - chrcoluk

Pages: [1] 2 3 4 5 ... 28
1
Traffic Shaping / Re: playing with fq_codel in 2.4
« on: January 26, 2018, 06:58:44 pm »
I agree with a new thread, I would also put findings by others as notes as well such as tuning the quantum size, I think someone mentioned using 300 is good for prioritising small packets?

This thread has a lot of pages, so its easy to miss stuff.

For reference I am still using the newer method, I have left pfsense unpatched and added the command in shellcmd so it applies every reboot. (still got the patch applied that enhances limiter diagnostics page on gui)

Doing a filter reload doesnt seem to break it so its fine for me.

It is possible I somehow mixed up the modules or something as I had added extra modules to add functionality, so doing a clean install of 2.4.2 to ensure both modules are synced will be done by me at some point, as right now I am still using the 2.4.0 modules.  Or I might install 2.4.2 elsewhere and just copy the modules from that across.

2
General Questions / Re: Ping spikes on new install
« on: January 25, 2018, 11:52:33 pm »
Also to add if my tip doesnt fix it then go back to defaults, what I offered is troubleshooting tips and could possibly be cause of problem, but dont set the changes if the defaults are ok.

3
Traffic Shaping / Re: playing with fq_codel in 2.4
« on: January 25, 2018, 11:46:52 pm »
I was using fairq+codel, it was doing a reasonable job but had perhaps about double the jitter I see now on thinkbroadband upload tests at same throughput.

4
Messages from the pfSense Team / Re: An update on Meltdown and Spectre
« on: January 25, 2018, 07:57:24 pm »
The sane thing to do is hold station I reckon

A good security approach is always layered means you never need to rely on one particular mitigation, and mitigation's which are unstable or bad performing can then be skipped over.

These mitigation's are not desirable with the performance and stability impacts been reported.

5
General Questions / Re: Ping spikes on new install
« on: January 25, 2018, 07:55:05 pm »
offloading is in the pfsense gui under advanced system settings.

you can disable hyperthreading in the loader.conf

add this to /boot/loader.conf.local and reboot

also Ill add lines to force your network over single threaded.  Note the igb line is only useful if your intel uses the igb driver, it may be using the em driver, which can be confirmed by running ifconfig.

net.isr.maxthreads=1
hw.igb.num_queues=1
machdep.hyperthreading_allowed="0"

I would say the LAN debugging takes priority over the WAN, the LAN spikes shouldnt be happening period, whilst WAN can be caused by all sorts of things outside your control.

6
Traffic Shaping / Re: playing with fq_codel in 2.4
« on: January 25, 2018, 07:48:24 pm »
now its working and I have had a period of time using it I can provide some feedback on its performance.

Affects on latency for uploads are better than ALTQ which matches my earlier experience.
Downloads seem similar performing to ALT+HFSC but the setup in terms of managing priorities etc, is greatly simplified for the similar result although I have done no recent steam testing.  The setup here is no priorities just the one downstream pipe for all traffic.

7
General Questions / Re: Ping spikes on new install
« on: January 25, 2018, 03:32:21 am »
disable any c-states on cpu, also if powerd is running either set it to performance mode or shut it down, this is so we can rule out power saving technologies misbehaving.
disable hyperhtreading if enabled and supported
consider disabling checksum offloading on nic, although as its intel I consider this unlikely to be the problem
make sure large receive offload and send equivelent is disabled, and promiscious mode.
google how to do it, but set the intel nic to a single queue mode only.  (loader.conf.custom).

This is to test and resolve the lan side spikes, ignore whats going on wan side for now.

8
Traffic Shaping / Re: playing with fq_codel in 2.4
« on: January 25, 2018, 03:01:20 am »
another quick update.

I copied ipfw.ko and dummynet.ko from 2.4.0 and it works properly.

I have q00001 and q00002 in ipfw sched show and the masks are present.

Code: [Select]
00001:  80.000 Mbit/s    0 ms burst 0
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 7ms interval 100ms quantum 2000 limit 10240 flows 2048 ECN
   Children flowsets: 1
00002:  20.000 Mbit/s    0 ms burst 0
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 7ms interval 100ms quantum 2000 limit 10240 flows 2048 ECN
   Children flowsets: 2

So I plan to reinstall pfsense from a new clean install image to see if the modules are ok that way, as seems I got a module issue somewhere, thats exhibited itself on this 2.4.2 installation. The internet no longer dies now with the limiters activated.

9
Traffic Shaping / Re: playing with fq_codel in 2.4
« on: January 25, 2018, 01:31:53 am »
ok tonight I will unpatch pfsense so it doesnt use the /root/rules.limiter.

I will just hit apply in the limiter config to apply, and then add the command to boot script same as john, and reboot for it to apply.

Then I will check to see if the mask appears on ipfw sched show.  (PF firewall rules will have no impact on that).

The masks have been checked I have lost count amount of times, they are no different to the guide posted by OP and what john has set.  It is an interesting observation its missing from ipfw sched show, but thats not down to a misconfiguration in the GUI.  Although I cannot rule out the patch been the culprit until I unpatch and retest which I will do tonight, but remember I said this is working fine on pfsense 2.4.0.

If you guys consider the OP's post wrong, then maybe a new thread should be made as I expect most people to follow the first post, not go several pages into something posted halfway thru :)

--edit--

I have done it just now running the command via the gui command as you suggested.

output of ipfw sched show

Code: [Select]
00001:  79.987 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 7ms interval 100ms quantum 2000 limit 10240 flows 2048 ECN
   Children flowsets: 1
00002:  20.000 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 7ms interval 100ms quantum 2000 limit 10240 flows 2048 ECN
   Children flowsets: 2

The gui config is no different to the screenshots I already posted, the only difference been the patch is disabled, and I ran the command provided to apply fq_codel.

attaching gui limiter config, I didnt post that before sorry, but the LAN rule is already posted. You can see is just cosmetic differences in the naming of the limiters, and the bandwidth limit.

I enabled logging of the outbound LAN rule and sure enough the logs verify the rule is correctly been hit.

Code: [Select]
Jan 25 08:07:51 LAN Default allow LAN to any rule (1513840726)   192.168.1.124:49240   80.249.103.8:443 TCP:S
Jan 25 08:07:51 LAN Default allow LAN to any rule (1513840726)   192.168.1.124:49239   80.249.103.8:443 TCP:S
Jan 25 08:07:51 LAN Default allow LAN to any rule (1513840726)   192.168.1.124:49238   80.249.103.8:443 TCP:S
Jan 25 08:07:51 LAN Default allow LAN to any rule (1513840726)   192.168.1.124:49237   80.249.103.8:443 TCP:S
Jan 25 08:07:51 LAN Default allow LAN to any rule (1513840726)   192.168.1.124:49236   80.249.103.8:443 TCP:S
Jan 25 08:07:51 LAN Default allow LAN to any rule (1513840726)   192.168.1.124:49235   80.249.103.8:443 TCP:S
Jan 25 08:07:50 LAN Default allow LAN to any rule (1513840726)   192.168.1.186:55112   129.70.132.34:123 UDP
Jan 25 08:07:49 LAN Default allow LAN to any rule (1513840726)   192.168.1.186:45591   89.238.136.135:123 UDP
Jan 25 08:07:49 LAN Default allow LAN to any rule (1513840726)   192.168.1.186:59559   194.1.151.226:123 UDP
Jan 25 08:07:49 LAN Default allow LAN to any rule (1513840726)   192.168.1.186:43958   85.199.214.99:123 UDP

From where I sit, with the facts in front of me.

Configuration looks good.
Works properly on 2.4.0
I have tested using john's method.

PfSense 2.4.2 is now closed kernel source (no public repo), so I cannot rule out any code changes they may have done breaking dummynet.

If I set the in/out pipe to none on the LAN outbound rule everything works again (albeit without the limiter processing the traffic).  If I had an issue with the LAN rules not processing the right traffic then that wouldnt be happening.

I appreciate your help of course, and its a very nice catch to notice the mask is missing from my schedulers, even tho it is configured, and does show on the queues.

Code: [Select]
ipfw queue show   
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000

I have just noticed another problem myself.

See in my ipfw queue show the 2 queues are q00001 and q00002

Yet in ipfw sched show the 2 queues are q65537 and q65538

In ipfw pipe show the 2 queues are q131072 and q131074

Now the OP has the same anomaly, but on john's post and your own post, you both dont, the queues on the ipfw sched show match the queues in ipfw queue show so q0001 and q00002

10
Traffic Shaping / Re: playing with fq_codel in 2.4
« on: January 22, 2018, 03:34:00 am »
probably wont be until saturday, but I will post I tried an install of 2.4.0, restored the config, and it functions correctly, 100% same configuration. Back to 2.4.2 and problem comes back.  (tested yesterday).

I can post limiter config now I guess as its in the GUI but the enable box is unticked.

I got 2 issues with what john posted.

1 - It only applies fq_codel at boot, it will get lost on a limiter reload.
2 - He posted some instructions that were not detailed, meaning I cannot be sure if I follow his setup I am doing it right.

I can tell you that the outbound LAN rule is been hit by the counters that are displayed and also by the fact when I edit the rule to stop using the pipes traffic works again, it wouldnt have that impact if there was another rule above it intercepting the traffic.  Not to mention I dont have any blocking outbound rules other than pfblockerNG used for dnsbl stuff.

What is missing on the ipfw sched show? I dont notice anything.

11
Traffic Shaping / Re: playing with fq_codel in 2.4
« on: January 20, 2018, 08:33:37 pm »
playing with this again

Code: [Select]
root@PFSENSE ~ # ipfw pipe show
00001:  17.987 Mbit/s    0 ms burst 0
q131073 50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
 sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002:   9.200 Mbit/s    0 ms burst 0
q131074  50 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail
 sched 65538 type FIFO flags 0x0 0 buckets 0 active
root@PFSENSE ~ # ipfw sched show
00001:  17.987 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
00002:   9.200 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
root@PFSENSE ~ # ipfw queue show
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
q00002  50 sl. 0 flows (16 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000

after routing on default lan outbound rule

Code: [Select]
root@PFSENSE ~ # ipfw pipe show
00001:  17.987 Mbit/s    0 ms burst 0
q131073 50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
 sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002:   9.200 Mbit/s    0 ms burst 0
q131074  50 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail
 sched 65538 type FIFO flags 0x0 0 buckets 0 active
root@PFSENSE ~ # ipfw sched show
00001:  17.987 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
00002:   9.200 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     109248  9956265 27 3497   9
root@PFSENSE ~ # ipfw queue show
q00001  50 sl. 0 flows (16 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
q00002  50 sl. 0 flows (16 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000

This rises rapidly even tho connection idle

Code: [Select]
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     109248  9956265 27 3497   9

console full of

fq_codel_enqueue over limit
and
fq_codel_enqueue maxidx = <random 3 digits usually between 400 and 525>

screenshot of relevant part of rule attaching to post

Finally if it helps, connectivity doesnt die if the traffic is only light, but any type of bulk download, streaming, speedtest will make "everything" timeout until rules are reloaded or ipfw is flushed or outbound rule is configured to not route to dummynet pipes.

12
Traffic Shaping / Re: playing with fq_codel in 2.4
« on: January 17, 2018, 06:32:04 pm »
1 - Not a script actually, just creating the /root/rules.limiter file and editing the shaper.inc code to use it.

As instructed by the OP of this thread.

2 - The pipe is configured in the LAN rules section (outgoing).

Configuration same as the screenshot you linked to in your previous post.

13
Traffic Shaping / Re: playing with fq_codel in 2.4
« on: January 17, 2018, 04:57:54 pm »
Mine only has one queue for each limiter, and is basically like the post you linked to except with different limits.

I of course have the script to override using fq_codel meaning the GUI setting stops controlling dummynet, but the GUI settings are the baseline with the only adjustment been the switch to fq_codel.

I have not had time yet to retest this so I still have no live ipfw pipe etc. outputs, I wish you would accept my word without me having to paste it all, but I am now glad someone has repeated the problem.

14
General Discussion / Re: *RANT* Why pfsense is popular
« on: December 23, 2017, 01:54:07 am »
other ideas (without commands sorry as I been up all night).best to try these step by step one at a time, and retest in each step.

Disable energy efficient ethernet.
Disable (at least temporarily checksum offloading on the NIC).
Reduce network queues to 1, this to make sure no packet ordering issues causing problems or driver bugs.
Disable TSO/RSO if enabled.
Disable interrupt moderation on NIC if enabled.
If powerd is enabled set to the performance mode or disable it.

Its unlikely to be a widescale pfsense issue, there would be many complaints if it was, its either a bug that only kicks in a specific scenario which you hitting or a compatibility issue whether it be hardware or isp config.

WOW has a known issue where if nagle is enabled (Delayed acks) it will show high lag because it uses tcp not udp for the game packets.  But pfsense as the router doesnt control nagle for client LAN devices, however just in case you can disable nagle on pfsense side via this shell command.

'sysctl net.inet.tcp.delayed_ack=0'

15
Installation and Upgrades / Re: After upgrade 502 Bad Gateway
« on: December 23, 2017, 01:21:20 am »
https://forum.pfsense.org/index.php?topic=110515.msg766964#msg766964

More information on what me and Martin worked on, I am hoping my patch which Martin submitted on github got accepted, but maybe it didnt, as the patch should resolve this issue for anyone using pfsense on hardware with at least 1 gig of ram.

--edit--

It has been approved 2 days ago.

https://github.com/pfsense/pfsense/pull/3881
https://redmine.pfsense.org/issues/8125

Pages: [1] 2 3 4 5 ... 28