Netgate Store

Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Messages - jtl

Pages: [1] 2 3

Trying out a HA setup with two machines running pfSense 2.4.3. The primary is a Supermicro with a Intel em driver NIC and the secondary is a APU1d4 which has Realtek re driver NIC(s).

Not even going to read any further because that is a completely unsupported configuration. HA nodes must match. If you want to test HA, use VMs.

OK. I've reproduced the same issue with "matching" VM hardware (Proxmox KVM). Just disregard the first sentence of my first post if it makes you happy.

Still running pfSense 2.4.3. I have CARP VIPs on both the "WAN" and "LAN" interfaces, which are just VLAN's on my core switch. I have a server of mine on the LAN VLAN running an iperf3 server to simulate LAN->WAN throughput and vice versa, and my desktop on the WAN VLAN as the iperf3 client.

I suck at diagrams, so here's a brief description of my network topology.

Code: [Select]
Test network WAN -
Test network LAN -
PFSYNC/Management network -
pfsense-dev-master WAN - (VIP
pfsense-dev-master LAN - (VIP
pfsense-dev-slave WAN - (VIP
pfsense-dev-slave LAN - (VIP

jtl-desktop - (for testing LAN->WAN and vice versa using iperf)
angrybear (server) - (port forwarded 5201 TCP/UDP to WAN VIP)

Still have the same issue when I'm testing LAN->WAN bandwidth, which is traffic shaped, and I failover to the secondary, the shaped traffic ends up in the root interface queue.

I made a video here as well

Grumble grumble. Maybe on the weekend I'll attempt the same with a VM lab on my desktop and report back


Trying out a HA setup with two machines running pfSense 2.4.3. The primary is a Supermicro with a Intel em driver NIC and the secondary is a APU1d4 which has Realtek re driver NIC(s).

Before you say "pfsync doesn't work with differing interface names". That's true but I discovered you can use <earlyshellcmd> to rename the interfaces early enough in the boot process to not cause other issues and it's persistent within the running system (ie, re1 (WAN) on the secondary is renamed to em3 (WAN) of the primary).

Because I only have a DHCP WAN connection I'm working on a prototype of a concept I call "bettercarp" where I have my WAN connection terminated into a managed switch and the primary and secondary shutdown the other WAN port by SNMP upon failover to avoid any MAC address conflicts, all the LAN and VLAN interfaces are CARP VIP's. This seems to work fairly well after sorting out issues with devd.

If I get this fully working I'm planning on upgrading my internet connection to 5 usable IP's. Sadly they are still DHCP assigned but then I could have a second WAN interface on the primary and secondary so I could do updates without needing to failover the connection.

An issue I haven't been able to figure out is that I use HFSC ALTQ traffic shaping on my WAN to give priority of my LAN over my DMZ network. Upon failover to the secondary all traffic from connections that was assigned to subqueues ends up in the interfaces root queue. Because pfsync otherwise works, no connections are dropped. To rule out this being a issue with my WAN "fencing" setup I tested limiting the bandwidth of LAN to DMZ (both VIP interfaces) using HFSC upper limit to 10Mb and ran an iperf from my workstation (LAN) to a host on my DMZ network.

Upon CARP failover to the secondary the traffic ended up in the root interface queue and upon failover back to the primary the traffic was in the correct queue again. If I reboot the primary while the secondary is master and it fails over to the primary, the connection won't be dropped but the queue for the connection would be lost.

(at 14-15 seconds) I failover to the secondary with CARP maintenance mode and I disabled the CARP maintenance mode soon after, and it moves back the primary at 40-41 seconds with the queues still intact.


Development / Where to find source code of pfSense DHCP and DHCPv6 cleints
« on: February 01, 2018, 01:51:03 am »

I'm interested in the source code for the IPv4 and IPv6 DHCP clients (dhclient, dhcp6c) as used in pfSense for working on some changes.

Where can I obtain the source code for both?


OpenVPN / Re: Cannot Access WebGUI over OpenVPN.
« on: January 30, 2018, 07:20:53 pm »
I fixed the issue.

From memory I had to create a BRIDGE interface between my MGMT VLAN interface and OpenVPN TAP interface and remove the assigned IP from the MGMT VLAN interface and assign it to the BRIDGE interface.

I now use a tun routed setup though.

DHCP and DNS / Possible to supersede prefix lifetime with dhcp6c?
« on: January 12, 2018, 05:55:27 pm »
I have a TELUS FTTH connection. I then terminate the Ethernet handoff from the ONT into my own switch as untagged VLAN 666, of which two other ports are untagged VLAN 666, one going to the Actiontec crappy router for IPTV boxes and another going to my pfSense router.

Problem is, it appears sometimes Telus does maintenance or something at night with the DHCP server at least once a month in my experience, which causes the IP source guard/Dynamic ACL binding at their edge switch to stop routing the IP address given to my pfSense router. Sometimes the connection comes back after 10 minutes or so (happened once when I was away from home) but it often doesn't come back until the DHCP lease is renewed which can take up to 2 hours (lease time is 4 hours). dpinger shows 100% loss for both the IPv4 and IPv6 interface.

If I manually release and renew the DHCP lease under Status->Interfaces the connection comes back instantly.

Similar to the issue in this forum thread, but last time it happened on December 25th 2017 at 21:29 PST I still had the same IPv4 and v6 IP upon renewal:

An idea I had to fix this problem is set the DHCP renewal time to a short value regardless of what the server sets. It's possible to do this for IPv4 with dhclient by adding
Code: [Select]
supersede dhcp-lease-time 1800; to the options. But I want to synchronize the DHCP renewal life time with both DHCP and DHCP6 to ensure a seamless and reliable connection as I host servers from home (And no, upgrading to a business connection wouldn't help as they still use DHCP, just registering the MAC address of your router in some clunky web UI)

I tried setting a manual prefix lifetime in a custom DHCP6 config file, but it just seems to be overridden by the sent server value.

Code: [Select]
id-assoc pd 0 {
prefix ::/56 1800 1800;
prefix-interface vtnet1 {

I tried looking for the source of dhcp6c on pfSense github so I can get at hacking it, but either I'm blind or it's not there.


Figured it out to satisfaction. I will generalize steps below.

Traffic Shaper->Create WAN shaper type CBQ with ~95% of WAN upload bandwidth
Create WAN_OUT queue, priority 1, set as default queue and allow borrowing from other queues
Create DMZNET_OUT queue, priority 2, set as required, allow borrowing from other queues.

On the DMZNET out rule (for IPSec tunnel) edit the rule, go to advanced and set DMZNET_OUT as the queue.

Reset states.

Test by doing various iperf3 tests and watching queue status

Traffic Shaping / Shaping upload of DMZ network to give priority to LAN.
« on: November 25, 2017, 07:26:53 pm »

First I should explain some things.

Interfaces concerned are LAN, WAN, and DMZ interface (hereby referred to as DMZNET). Now DMZNET is a VLAN interface I use for hosting publicly available services from my server. Firewall rules are used to prevent hosts on the DMZ network from connecting out to other hosts on my LAN(s) and hosts on other networks are allowed to connect in. Due to ISP shenanigans I use an IPSec tunnel to a datacenter terminated on one of my servers connected to DMZNET (not my main router)

[REDACTED] is the datacenter host that IPSec tunnel terminates to.

I want to shape my WAN upload, so traffic from LAN->WAN gets priority over DMZNET->WAN traffic, and so LAN can borrow from the DMZNET queue when needed. I have a symmetrical connection and my ISP applies traffic shaping of their own in the download direction for their IPTV service so that's not as needed right now.

I don't need to shape individual applications to how the traffic wizard does it, just need to give outgoing LAN traffic priority over DMZNET.


Hardware / Re: Ryzen 3 Restarts under Load
« on: September 03, 2017, 02:26:32 pm »
I'd run Prime95 from Linux for 24 hours (if you can)

Good luck

Packages / Re: BIND creating forward zone has an empty resulting zone config
« on: September 02, 2017, 11:08:54 pm »
Also forgot to mention. I was running 2.3.4 so I decided to upgrade to 2.3.4-p1 and no change. It did update BIND though.

Packages / BIND creating forward zone has an empty resulting zone config
« on: September 01, 2017, 10:10:11 pm »

(DNS name of my internal company has been changed to protect the innocent, etc.)

I'm trying to setup a forward zone in BIND [for] so I can have certain records point to internal IP's on my LAN and the rest would go out to the internet to the public nameservers.

The problem I'm having is after I fill in all the values and save, the resulting zone config doesn't get created and thus this doesn't work.

I apologize if full-page screenshots aren't the best way to show my configuration, but here goes.

(With regards to the views. I already have different BIND views created that correspond with different VLAN's whose internal DNS hosts I want to keep separate from each other) 


Hardware / Re: Hardware for dedicated hypervisor running only pfSense
« on: August 09, 2017, 05:10:44 pm »
Why dedicated hypervisor running only pfSense?

I already have another hypervisor running my other projects.

I understand not all motherboards do PCIe passthrough well. Does anyone have experience with this?

As long as your CPU and motherboard supports VT-d, you're good.

Really? I thought it needed IOMMU support, which some people have had trouble with.

If this is your only concern, 2.4 is a better choice as it supports ZFS.

Config backups and restore is a great way to get back online after bad configuration. You can always restore recent config from the console (option 15). These are automatically made every time you make a change within the GUI. Because of that, I believe you may be overthinking it with virtualization :)

Hmm, maybe. I have a friend that does a virtualized setup so he can easily test multiple pfSenses snapshots and that like. I also might be doing some custom modifications to pfSense so I would like having separate installs under a hypervisor as well.

Most x86 hardware except for super embedded platforms supports virtualization as I can gather, just concerned about PCIe passthrough.


Hardware / Hardware for dedicated hypervisor running only pfSense
« on: August 06, 2017, 10:36:37 pm »

Thinking of upgrading my old circa 2010 core i3 pfSense box (I built it only recently but most of the parts were "free")

Reasons to upgrade:
a) AES-NI (for VPN and similar, even this old system can do over 100 mbps using OpenVPN though)
b) The motherboard I'm using (Intel DH55HC) only seems to like one particular stick of Corsair DDR3 1333 memory I "stole" from my current desktop, this is obviously a problem as now my desktop has only 1X4GB, instead of 2X4GB of memory.
c) Although I've tested with iperf3 and I can do a single connection and get 940mbps throughput WAN->LAN with local testing hardware, but not get that sum with multiple connections, although I only have 150/150 internet right now so it doesn't  matter. Just planning for the future as my ISP might have a gigabit plan eventually, and/or I might move to another location that has symmetrical gigabit available.

My current system is built in a Rosewill 4U case with a Noctua heatsink and fan. I'm thinking of "downgrading" to a 2U case and heatsink to save rack space, as I only have 1 boot SSD in there. So I would need a Micro ATX motherboard.

I'm thinking of getting a Sky/Kabylake Core i3 and getting a motherboard with VT-d so I can run pfSense in a hypervisor (Proxmox or ESXi) with my existing 4-port HP branded server NIC with PCIe passthrough for the pfSense VM. This is so I can take snapshots of my working setup and do testing easily without having to take the system down and reinstall, etc. I understand not all motherboards do PCIe passthrough well. Does anyone have experience with this?

I live in Canada, and only need suggestions for the CPU and motherboard, should I wait for AMD's Ryzen based APU's or go with an i3?


Packages / Re: BIND override A record possible?
« on: July 25, 2017, 08:02:07 pm »
Fair enough.

Thanks for the help.

I could have sworn I've seen such functionality (replace a matched returned 'A' record with another where the FQDN is unknown) in some "penetration testing" MiTM DNS server tools, but I take it that it's not possible with any "mainstream" DNS server (Unbound, BiND, etc.)

Pages: [1] 2 3