Howdy! First - THANK GOD FOR BOOT ENVIRONMENTS!!
My system is a whitebox (SuperMicro) running pfSense Plus (initially 23.09.1), and has been rock stable for years (on this and previous releases). I finally got around to upgrade to 24.03 this afternoon, and the update finished just fine. I was using an IPMI connection to monitor the update/reboot and after it came up, everything looked just fine, BUT, I couldn't ping my LAN address, and none of my devices could actually access the internet over that interface. Other systems on other interfaces/VLANs were fine, but my LAN is a 25G connection between a Unifi Switch and the SuperMicro pfSense, and while the link was up, and everything LOOKED ok, no traffic would pass.
I left it that way for about 30 minutes, figuring it was just all my packages reinstalling, etc. When it still didn't work, I rebooted from the console, and I still couldn't ping the LAN interface, even though the command line says it was UP. pfSense also couldn't ping OUT from that interface either.
I reverted to 23.09.1, and interestingly enough, it STILL didn't come back. Thinking I was going crazy, i rebooted again... And then everything came up fine... Investigation started...
I let it stabilize for another 30 min, working perfectly fine. Rebooted into the 24.03 Boot Environment, successful boot, waited for another 10-15 min... Still no LAN interface. Reboot again into 24.03, wait 10-15... No traffic.
Rebooted into 23.09.1, booted successfully, no traffic on LAN... Wait 5 min, reboot again, into 23.09.1, traffic is back to normal...
I did this whole iteration twice to see if it was repeatable. It is.
So... Something is messing with my interface in 24.03 such that it requires 2 reboots in 23.09.1 to resolve the problem.
So I'm back in 23.09.1, hoping someone here knows what MIGHT be going on.
Thanks!!
Device Specifics:
System: Supermicro SYS-E300-8D
Problem NIC:
ixl0@pci0:7:0:0: class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x158b subvendor=0x8086 subdevice=0x0002
vendor = 'Intel Corporation'
device = 'Ethernet Controller XXV710 for 25GbE SFP28'
class = network
subclass = ethernet
bar [10] = type Prefetchable Memory, range 64, base 0xf7000000, size 16777216, enabled
bar [1c] = type Prefetchable Memory, range 64, base 0xf8808000, size 32768, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 129 messages, enabled
Table in map 0x1c[0x0], PBA in map 0x1c[0x1000]
cap 10[a0] = PCI-Express 2 endpoint max data 256(2048) FLR
max read 4096
link x8(x8) speed 8.0(8.0) ASPM disabled(L1)
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 103babfffffefd3c
ecap 000e[150] = ARI 1
ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
0 VFs configured out of 64 supported
First VF RID Offset 0x0110, VF RID Stride 0x0001
VF Device ID 0x154c
Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
ecap 0017[1a0] = TPH Requester 1
ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable
P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable
P2P Upstream Forwarding unavailable, P2P Egress Control unavailable
P2P Direct Translated unavailable, Enhanced Capability unavailable
ecap 0019[1d0] = PCIe Sec 1 lane errors 0
PCI-e errors = Correctable Error Detected
Unsupported Request Detected