@ kejianshi You are a special kind of stupid. Its morons like you that turn ppl off to new ideas you are nothing more than a troll in the truest sense of the word.
Sorry but I took time off for the holidays and got it resolved, though I have no idea how.
I decided to try and remove one of the 2 final variables. Since I could not try another software option due to GFiber requirements, I decided to get cable internet. Long story short, Once I configured the pfsense router for dual wan and had cable as my primary and GF as the failover, my latency was stable at 70-80 ms. I then switched the roles of primary and secondary wan and latency stayed stable at 70-80 ms. I just disconnected the the cable connection and rebooted the router and still has stable 70-80 ms latency. So whatever happened in setting up the dual WAN interfaces fixed the issue, going on 7 days straight.
My Setup:
A10-8750K
8gb DDR3 ram
60gb ssd
4 port GB intel nic ( 5GB ports total)
LGS 318P
dual port LAG between the router and switch
Dual WAN with failover (doubt I could saturate the GF connection to the point that pfsense would load balance anyway)
As a side note to anyone find this forum posting. I see nothing wrong with pfsense or Google Fiber, its was a strange combination between pfsense + GFiber +WOW that was the issue. Still not sure what fixed it, but it works just fine. Too my knowledge none of the ppl replying to this have GFiber and thus don't have all the information noe the setup to test anything. Cable networking and Fiber networking are different. With cable the modem you use its basically in bridge mode and all you have to do is connect a cheap consumer wireless router and off you go. You can't do that with Fiber you have to use the supplied network box of find a solution to replicate the require WAN protocols. Most of the replies were helpful, and you can see who the moron(s) were.