The pfSense Store

Author Topic: STP and network  (Read 1052 times)

0 Members and 1 Guest are viewing this topic.

Offline fireix

  • Full Member
  • ***
  • Posts: 111
  • Karma: +0/-0
    • View Profile
Re: STP and network
« Reply #45 on: November 08, 2017, 02:16:27 pm »
Let me know if this drawing is correct. The goal is to have redundancy against one failing switch and one failing pfSense fw (or one cable).

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9088
  • Karma: +1037/-306
    • View Profile
Re: STP and network
« Reply #46 on: November 08, 2017, 03:11:40 pm »
In that configuration you are trusting the ISP switch to properly-propagate the CARP traffic on the WAN interfaces which might or might not be the case.

Also, if the WAN link stays up and CARP continues to pass but there is not internet access there will be no failover. But there probably won't be any internet for the secondary either so... It is possible for a strange layer 2 issue that could cause that.

I would rather have a switch under my control connected to WAN and the ISP. Preferably another stack and preferably LACP as in my diagram.

Note that, if you are very careful, you can use a blank VLAN (blank meaning no layer 3 and no other ports configured on it) on the existing stack for the wan traffic. Many (including me) do not particularly like mixing inside and outside traffic on one switch but in practice it can be done safely.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline johnpoz

  • Hero Member
  • *****
  • Posts: 14295
  • Karma: +1330/-193
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: STP and network
« Reply #47 on: November 08, 2017, 03:30:41 pm »
"Many (including me) do not particularly like mixing inside and outside traffic on one switch but in practice it can be done safely."

This is very common practice in the enterprise for sure.  But if budget, space, power constraints, etc I concur it can be done safely.. Just make sure you know what your doing with vlans and your fine.
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.3.4_p1 (work)
1x 2.4.2-RELEASE on VM esxi 6.5 (home)

Offline fireix

  • Full Member
  • ***
  • Posts: 111
  • Karma: +0/-0
    • View Profile
Re: STP and network
« Reply #48 on: November 08, 2017, 04:29:02 pm »
Many (including me) do not particularly like mixing inside and outside traffic on one switch but in practice it can be done safely.

Do you mean the LAN-side of the configuration since I have public IP-space instead of doing NAT?

I have a seperate network behind this that is not connected to the common network at all (for IPMI, console, NAS, monitoring on dedicated switch and port etc). The traffic on the LAN is mostly https/https and all equipment has their firewalls, so hopefully it is safe enough. It is the setup I usually see when I rent space in other data centers as well. But probably not in an office where you just have a few VPN, AP and maybe no ports incoming. If I was to do NAT for all servers and its services using port-forwarding, we would talk thousands of rules in the fw and I doubt it would handle it very well - at least it would be messy compared to gathering the same type of servers/ports in common groups like "cPanel-serverports" :) Also, the license-validation done by cPanel, DirectAdmin etc. would need to match at all times, reverse.. etc. And the client would be given a private IP so that install of software works.. I see a lot of issues doing it any other way...
« Last Edit: November 08, 2017, 04:44:44 pm by fireix »

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9088
  • Karma: +1037/-306
    • View Profile
Re: STP and network
« Reply #49 on: November 08, 2017, 06:18:05 pm »
Quote
Do you mean the LAN-side of the configuration since I have public IP-space instead of doing NAT?

No. I mean using a blank VLAN on the same switch stack for the outside. The ISP side.


The firewall does not change just because you have public addresses inside. You just get to skip the NAT step.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline johnpoz

  • Hero Member
  • *****
  • Posts: 14295
  • Karma: +1330/-193
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: STP and network
« Reply #50 on: November 09, 2017, 07:41:06 am »
The point Derelict is taking about is this scenario.  That some places will frown on.

Where the public traffic (vlan) flows through the same physical switch and lan side traffic, vs using 2 different physical switches for for traffic outside the firewall and traffic behind the firewall.  See attached.

Unless your running some sort of dod facility where this is mandated, its not an issue as long as you vlan the switch correctly so the traffic is isolated from each other at the switch and has to flow over the firewall.  If you have it misconfigured, then its possible for vlan bleed through, etc.  Which is not possible if you use 2 different physical switch(stacks) for outside and inside traffic.

It is common practice though to use different physical switches.. But it is not a requirement for sure.. You see this most common in smaller setups where the wan runs through the same physical hardware.  As a company grows to enterprise size normally they tend to go with different physical hardware for wan and local traffic.  We run multiple different physical switch stacks.. There is the customer side switches, admin side, internet side, dmz side..  All of which have multiple vlans on them - but the physical hardware is normally dedicated to a specific zone of traffic..

All of these different zone will have multiple switches in them core, distribution, access, etc..  Depending on the size of the zone - some customers don't mind sharing hardware for cost savings - but some customers may "require" different physical hardware their setups, etc.

But if your limited in hardware its fine to run different zones of traffic types on the same physical switching hardware as long as you properly vlan it.
« Last Edit: November 09, 2017, 07:52:42 am by johnpoz »
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.3.4_p1 (work)
1x 2.4.2-RELEASE on VM esxi 6.5 (home)

Offline fireix

  • Full Member
  • ***
  • Posts: 111
  • Karma: +0/-0
    • View Profile
Re: STP and network
« Reply #51 on: November 09, 2017, 01:23:14 pm »
I'm still a but curios to how this traffic can get through :p Purely physical, the traffic HAS to go though the fw and the WAN-side-danger-side in order to get to my LAN. I have no switches before my WAN as you are correctly pointing out. You are saying that traffic still may overcome this and get through in some situations?

In the drawing, the 2nd one is more how I have it. You have the Internet (the sky), the router (my ISPs equipment, still hostile), then my fw WAN-side and then another physical different port (LAN) where my switches is connected. The first drawing would be true if I had both a switch and a fw connected to my ISP.

When I'm testing if a firewall works, I do this by using nmap outside my fw (I do this by automated scripting, just as a precaution due to earlier mistakes in rules), in addition to checking if the service is accessible. Are what you talking about something that would not show up on such a test or are you more saying that if I do something wrong, THEN I could open up the traffic to bypass the physical fw? Are you talking on a logic level and somehow the fw would bypass checking it because it is on another VLAN not detected by fw - since I have a kind of accessible network on my LAN?
« Last Edit: November 09, 2017, 01:31:26 pm by fireix »

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9088
  • Karma: +1037/-306
    • View Profile
Re: STP and network
« Reply #52 on: November 09, 2017, 03:57:40 pm »
When you do a transparent, filtering bridge it makes sense to filter on the bridge interfaces, with rules like "WAN" rules on the interface connected to the ISP and rules like "LAN" rules on the interface connected to the inside hosts.

When you try to make a "switch" it makes sense to filter on the bridge interface, with no rules on the interfaces themselves and "LAN"-type rules on the bridge.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline johnpoz

  • Hero Member
  • *****
  • Posts: 14295
  • Karma: +1330/-193
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: STP and network
« Reply #53 on: November 09, 2017, 04:17:40 pm »
"can get through :p Purely physical"

If you are physically isolated than it can not... I think we are getting a bit off topic to the discussion at hand.  But I think it came up with how your actually connected to the ISP be it via CE or PE..  If you were passing the traffic from wan "internet or hostile" through your CE (your equipment in this case Customer Equipment) that the your local traffic also flowed through then there it is possible for barrier incursions.. If the switch is mis configured..

I think this sort of discussion is beyond what the actual question was ;)

I think where we want to get with this is how you connect the Provider network to your network.. And how we mitigate any sort of SPOF issues.. Right.. We want to make the setup as robust as possible and minimize all SPOF (single point of failure)..

So you want lagg to a stacked switching setup both on the Provider side and your Side the Customer.. Do you own/manage the switching infrastructure before pfsense.. Or is the wire all you get from the provider equipment that you are going to plug direct into your pfsense hardware.. And then you manage all the switches behind pfsense?
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.3.4_p1 (work)
1x 2.4.2-RELEASE on VM esxi 6.5 (home)

Offline fireix

  • Full Member
  • ***
  • Posts: 111
  • Karma: +0/-0
    • View Profile
Re: STP and network
« Reply #54 on: November 09, 2017, 09:51:45 pm »
"Do you own/manage the switching infrastructure before pfsense."

No - it is in my rack/building, but I do not have access to its configuration. My ISPs Cisco Catalyst (where I get two ports I can connect to) -> My two pfSense (CARP) -> My two switches (now stacked). This Cisco Catalyst is basically my only single point of failture (but I can call my ISP, get them to go to my location and replace it for me - I can't do that with my stuff).

I think the action-plan is like shown above in my steps. One cable from my ISPs Catalyst to WAN on fw1 and also one to WAN on fw2. Using CARP and sync (dedicated interface with local IP)with one public IP/VIP on WAN-side of the cluster so that my ISP can "put" or route the /24-network to that single IP.

Each pfSense will have a LAG (two ports) that I will name "LAN" (with a .2 and .3 local IP set on each of the two pfSense-servers) that connects to the Switch-stack through LACP (one port from each switch into the LACP-group).

With this setup, I have the hostile/WAN on one side and it is enough for me to add firewallrule on WAN-side against the internal public static IP - along with ports - to let traffic flow. No NAT and no bridge (I think ;).

Any change in this solution? I mean, this looks very similar to the transparent bridge I had, but now it is just a bit more redundant. And all I really wanted was the redundancy on all my own stuff.
« Last Edit: November 09, 2017, 10:02:33 pm by fireix »

Offline fireix

  • Full Member
  • ***
  • Posts: 111
  • Karma: +0/-0
    • View Profile
Re: STP and network
« Reply #55 on: November 10, 2017, 07:47:54 am »
My ISP says they can fix a /29 linknet this way, I will have it available on both ports. It will be downtime in relation with this they say (not the linknet, but the final routing of the /24 from the old method to the new one).

So far things looks good unless my drawing is in-correct ;) They also say I need to route the traffic back myself, but I assume they are just talking about the individual-hosts on my LAN (setting gw=my local CARP VIP=4.4.4.1). This way, all servers have a route back to the pfSense and pfSense of course have the CARP-VIP=8.8.8.1 on WAN-interface with the transport-IP I get from ISP as GW. No more routing is needed, correct?

Any potential showstoppers in this?

They will start by giving me the /29 linknet, så I can prepare one of the pfSense and test some functionality before live.
« Last Edit: November 10, 2017, 08:39:00 am by fireix »

Offline fireix

  • Full Member
  • ***
  • Posts: 111
  • Karma: +0/-0
    • View Profile
Re: STP and network
« Reply #56 on: November 13, 2017, 02:10:43 pm »
Anything that shouldn't be possible with the drawing?

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9088
  • Karma: +1037/-306
    • View Profile
Re: STP and network
« Reply #57 on: November 13, 2017, 04:07:19 pm »
I would rather control my own WAN switch. As long as that switch handles CARP properly it should work like that.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline fireix

  • Full Member
  • ***
  • Posts: 111
  • Karma: +0/-0
    • View Profile
Re: STP and network
« Reply #58 on: November 13, 2017, 07:08:49 pm »
I would rather control my own WAN switch. As long as that switch handles CARP properly it should work like that.

Is there a way my ISPs Cisco Catalyst would not support CARP, so I need to check that? I mean, isn't CARP just to present a single (V)IP? pfSense and its CARP system will choose between themselves what pfSense of the two that should actually be handling the traffic and bind to the (V)IP. So the primary VIP/WAN VIP, is only available at one of the time.

I have presented the drawing to the ISP and they haven't said anything negative, but I assume they are not familiar with how pfSense works.

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9088
  • Karma: +1037/-306
    • View Profile
Re: STP and network
« Reply #59 on: November 13, 2017, 08:33:39 pm »
IDK dude. You'll have to ask them. ISP switches and switching gear is always broken. Always.

Instead of using the term CARP use VRRP. Same requirements apply.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM