You can't. Your interconnect with your ISP must be a /29. They should be willing to switch you over to that, it's not an uncommon request since basically every router/firewall redundancy protocol requires it.

Ok, thats what I thought.
I'll see about getting that changed.

I understand the normal setup for CARP requires multiple IPs on both the WAN and LAN side.
I have that, but not in the "normal" sense:
My ISP provides me with a /30 (so 2 useable IP addresses, one of which is my pfsense WAN IP and the other is its gateway IP)for my public IP, with an additional /28 (in a different address space) routed to my pfsense WAN IP.

Example with numbers (yes, the 10.x.x.x subnet isn't public, these are just examples):
pfSense WAN IP: 10.1.1.42/30
WAN_GW: 10.1.1.41

Additional External IP Addresses: 10.3.67.160/28

How can I set up CARP in this situation?

I have a strange situation, I am unable to telnet/http/ssh etc to hosts on another subnet, until after I do a ping or traceroute, after which the telnet/http/ssh is successful straightaway.

My network setup is like this:

Code:

internet---pfsense----L3 switch-----default subnet (VLAN 1) 10.10.0.0/16
                                               \
                                                -------2nd subnet (VLAN 20) 10.20.0.0/16

the L3 switch is doing the VLAN routing, and has an interface on each VLAN as follows:
VLAN 1: 10.10.0.100/16
VLAN 2: 10.20.0.1/16

all hosts on the default subnet can get to anywhere (i.e. internet, VLAN 1, VLAN 20).  They are using the pfsense firewall as the default gateway.

the pfsense firewall has default gateway the WAN IP
and has a route to the L3 switch for VLAN routing;
i.e. network 10.20.0.0/16 default gw 10.10.0.100/16

all hosts on the 2nd subnet (VLAN 20) have default gateway the L3 interface on VLAN 20
they can telnet to the L3 switch interfaces, either 10.10.0.100 or 10.20.0.1,  but they cannot telnet to any other hosts on VLAN 1.
(NB: I'm using telnet as a test tool; this applies to ssh, http etc)

however, if I then ping a host on VLAN 1, then I can magically telnet/ssh/http etc to that same host for the next few minutes (until some period of time, the inactivity resets something which an ICMP packet magically solved...)

The system logs show that the firewall return route is being blocked with TCP:SA or TCP:R.  However, my network diagram is pretty clear:

Code:

(1) vlan20 host -> L3 switch VLAN20 interface -> L3 switch VLAN 1 interface -> vlan1 host
(2) vlan20 host <- L3 switch VLAN20 interface <- L3 switch VLAN 1 interface <- pfsense gatway <- vlan1 host

and despite adding explicit rules to allow the block, it is still blocked, so obviously this is not a firewall issue per se.

In order to resolve this (I'm not sure if this is just a bandaid or considered a proper fix), I had to enable system -> advanced -> firewall and nat, static route filtering (Bypass firewall rules for traffic on the same interface ).

Does anybody have any suggestions as to why this is happening?

Personally, i think you've got this set up wrong.
The client machines should have the switches routing interfaces as their default gateways - IE, on vlan 1, the switch has an interface with an IP address.  That should be the default gateway for clients on vlan 1.  On vlan 20, same thing.  Then put a default route on the switch such that any other traffic gets routed to the pfsense box IP.  This keeps traffic off of your firewall for inter-vlan routing, and it only has to deal with traffic meant for the internet.  Unless you want to have a firewall between your vlans - which you might.

For routed subnets, you do not want VIPs (other than type Other), just have them routed to a CARP IP on your main IP block.

Ok, great.

Add one IP Alias VIP to get a foothold in the new subnet (for each CARP node), then you can add the rest as CARP VIPs.

That gets you the required address inside the subnet that CARP wants.

Ok, thanks!

And much simpler than the craptastic way I was trying to do this.

Using 'other' type VIPs for this should work fine, or even proxy ARP, or IP alias would work.

As they are routed to you, they'll hit the router no matter which type you choose.

So what didn't work when you tried?

Well, that worked fine.
I was trying to use CARP VIPs, which require to match an interface subnet, so i was trying to create a new interface containing the subnet and route it through the new WAN connection.
All in all, making it vastly more complicated.

However, what if I do need CARP?  What can I do then?

Hey guys,
I've been beating my head against this for a while today, and I just can't seem to get it to work the way I expect.

Basics:
Running pfsense 2.0.1
I have a multi-WAN setup.  All of that works fine.
Some of the WAN links are actually subnets, and I can create VIPs and NAT on those additional IPs appropriately.

One of the WAN links is new, and is only a /30 (for routing purposes only).  The IP is 38.104.aaa.bbb
I have been assigned a block of IP addresses (38.110.xxx.yyy/28) that is routed to me through the above wan link.

I want to be able to create VIPs in this new block, and NAT them accordingly to use various services through the public IPs.  However, my attempts to do so have failed.

What is the appropriate method to do this?

Going back to the original question, why not use something like this:
http://www.amfeltec.com/products/flexible-minipcie-to-minipci-adapter.php
It converts mini pci-e to mini pci, allowing the use of well know soekris vpn 1411 cards, etc.

2.0 is released, stop mucking about with 1.2.3.

Well, I plan on upgrading, but thought that maybe in the interim someone could offer a solution until my next maintenance window.

I just labbed this out with a spare ALIX and my current config (install 1.2.3, restored my current config to it, upgraded to 2.0) and the issue does appear to be resolved in 2.0.

Fair enough, I now know what I'll be doing in my next maintenance window.

2.0 is released, stop mucking about with 1.2.3.

Well, I plan on upgrading, but thought that maybe in the interim someone could offer a solution until my next maintenance window.

Background:
I am running the nanobsd version of pfsense on an ALIX box.  As such, I am using vlans and trunking to accommodate the various subnets, etc.
Currently, NAT reflection is working - I am using NAT reflection for an SSL protected web site (https).

Issue:
If I go into interfaces, and add another vlan to one of the physical interfaces, and then assign that vlan to a new interface (without even enabling the new interface!), shortly, I begin to have an issue where I get certificate warnings:  For some reason, the certificate of the pfsense box is being presented to internal attempts to access the host.  If I bypass the certificate error, I get a 404 - somehow, the pfsense box is attempting to respond rather than forwarding the request.
I have tried enabling/disabling NAT reflection, I have tried re-creating the rules, I have rebooted the pfsense box.

The only way to get NAT reflection working again is to delete the new interface and then delete the vlan that it was using.  Then the problem goes away.

I'm baffled.  Anyone have any ideas?

You know what?
Thanks for making me do the TCPDUMP.

Seriously.
Because now I looked at its output, and I can see the problem:  The Userland FTP helper is working fine - but the connection on the client isn't being accepted.  Its the local client firewall blocking the active FTP incoming connection.

I HATE ACTIVE FTP.

But at least this problem is sorted.

Thanks again!

WAN
07:08:26.493737 IP WANIPADR.35686 > FTPIPADR.21: S 3438673907:3438673907(0) win 65228 <mss 1460,nop,wscale 4,sackOK,timestamp 6964439 0>
07:08:26.569965 IP FTPIPADR.21 > WANIPADR.35686: S 2995781275:2995781275(0) ack 3438673908 win 16384 <mss 1460,nop,wscale 0,nop,nop,timestamp 0 0,nop,nop,sackOK>
07:08:26.570090 IP WANIPADR.35686 > FTPIPADR.21: . ack 1 win 4163 <nop,nop,timestamp 6964447 0>
07:08:26.648454 IP FTPIPADR.21 > WANIPADR.35686: P 1:182(181) ack 1 win 65535 <nop,nop,timestamp 39242163 6964447>
07:08:26.648536 IP WANIPADR.35686 > FTPIPADR.21: . ack 182 win 4151 <nop,nop,timestamp 6964455 39242163>
07:08:28.463359 IP WANIPADR.35686 > FTPIPADR.21: P 1:14(13) ack 182 win 4163 <nop,nop,timestamp 6964636 39242163>
07:08:28.539459 IP FTPIPADR.21 > WANIPADR.35686: P 182:218(36) ack 14 win 65522 <nop,nop,timestamp 39242181 6964636>
07:08:28.539578 IP WANIPADR.35686 > FTPIPADR.21: . ack 218 win 4160 <nop,nop,timestamp 6964644 39242181>
07:08:29.615695 IP WANIPADR.35686 > FTPIPADR.21: P 14:27(13) ack 218 win 4163 <nop,nop,timestamp 6964751 39242181>
07:08:29.700682 IP FTPIPADR.21 > WANIPADR.35686: P 218:412(194) ack 27 win 65509 <nop,nop,timestamp 39242194 6964751>
07:08:29.700855 IP WANIPADR.35686 > FTPIPADR.21: . ack 412 win 4150 <nop,nop,timestamp 6964760 39242194>
07:08:30.703392 IP WANIPADR.35686 > FTPIPADR.21: P 27:55(28) ack 412 win 4163 <nop,nop,timestamp 6964860 39242194>
07:08:30.779434 IP FTPIPADR.21 > WANIPADR.35686: P 412:442(30) ack 55 win 65481 <nop,nop,timestamp 39242205 6964860>
07:08:30.779565 IP WANIPADR.35686 > FTPIPADR.21: . ack 442 win 4161 <nop,nop,timestamp 6964868 39242205>
07:08:30.784431 IP WANIPADR.35686 > FTPIPADR.21: P 55:61(6) ack 442 win 4163 <nop,nop,timestamp 6964868 39242205>
07:08:30.863724 IP FTPIPADR.21 > WANIPADR.35686: P 442:507(65) ack 61 win 65475 <nop,nop,timestamp 39242205 6964868>
07:08:30.863835 IP WANIPADR.35686 > FTPIPADR.21: . ack 507 win 4158 <nop,nop,timestamp 6964876 39242205>
07:08:30.864630 IP FTPIPADR.55552 > WANIPADR.64724: S 3443460665:3443460665(0) win 65535 <mss 1460,nop,nop,sackOK>
07:08:33.785723 IP FTPIPADR.55552 > WANIPADR.64724: S 3443460665:3443460665(0) win 65535 <mss 1460,nop,nop,sackOK>

LAN
07:08:26.492337 IP CLIENTIPADR.55172 > FTPIPADR.21: S 363713193:363713193(0) win 8192 <mss 1460,nop,wscale 2,nop,nop,sackOK>
07:08:26.493090 IP FTPIPADR.21 > CLIENTIPADR.55172: S 3989763822:3989763822(0) ack 363713194 win 65228 <mss 1460,nop,wscale 4,sackOK,eol>
07:08:26.493308 IP CLIENTIPADR.55172 > FTPIPADR.21: . ack 1 win 2048
07:08:26.648999 IP FTPIPADR.21 > CLIENTIPADR.55172: P 1:182(181) ack 1 win 4106
07:08:26.848149 IP CLIENTIPADR.55172 > FTPIPADR.21: . ack 182 win 2002
07:08:28.463004 IP CLIENTIPADR.55172 > FTPIPADR.21: P 1:14(13) ack 182 win 2002
07:08:28.463122 IP FTPIPADR.21 > CLIENTIPADR.55172: . ack 14 win 4105
07:08:28.540028 IP FTPIPADR.21 > CLIENTIPADR.55172: P 182:218(36) ack 14 win 4106
07:08:28.739063 IP CLIENTIPADR.55172 > FTPIPADR.21: . ack 218 win 1993
07:08:29.569156 IP CLIENTIPADR.61858 > 69.28.145.172.27017: UDP, length 100
07:08:29.615324 IP CLIENTIPADR.55172 > FTPIPADR.21: P 14:27(13) ack 218 win 1993
07:08:29.615450 IP FTPIPADR.21 > CLIENTIPADR.55172: . ack 27 win 4105
07:08:29.701319 IP FTPIPADR.21 > CLIENTIPADR.55172: P 218:412(194) ack 27 win 4106
07:08:29.901423 IP CLIENTIPADR.55172 > FTPIPADR.21: . ack 412 win 1945
07:08:30.702966 IP CLIENTIPADR.55172 > FTPIPADR.21: P 27:51(24) ack 412 win 1945
07:08:30.703084 IP FTPIPADR.21 > CLIENTIPADR.55172: . ack 51 win 4104
07:08:30.780427 IP FTPIPADR.21 > CLIENTIPADR.55172: P 412:442(30) ack 51 win 4106
07:08:30.784166 IP CLIENTIPADR.55172 > FTPIPADR.21: P 51:57(6) ack 442 win 1937
07:08:30.784260 IP FTPIPADR.21 > CLIENTIPADR.55172: . ack 57 win 4105
07:08:30.864292 IP FTPIPADR.21 > CLIENTIPADR.55172: P 442:507(65) ack 57 win 4106
07:08:30.864833 IP FTPIPADR.59304 > CLIENTIPADR.55174: S 3443460665:3443460665(0) win 65535 <mss 1460,nop,nop,sackOK>
07:08:31.060065 IP CLIENTIPADR.55172 > FTPIPADR.21: . ack 507 win 1921
07:08:33.785804 IP FTPIPADR.59304 > CLIENTIPADR.55174: S 3443460665:3443460665(0) win 65535 <mss 1460,nop,nop,sackOK>

Hey guys,
I'm having an issue with outgoing active FTP.
Outbound passive FTP works fine.
However, I've got an app that we have to use that makes ftp connections via the ftp.exe (in windows) which doesn't do passive connections.

I'm running a multi-wan setup here.
I've got the ftp-helper enabled on the LAN interface (i previously had the checkbox checked to disable it so that FTP transfers wouldn't all occur via the WAN interface, which is slower than our WAN2/OPT1 interface), but still no dice (I can connect, but when trying to get a directory listing, it fails).
Any ideas?

I'm totally stuck here...

enable advanced outbound nat.
Set it so that all traffic from exchange server IP is routed out through WAN address: