next »

[rkelleyrtp]

Greetings all.  Today, I was successful in getting haproxy (v1.3.22) working on pfSense 1.2.3-RELEASE.  I scoured the forums looking for a how-to and came up empty.  So today, I spent some time getting this working and thought I would write what I did.  It took a while (there is a bug in the remote syslog setting), but I am now able to leverage my pfSense box as an haproxy front-end to my backend servers.  In summary, here is what I did:

* Create a new VIP for the front-end server - this MUST BE A CARP ADDRESS otherwise haproxy will not start!  Even if you don't have a standby pfSense firewall, you MUST create the VIP using the CARP setting (use bogus password as needed)

* Download and install the haproxy package on your pfSense machine(s)

* Configure HAProxy using default values except the following:
   --> On Settings Tab:
         --> Enable HAProxy
         --> DO NOT USE REMOTE SYSLOG HOST (bug in current version of pfSense haproxy package)

   -->  On Frontend Tab:
        --> External address of the VIP you created before

   --> On Servers Tab:
       --> Add your backend server(s) as necessary


* Configure the firewall rule - very impt!
   --> On the Firewall-->Rules tab
       --> New rule -
            --> Protocol=any
            --> Source=Any
            --> Port=80
            --> Destination=CARP VIP  <very important!!!>
 

After you save everything, you should be able to now leverage haproxy with your pfSense box(es).  Note that no NAT addresses are required for this to work properly.  I will follow-up this thread with some screenshots later.

[Abacabb]

Curious how much traffic you have going through haproxy.

Me after doing 100 mbit/s for a while, haproxy starts crashing. How is it running for you ?

[rkelleyrtp]

Rock solid with no problems,  but I am running no where near 100Mbit speeds (yet).  We have a pair of web servers getting hit about 820K connections per day, and I have "monit" monitoring the haproxy daemon on the pfSense boxes.  To date, I have seen no issues with haproxy.  Both pfSense servers are running less than 5% CPU utilization (2.6Ghz QuadCore CPU, 4G RAM, Intel GigE NICs running 1.2.3-RELEASE).

Hope this helps.

[kamus]

Hey @rkelleyrtp thanks for your last reply and by the way greate job!. Well I can tell me we are working/planning our cluster/failover for our company under pfsense (1.2.3 release) and just right now are bought some servers with identical specs. I would like to question two thinks, one is why we can't use remote syslog? because we didn't have another form to check activity of haproxy, local logs do not register nothing about it (this take me to second point) and second,  I will try to use haproxy statistics but always got an 404 error (not found), if you can explain me a litle bit detailed how we can enable and configure correctly these option we would be so appreciated.

Best regards 

[rkelleyrtp]

Hi.  To make this happen, you need to do the following things:

• Configure/enable a central syslog server (linux, etc) accessible via your LAN segment.
• Configure your pfsense server to send all logs to a remote syslog server. 
  
• Login via SSH (as admin) on your pfSense machine
• Edit the /etc/syslog file
• Comment out all lines starting with "local"
• Add a single like like this:

Code:

# -----------------------------------------------
# Send all local.* logs to remote syslog server
# -----------------------------------------------
local.* @192.168.100.25

• Next, edit your haproxy configuration (via pfSense GUI) and make sure the logging section is set for your remote syslog server

NOTE, the last time I tried this, haproxy would not start because the GUI does not supply the proper syntax for remote syslog servers.  The workaround is to hand-edit the haproxy.cfg file, make the correction, then manually restart haproxy.  If you have a failover pair of pfSense boxes, you will need to update this on both servers.  If you make changes to your haproxy configuration later you, you will need to hand-edit the file again.


As to why you are unable to to get haproxy stats, I don't know because I don't use that feature.  Your best bet would be to look at the haproxy main website - perhaps someone over there can help you.

[rumptis]

I installed it today without doing anything special and it seems to be working just fine so far.

I have it setup for a transparent proxy.

The only thing that doesn't seem to be working is there doesn't seem to be a Antivirus log I can find, I have logging enabled. I don't know if that worked before though, this is the first time I have used it.

This was also on a fresh install with no other packages installed yet.

[rkelleyrtp]

Just found some additional information. 

• By default, the haProxy configuration file hard-codes a value of ONE SECOND when checking the status of your backend servers
• The configuration file also uses a variable named "$numprocs" to identify how many haProxy threads get run.  This value is derived from the number of CPUs installed in your system

 

While these may not  seem like major issues, these default values can wreak havoc with your back-end servers.  In our case, we have two pfSense machines running haProxy for a web service with two back-end servers.  Our firewalls have 4 CPUs each.  Using the default configuration, EACH web sever was getting hit 8 times per second with haproxy status check queries ("HEAD /index.html HTTP/1.0" 200") causing noticeable performance issues with our application (and accounting logs).

Unfortunately, the current version of the haProxy package does not allow you to change these values.  And, if you hand-modify the haproxy.cfg file, the file will get overwritten when you restart the haproxy service.
 
The only way to change these values is to modify the /usr/local/pkg/haproxy.inc file

• To change the timeout for checking your back-end servers, look for "1000" (the only line with 1000) and change it accordingly.  I changed mine to 10000
• To change the number of threads, look for the line like "numprocs = trim(`/sbin/sysctl kern.smp.cpus | cut -d" " -f2`)" - comment it out using the "# sign at the front, then add a line below specifying the number you want.  In my case, I added a new line that read, "$numprocs=1;"  (don't forget the trailing semi-colon).

After making these changes, I restarted the haproxy service (/usr/local/etc/rc.d/haproxy.sh restart) and immediately noticed the log files on our back-end servers cleaned up quite a bit.

Hopefully this makes sense.  Please let me know if you have any questions...


BTW - I really like pfSense and the haproxy package.  Both are rock-solid applications and I swear by them (I put my name on the line every day).  My goal is to help improve this project by providing as much feedback as I can.

[kamus]

thank you so much for shared this information, would be nice if in future versions of haproxy could be able to change these values. by the way I have noticed that current package, service restart function do not reload haproxy, I have to do manually and then run start from web interface and well, menwhile, I will continue working to up haproxy statistics in web interface. regards

[rkelleyrtp]

No problem - glad to help.

One clarification to my earlier post - it appears you CAN set the value when checking the status of your backend servers.  I completely missed it!  Look for the "Check inter" entry box on the Servers tab (double click on the server name then look toward the bottom).  In fact, you can set this value per server.

BTW - I am hoping to catch up with the author of haProxy so we can discuss some of these issues.  I ran into another bug this morning...

[rkelleyrtp]

by the way I have noticed that current package, service restart function do not reload haproxy, I have to do manually and then run start from web interface and well, menwhile, I will continue working to up haproxy statistics in web interface. regards

Make sure the /usr/local/etc/rc.d/haproxy.sh file has the correct "enable" setting.  By default, it is disabled and won't start.  Edit the file and make sure the line reads:

Code:

haproxy_enable=${haproxy-"YES"}

That should fix your start/stop problems.

[kamus]

No problem - glad to help.

One clarification to my earlier post - it appears you CAN set the value when checking the status of your backend servers.  I completely missed it!  Look for the "Check inter" entry box on the Servers tab (double click on the server name then look toward the bottom).  In fact, you can set this value per server.

BTW - I am hoping to catch up with the author of haProxy so we can discuss some of these issues.  I ran into another bug this morning...

great! heheh. I have been searching "Check Inter" entrybox but I can't see it, maybe we are using different versions (currently I'm working on 1.2.3 and haproxy 0.26)
-----------------
I haven't see it that a new version has been released! hehe, now  I can see it . I will try some tests with this new version. Thanks!

[Itwerx]

Quick question for anyone hosting multiple websites, (and web servers).  It appears that if two back-end servers are used that there is a bug in the way the inbound requests are handled?  (FYI we are using multi-WAN config and CARP).

Here's an example:

Frontends:
abcco.com-WAN1
xyzco.com-WAN1    
abcco.com-WAN2
xyzco.com-WAN2

Servers:
abcco.com-DMZ      10.1.1.1:      active     abcco.com-WAN1           1     
                                                                   abcco.com-WAN2
abcco.com-DMZ      10.2.2.1:      active     xyzco.com-WAN1           1     
                                                                   xyzco.com-WAN2

The home page of each site is proxied properly but pages linked within the site all seem to go to one back-end server, (and get 404s if they're for the site not hosted on that server of course.)

Ideas?

[rkelleyrtp]

Can you post your full haproxy config file? 

For what its worth, I changed our config to host two separate back-end clusters and it has been working beautifully.  Here is our config:

# more haproxy.cfg
global
        maxconn                 1000
        uid                     80
        gid                     80
        nbproc                  1
        chroot                  /var/empty
        daemon

listen Cluster1
        bind                    20.30.40.51:80
        mode                    http
        log                     global
        option                  dontlognull
        option                  forwardfor
        maxconn                 10000
        clitimeout              3000
        balance                 roundrobin
        contimeout              6000
        srvtimeout              6000
        retries                 2
         option                 httpchk HEAD /index.html HTTP/1.0
        server                  WEBSVR-1 10.1.1.11:80   check inter 10000 weight 1
        server                  WEBSVR-2 10.1.1.12:80   check inter 10000 weight 1

listen Cluster2
        bind                    20.30.40.52:80
        mode                    http
        log                     global
        option                  dontlognull
        option                  forwardfor
        maxconn                 10000
        clitimeout              3000
        balance                 roundrobin
        contimeout              6000
        srvtimeout              6000
        retries                 2
        option                 httpchk HEAD /index.html HTTP/1.0
        server                  WEBSVR-3 10.1.1.21:80   check inter 10000 weight 1
        server                  WEBSVR-410.1.1.22:80   check inter 10000 weight 1

BTW - if you look closely, you will see a global option called "nbproc".  The current pfsense haproxy package calculates this based upon the # of CPUs on your firewall, and spawns an haproxy thread for each CPU.  As I mentioned earlier in this thread, this can cause undue logging to your back-end servers.  Just something to keep in mind...

[Itwerx]

@rkelleyrtp
   Except those are listening on different public IPs.  (Sorry, I should have clarified this is for a single public IP address. 
   To clarify further, we currently are using Apache reverse proxy to handle inbound requests to a single IP address assigned to multiple sites.  Apache does a good job of reading the site address out of the request header and redirecting to the correct internal server.  Only problem is it is slo-o-ow, (and HAproxy is rly-fst! 
   Is this not a good job for HAproxy...?

[rkelleyrtp]

Itwerx:

So, are you asking if haproxy can read the incoming http header and forward to the appropriate back-end server(s) when a single IP Address is used?  I think what you are looking for are ACLs inside haproxy.  Here is a quick link:

http://agiletesting.blogspot.com/2009/02/load-balancing-in-amazon-ec2-with.html

Essentially, you have a single IP address on the outside (your "listen" address) and multiple servers on the back-end.  Since haproxy can read the http header request, it can make intelligent decisions as to where the requests should go.  From that link above, you define your frontend section like this:

frontend myfrontend *:80
log    global
maxconn 25000
option forwardfor
acl acl_example1 url_sub example1
acl acl_example2 url_sub example2
use_backend example1_farm if acl_example1
use_backend example2_farm if acl_example2
default_backend default_farm

This creates ACLs (access-control-lists) that match the incoming HTTP header.  Next, setup your back-end section to route the requests:

Code:

backend example1_farm
mode http
balance roundrobin
server server1 192.168.1.1:80 check
server server2 192.168.1.2:80 check
backend example2_farm
mode http
balance roundrobin
server server3 10.0.0.3:80 check
server server4 10.0.0.4:80 check
backend default_farm
mode http
balance roundrobin
server server5 192.168.1.5:80 check
server server6 192.168.1.6:80 check

Hope this helps...

-Ron