Netgate SG-1000 microFirewall

Author Topic: HAProxy as SSL Reverse Proxy Behind Single IP  (Read 20052 times)

0 Members and 1 Guest are viewing this topic.

Offline Brailyn

  • Jr. Member
  • **
  • Posts: 40
  • Karma: +0/-0
    • View Profile
HAProxy as SSL Reverse Proxy Behind Single IP
« on: December 09, 2015, 09:07:54 am »
Hello,

I'd like to implement a reverse proxy that can make all of my webserver-lan-IPs:Ports and be directed appropriately by using a ddnshost-wan-IP.ca:443 from the outside. HTTPS to start, but other protocols are welcome later if HAProxy can make sense of them.

I have a single IP from my ISP since statics are $15/month! :o

Mirroring this inside my network would be the preference, but one step at a time here.

I have not been successful at implementing this and am looking for some help. PfSense support told me that one could make HAProxy work for me but it would be 6 hours to build. I'm not going to say the rate, but it is not cheap. That wasn't a big surprise, so I've been attempting this on my own.

This is only for my networking desire at home. Nothing major...or I would buy statics no question.

Do any of you have this working in some capacity? Currently, I do not have much built as I keep blowing it all away.

One example:

LAN IP         LAN PORT      PUBLIC IP                      PUBLIC PORT
10.0.0.1      5001              https://my-ddns1.ca     443
10.0.0.2      80                  https://my-ddns2.ca     443
10.0.03       443                https://my-ddns3.ca     443

Eventually I'd like to mirror the my-ddns#.ca:443 (or port 80 to relief SSL processing in the LAN) hostname to my LAN even though the service may be on 5001. Forward and Reverse Proxy. No load balancing.

Am I crazy?
« Last Edit: December 09, 2015, 12:47:36 pm by Brailyn »

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy as SSL Reverse Proxy Behind Single IP
« Reply #1 on: December 10, 2015, 03:15:15 pm »
Hi Brailyn,

Proxying HTTPS and or HTTP to multiple backends from one public-ip should be possible with haproxy. Doing that myself, so no your probably not crazy :o , not that i'm legally allowed to make such judgement.

Anyway for HTTPS you have a few choices what to do:
1-use haproxy in HTTPS/SSL mode and use SNI information from the ssl handshake to decide.
2-load certificates on haproxy and decypher HTTPS on haproxy then decide by host header where to send the traffic
If you go with option 2, then the question is on the backend side if you want to encrypt traffic again or perhaps keep it as http.. Re-encrypting costs more cpu.. But might avoid some trouble with absolute urls returned by the webapplication..

Use haproxy-devel package untill 1.6 stable package arrives :).. Haproxy 1.6 itself is stable so nothing much to worry there..

Create 3 backends pointing to your servers:
Create one frontend. And add acl's and actions there to direct traffic to the proper backend.

Check in stats that the servers are seen 'UP' when using healthchecks. And fix settings where needed..

If you still have trouble please post the content of haproxy.cfg so i can advice further.

Regards,
PiBa-NL

Offline Brailyn

  • Jr. Member
  • **
  • Posts: 40
  • Karma: +0/-0
    • View Profile
Re: HAProxy as SSL Reverse Proxy Behind Single IP
« Reply #2 on: December 14, 2015, 10:12:04 pm »
Sorry for the huge delay, I've been dreading how difficult this is going to be.

I'm just going to walk through the steps as I do them so you can point out my mistakes. If it is a waste of your time, just say so and I'll post my config file that you mentioned (if I can find it)...


BACKENDS

I'll start with the DSM subdomain.

Balance=none
Transparent ClientIP = yes and LAN for backend.

ACL1
Name= DSM
Host starts with: https://dsm.root-domain.ca

ACL2
Name= root
Host starts with: https://root-domain.ca


FRONTEND

External Address

WAN address, 443, SSL-offloading=yes

ACL1
Name= DSM
Host starts with: https://dsm.root-domain.ca

ACL2
Name= root
Host starts with: https://root-domain.ca

not sure if it matters, but saving everything gives this error:

Code: [Select]

[WARNING] 347/215713 (62182) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.


Firewall, I allowed "any source, 192.168.1.100 dest" on 443 so I could test. I'm behind a 2wire gateway so you know. 443 is port forwarded.

I jumped onto a remote network to test. Going to subdomain DSM resolved in an HSTS errrr. Going to my root domain shows a HSTS error too.

Edit: I see that this morning from work, I get a "503 Service Unavailable | No server is available to handle this request.:D so it looks like HAProxy is listening to some extent.

My questions so far:

1) How do I specify the ports of each service within my LAN?

2) Where is the haproxy.cfg file?

3) Is there any guides on how HAProxy is configured to do something similar to this? Or is it supposed to be straight forward? (It sure isn't that obvious to me :o )

Again, thank you for your help!
« Last Edit: December 15, 2015, 07:03:58 am by Brailyn »

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy as SSL Reverse Proxy Behind Single IP
« Reply #3 on: December 15, 2015, 01:00:17 pm »
I think your acl 'host starts with' should not include 'https://'

1- On the backend you should add servers that point to the apropriate ports.
2-On the config tab at the bottom there is a link to show the config.

3- the zip file attached here has some info: https://forum.pfsense.org/index.php?topic=93766.msg527268#msg527268
Also creating a new wiki here https://github.com/PiBa-NL/pfsense-haproxy-package-doc/wiki but thats a work in progress..

Offline Brailyn

  • Jr. Member
  • **
  • Posts: 40
  • Karma: +0/-0
    • View Profile
Re: HAProxy as SSL Reverse Proxy Behind Single IP
« Reply #4 on: December 15, 2015, 10:46:42 pm »
Hello again,

I appreciate your guides, PiBa. I am a lot further ahead than last night, and have spend a lesser amount of time attempting it.

I decided to go with SSL offloading...I do not have many web servers... and I like the ability to manage their certificates (and CAs?) accordingly.

I've got one and only one website barely working. Issues seem to be:

HSTS Browser Security Limitation

1)On my OSX via separate LAN, both Chrome and Safari give the HSTS error and man-in-the-middle attacks (Proxy in the middle I would say)

2) I created a certificate with the CN matching the hostname in an attempt to resolve. Same issue.
My iPhone allows me to connect via it's safari:) I changed the LAN server a couple of times to prove it was not just a cached site.

3) What's going on here?

Self-signed Internal Certificates

4) I created them a few times with different CNs. No luck there either. Still URL matching issues.

5) I don't want to buy a legit wildcard CA until I know that I can make it work with a self-signed one. At which point I would probably just add it to the trusted list on mine/friends machines.

6) The alternative names section seems like the way to go....just plug in all of the domains pointing to my single IP address right? Type =DNS, Value=DNS hostnames.

7) The CA and Cert thing always seems to cause issues with us noobs.

My Config

Code: [Select]
/var/etc/haproxy.cfg file contents:
global
stats socket /tmp/haproxy.socket level admin
uid 80
gid 80
nbproc 1
chroot /tmp/haproxy_chroot
daemon

listen HAProxyLocalStats
bind 127.0.0.1:444 name localstats
mode http
stats enable
stats admin if TRUE
stats uri /haproxy_stats.php?haproxystats=1
timeout client 5000
timeout connect 5000
timeout server 5000

frontend websites
bind 192.168.1.100:443 name 192.168.1.100:443 ssl  crt /var/etc/haproxy/websites.pem 
mode http
log global
option http-keep-alive
timeout client 30000
acl dsm hdr(host) -i dsm.my-domain.ca
default_backend dsm_http_ipvANY

backend dsm_http_ipvANY
mode http
log global
timeout connect 30000
timeout server 30000
retries 3
option httpchk OPTIONS /
server dsm 10.0.0.1:5001 ssl check inter 1000  weight 1 verify none

General Questions Again

Q1) How do I add the same frontend to multiple backends? It didn;t seem to allow this for SSL offloading if I remember correctly.

Q2) When is it good to use the Transparent ClientIP option in the backend servers? I know my dsm server has it's own firewall which could cause issues here.
Edit: Just tried it from work this morning, and it is indeed working on chrome with a Transparent IP. I'd bet it's safer that way it doesn't keep accounts logged in for 10.0.0.254 even though it could be from another public IP outside my network. Could this actually be an issue when not using Transparent ClientIPs?

Q3) Once I get multiple sites going visible to the outside world, do I just add alias hostnames to pfsense and HAProxy will send them to the appropriate LAN devices from within the LAN, or is it not that easy?

Q4) To prevent the Man-in-the-Middle errors, does the proxy need some sort of intermediate cert? or is it actually the "end" point to the outside world?

Thank you for the guides! I'd love to help if I was capable.
« Last Edit: December 16, 2015, 07:25:36 am by Brailyn »

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy as SSL Reverse Proxy Behind Single IP
« Reply #5 on: December 16, 2015, 01:03:32 pm »
Haproxy does not need the CA for sending it to the client, the client should already have the ca stored in the trusted certificate store.

1) HSTS is a security measure which makes browsers verify that a valid and trusted certificate is used for the connection. So if CN does not match, intermediate certificate is missing, or the CA that signed the certs is not already trusted the browser will refuse to connect.

5) adding 'your' ca to the trusted list is important for getting the certificate checked as valid.

6) yes using alternate names is possible, or create a wildcard cert.

Config)
-The acl currently does nothing, it needs a use_backend action (this 'recently' changed in the package..)

Webserver shows up in green on the stats page?

Q1) use shared frontends, or simply configure multiple acl's  and use_backend actions.
Q2) if the backend needs the client-ip and cannot use the x-forward-for header, but im not sure what issue your referring to exactly.? It does come with its own set of problems...
Q3) every server haproxy will use needs to be defined in a backend, aliases wont do much for this purpose.
Q4) if the servercert loaded into haproxy was signed by a intermediateca then that does need to be loaded into the pfSense certificate manager.


Offline Brailyn

  • Jr. Member
  • **
  • Posts: 40
  • Karma: +0/-0
    • View Profile
Re: HAProxy as SSL Reverse Proxy Behind Single IP
« Reply #6 on: December 16, 2015, 01:37:02 pm »
Thank you for the concise answers. I'm glad I have been numbering my convoluted rambles :)

Before I attempt some of these things, I would like to clarify a few things and respond to what you asked.

"Webserver shows up in green on the stats page?" Yes, it is green. Port 10.0.0.254:444 does not work for some reason though.


1) Any tips on making a wildcard cert? Or is it simply *.domain.ca for the CN?

2) How/where do I add "use_backend" to the config? via SSH?

3) I think I've sorted out the transparent IP thing enough. I am aware that it does come with it's own set of problems. Discussion for later maybe.

4) (Q3) so once HAProxy works for the WAN, the LAN will work for the entries as well? It sort of seemed to work last night without horrible loopback issues, but I didn't test it thoroughly.

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy as SSL Reverse Proxy Behind Single IP
« Reply #7 on: December 16, 2015, 02:11:13 pm »
The 444 port that is confgured for HAProxyLocalStats can only be accessed by pfSense itself, and is used 'internally' by the stats tab and Status/haproxy-stats of the haproxy package.

1) yes just put "*.domain.ca" as the CN, and perhaps also "domain.ca" as an alternative name.

2) below the 'acl' on the frontend you can add a 'action' that should allow to choose use_backend

3) yup later 8)

4) yes the haproxy wan-ip frontends can be accessed by lan users and it should work properly.

However if you then enable 'transparent-client-ip' the horrible loopback issues will happen if client and server are on the same subnet. Server will try to reply directly to client, while haproxy waits for response but doesnt get it, and client expects response from haproxy..

Offline Brailyn

  • Jr. Member
  • **
  • Posts: 40
  • Karma: +0/-0
    • View Profile
Re: HAProxy as SSL Reverse Proxy Behind Single IP
« Reply #8 on: December 26, 2015, 01:10:16 am »
Hello again,

Took a break from this for a while.

Been having trouble getting a single frontend to point to multiple backends. Any request gets forwarded to the first default in any case.

Here's my config:

Code: [Select]
/var/etc/haproxy.cfg file contents:
global
stats socket /tmp/haproxy.socket level admin
gid 80
nbproc 1
chroot /tmp/haproxy_chroot
daemon

listen HAProxyLocalStats
bind 127.0.0.1:444 name localstats
mode http
stats enable
stats admin if TRUE
stats uri /haproxy_stats.php?haproxystats=1
timeout client 5000
timeout connect 5000
timeout server 5000

frontend websites
bind 192.168.1.100:443 name 192.168.1.100:443 ssl  crt /var/etc/haproxy/websites.pem 
mode http
log global
option http-keep-alive
timeout client 30000
acl dsm hdr(host) -i dsm.my-domain.ca
acl pfsense hdr(host) -i pfsense.my-domain.ca
acl webroot hdr(host) -i my-domain.ca
default_backend dsm_http_ipv4
default_backend webroot_http_ipv4
default_backend pfsense_http_ipv4
default_backend dsm_http_ipv4

backend dsm_http_ipv4
mode http
log global
timeout connect 30000
timeout server 30000
retries 3
source ipv4@ usesrc clientip
option httpchk OPTIONS /
server dsm 10.0.0.1:5001 ssl check inter 1000  weight 1 verify none

backend webroot_http_ipv4
mode http
log global
timeout connect 30000
timeout server 30000
retries 3
source ipv4@ usesrc clientip
option httpchk OPTIONS /
server webroot 10.0.0.1:443 ssl check inter 1000  weight 1 verify none

backend pfsense_http_ipv4
mode http
log global
timeout connect 30000
timeout server 30000
retries 3
source ipv4@ usesrc clientip
option httpchk OPTIONS /
server pfsense 10.0.0.254:443 ssl check inter 1000  weight 1 verify none

I created a *.my-domain.ca CA (with my-domain.ca as a alternate name) and cert and I keep getting HSTS errors on a lot of browsers. Seems as though I have not created the SSL stuff correctly. I also cannot add the CA to my trusted CA list without a "password" in my OSX keychain--that is not a parameter in PfSense cert configurator... Haven't tried adding the CA to windows machines yet.

Thanks for the help!

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy as SSL Reverse Proxy Behind Single IP
« Reply #9 on: December 26, 2015, 01:03:50 pm »
Looks like in each frontend you configured the 'default backend' so you ended up with 4 defaults.

Please set it to 'none' on 3 and add a 'action' to look like this:

action: Use Backend
parameter backend: dsm
condition acl name: dsm

action: Use Backend
parameter backend: pfsense
condition acl name: pfsense

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy as SSL Reverse Proxy Behind Single IP
« Reply #10 on: December 26, 2015, 01:26:35 pm »
p.s. the 'Templates' tab contains a link for creating a example for "Serving multiple domains from 1 frontend."
If you checkout the configuration that link creates maybe it helps a little..

Offline Brailyn

  • Jr. Member
  • **
  • Posts: 40
  • Karma: +0/-0
    • View Profile
Re: HAProxy as SSL Reverse Proxy Behind Single IP
« Reply #11 on: December 29, 2015, 12:15:54 am »
Okay,

I finally got a few backend servers running via one front end from the outside :)

1) Most of my servers have their own SSL settings... Would this explain the HSTS error from some browsers? I don't mind removing LAN SSL from them, it just takes a bit of time... I hate to see efforts wasted if I know they wouldn't help.

2.1) Is it normal for servers to be bound to their specific backend from the WAN? Once I create the "dsm" subdomain for my synology, I can no longer access it via its old port (5001). Is this normal?

2.2) If you use synology products at all... I will also include that I am having trouble getting mobile applications to work through the rProxy on 443 from outside my LAN or "on-the-road". This doesn't surprise me, but I am also positive it did work for me in the early stages... Maybe it is just a cert based issue that will be solved by proceeding with (1).

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy as SSL Reverse Proxy Behind Single IP
« Reply #12 on: December 29, 2015, 08:09:21 am »
That you got a few running is great :)

1) HSTS is a header send by the server, and is cached for X amount of time (1 year is the usual setting..) , and can even be persisted if submitted to a online list, then you might never get rid of it...

Its actually good to have, as it 'forces' all future connections from clients to be over https with a VALID certificate for the url they request in the address bar. (You will need to get a valid certificate, this means installing the CA you used to sign the certificate on all client computers, or get a certificate from a real CA like LetsEncrypt or buy one..)

If you already have a valid certificate then perhaps you forgot to load the intermediate certificate into pfSense that could cause issues.. Check with for example https://www.ssllabs.com/ssltest/ if indeed the chain not complete.?.

2.1) This is due to the setting "transparent client ip", sadly the implementation is not 'perfect' all reply traffic is 'captured' from the server and send to haproxy.. Even if the inital request did not go through haproxy.. (The webgui does warn for this effect, sorry..)

Workaround is possible by making the server listen on a second port or a second ip, but depending on the machine running the website that might be difficult to configure on that side..

2.2) not using synology myself.. But yes if the certificates are not 'valid' that could cause issues..

Offline Brailyn

  • Jr. Member
  • **
  • Posts: 40
  • Karma: +0/-0
    • View Profile
Re: HAProxy as SSL Reverse Proxy Behind Single IP
« Reply #13 on: January 05, 2016, 01:12:39 pm »
Before I let you go for now,

Is there any way to do HTTPS redirects using HAProxy?

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy as SSL Reverse Proxy Behind Single IP
« Reply #14 on: January 05, 2016, 01:50:13 pm »
Depending what you intend to do exactly my first thought is, yes it can.

Use either a 'action' or put this into advanced passthrough:
Code: [Select]
redirect scheme https if !{ ssl_fc }
And make the frontend listen on :80 as well as the current :443 with offloading.