Netgate SG-1000 microFirewall

Author Topic: HAProxy + ACME [FIXED]  (Read 550 times)

0 Members and 1 Guest are viewing this topic.

Offline uwscia

  • Newbie
  • *
  • Posts: 13
  • Karma: +0/-0
    • View Profile
HAProxy + ACME [FIXED]
« on: December 01, 2017, 07:46:02 pm »
I have to say I've tried multiple different tuts trying to get this working...
It seems that there is just 1 thing every tut is missing cause it doesn't matter which one I used nothing seems to work...
Currently getting a 503 which makes me think that haproxy is working... but... This is as close as I've come.

Installed
  • pfsense 2.4.2
  • haproxy package 0.54_2
  • acme package 0.1.23

Looking to setup multiple sub domains to pass through firewall
And also setup multiple local domains
  • plex.local or plex.domain.local
  • http1.local
  • http2.local

And I would like to get ACME to server SSL certs for them all.

« Last Edit: December 10, 2017, 01:05:56 pm by uwscia »

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy + ACME Help!
« Reply #1 on: December 03, 2017, 10:50:29 am »
A nice wish-list, only need 3 steps:
- configure pfSense so it works
- configure haproxy so it works
- configure acme package so it works
And your done :o , besides what you 'want', it is important for me to know what you 'did'.

As currently there is just to little information here to tell what setting you might have missed that causes a 503.
Please share haproxy.cfg (from bottom of haproxy settings tab) and some additional screenshots that you have for acme configuration. Also check and tell what haproxy's stats page LastChk is telling about the servers, are they marked as 'down' ?

https://github.com/PiBa-NL/pfsense-haproxy-package-doc/wiki/haproxy_troubleshooting

Other than that, acme cannot provide certificates for your http1.local's domain as it wont be able to check for its existence / ownership on pubic accessible http webserver or public dns txt records..

Offline uwscia

  • Newbie
  • *
  • Posts: 13
  • Karma: +0/-0
    • View Profile
Re: HAProxy + ACME Help!
« Reply #2 on: December 04, 2017, 02:53:18 pm »
I'm sitting with clean install, and the packages installed...

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy + ACME Help!
« Reply #3 on: December 04, 2017, 04:54:21 pm »
Ok so your sitting, that seems like a safe position.. and now what do you expect will happen.?

Have you tried anything else?

Offline uwscia

  • Newbie
  • *
  • Posts: 13
  • Karma: +0/-0
    • View Profile
Re: HAProxy + ACME Help!
« Reply #4 on: December 04, 2017, 05:59:09 pm »
As stated, I've tried multiple tutorials...
and to clear my many attempts I cleared everything, Cause I was still seeing squid in the menu even though I had removed it.
I wanted a fresh install to start from.

Offline uwscia

  • Newbie
  • *
  • Posts: 13
  • Karma: +0/-0
    • View Profile
Re: HAProxy + ACME Help!
« Reply #5 on: December 04, 2017, 06:13:19 pm »
So things I was stuck with... that I thought could be issues.
- Dynamic DNS as I only have a DHCP IP from my ISP a few tutorials utilized DDNS which I have not setup.
  • A (IPv4) record .domain.com to my IP
  • CNAME(s) set to subdomain.*

- HAProxy listening on same port as pfsense

- Or simple Rule not set correctly

- My original issues were while playing with vlan that didn't work and then later finding out I need a switch that supported them, which is ordered and in route... but I figured I could still set the rest up while I waited.

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy + ACME Help!
« Reply #6 on: December 05, 2017, 12:16:52 pm »
So you have a domain name www.domain.com , if you ping that domain it results in your wan-ip? DDNS is only 'required' when the ip changes and you want to update it dynamically.. A static A record should work fine while it doesnt.. (most dynamic ip's for residential connections ive worked with stay the same for weeks or even over a year..) so not using DDNS should not be a problem not in the short term / testing phase anyhow.

pfSense webgui and haproxy using the same port should 'technically' not be a problem, however i prefer to use different ports as from a security and simplicity standpoint it could be ambiguous which service is accepting the connection if either service is not running properly ..

You did have a firewall rule that allows TCP traffic on the WAN interface from any:any to wan-ip:80 ?
You did have haproxy configured with a frontend listening on wan-ip:80 ?

What happened when you try to browse the http://domain.com ? does it timeout? does it show a 503 error? Does stats page of haproxy show the backend is 'up'? Does it work internally?

If you want to properly use vlan's indeed you need a switch that supports vlans, and know/learn how to configure them on the switch.. trunk-port vs access-port with 1 selected native and/or several tagged vlans over a port.. will need a bit of practice to get it configured right..

Anyhow if you can configure things 90% like you think they should be, and show that configuration in screenshots, and explain what part of it doesn't work yet then i can probably help you get it working..

Offline uwscia

  • Newbie
  • *
  • Posts: 13
  • Karma: +0/-0
    • View Profile
Re: HAProxy + ACME Help!
« Reply #7 on: December 05, 2017, 07:46:11 pm »
Ok, so DDNS not so much a requirement which is what I thought, I've also held my IP for over a year before.

The pfSense port I'll leave as is for simplicity.

Yes, I originally had a WAN Rule ipv4 and ipv6 any:any to wan on 80 an 443

Trying to browse my domain would either send me to a 503 or a pfSense binding error screen, depending if the domain was set in HAProxy ACL
If I manually use the local IP it would obviously work.

I'll start configuring HAProxy

Offline uwscia

  • Newbie
  • *
  • Posts: 13
  • Karma: +0/-0
    • View Profile
Re: HAProxy + ACME Help!
« Reply #8 on: December 05, 2017, 08:05:34 pm »
Ok, I've gotten a little further... this is without ACME at this point... Starting to think the squid packages that I removed were causing background issues...

www.domain is getting pointed to server locally & externally
  • don't know how since I never set www. to do anything yet... is the default backend setting causing this?

sub1.domain, and .domain are getting pointed to server externally but not internally... so thinking loopback issue... or caching...

Will test a few more things after dinner...
« Last Edit: December 05, 2017, 08:30:46 pm by uwscia »

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy + ACME Help!
« Reply #9 on: December 06, 2017, 12:22:30 pm »
yes first make sure the basic's work for a simple :80 webservice and perhaps a selfsigned certificate for testing before going for complete acme configuration..

If haproxy returns a 503 usually that is because the server healthchecks are failing. Check on stats page why if that happens.

haproxy indeed sends any traffic it receives on a frontend to the default backend. If traffic arrives on the frontend depends on what ip the dns resolution of www.domain returns.

If it works externally and not internally then a likely cause is the transparent-client-ip feature in the backend.. try disabling that option. Or better make sure client and server are on different networks.. (vlans might help there when the switch arrives)

Offline uwscia

  • Newbie
  • *
  • Posts: 13
  • Karma: +0/-0
    • View Profile
Re: HAProxy + ACME Help!
« Reply #10 on: December 06, 2017, 07:14:42 pm »
Ok, That was going to be my next question... how to stop the loopback issue.

Transparent ClientIP not checked...

So if I have my servers on 192.168.1.1 and my devices on 192.168.2.1 ect... I'll be able to access the domains locally?
Or, Is there a way to redirect my domain internally to a local domain... sub1.domain.com to sub1.domain.local?
Which would be best practice?

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy + ACME Help!
« Reply #11 on: December 07, 2017, 01:15:09 pm »
Internally you could go for a split-dns solution to reply the webserver 192.168.1.100 ip when a request for sub1.domain.com is made.. But i prefer to keep it simple and have all clients both externally and internally follow (more or less) the same route to the destination.

Sending a redirect could be done also, but well the users would be using 2 different domains for the same website / or a specific page on it, making a favorite to a page wont work when going outside anymore.. Or sending a link to a colleague thats outside..

There shouldn't be much of a 'loopback issue' when using haproxy (not like you would have with portforwards..). as the client resolves the pfSense wan-ip, haproxy accepts the connection, and makes another connection to the webserver.. That should 'just work'.. or is the wan-ip not actually on pfSense but on a upstream isp provided router.?.

So to check, the clients do resolve the wan-ip of pfSense when internally requesting the domain? You have made simple 'pass' rules? (no portforwards should be needed..) Does any error appear? Does stats page 'count' a new connection on frontend when you try to connect?

Offline uwscia

  • Newbie
  • *
  • Posts: 13
  • Karma: +0/-0
    • View Profile
Re: HAProxy + ACME Help!
« Reply #12 on: December 08, 2017, 04:56:09 pm »
Ok so I started working on getting HTTPS to work...

I've set HTTP redirect to HTTPS, but can't seem to disable it to just test basic http...  not a big deal right now... but a pain for troubleshooting...

I'm having trouble finding "good practice" guides for setting up the backend SSL to talk with cert manager...

Here is my HAproxy settings ATM..
Code: [Select]
# Automaticaly generated, dont edit manually.
# Generated on: 2017-12-07 22:10
global
maxconn 10
stats socket /tmp/haproxy.socket level admin
uid 80
gid 80
nbproc 1
chroot /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param 2048
log-send-hostname HAproxy
server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
bind 127.0.0.1:2200 name localstats
mode http
stats enable
stats admin if TRUE
stats uri /haproxy/haproxy_stats.php?haproxystats=1
timeout client 5000
timeout connect 5000
timeout server 5000

frontend WAN_HTTP
bind wan_ip:80 name wan_ip:80   
mode http
log global
option http-keep-alive
timeout client 30000
default_backend ssl-redirect_http_ipvANY

frontend WAN_HTTPS
bind wan_ip:443 name wan_ip:443   
mode http
log global
option http-keep-alive
timeout client 30000
acl www-acl hdr(host) -i www.domain.ca
acl cloud-acl hdr(host) -i cloud.domain.ca
acl aclcrt_WAN_HTTPS hdr_reg(host) -i ^www\.domain\.ca(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS hdr_reg(host) -i ^cloud\.domain\.ca(:([0-9]){1,5})?$
acl aclcrt_WAN_HTTPS hdr_reg(host) -i ^domain\.ca(:([0-9]){1,5})?$
use_backend www_http_ipvANY  if  www-acl aclcrt_WAN_HTTPS
use_backend cloud_http_ipvANY  if  cloud-acl aclcrt_WAN_HTTPS

backend ssl-redirect_http_ipvANY
mode http
log global
timeout connect 30000
timeout server 30000
retries 3
server redirect 127.0.0.1 

backend www_http_ipvANY
mode http
log global
timeout connect 30000
timeout server 30000
retries 3
server www-acl 192.168.1.100:8090 ssl  verify none crt /var/etc/haproxy/server_clientcert_5a22d36348213.pem

backend cloud_http_ipvANY
mode http
log global
timeout connect 30000
timeout server 30000
retries 3
server cloud 192.168.1.100:8082

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy + ACME Help!
« Reply #13 on: December 08, 2017, 05:13:45 pm »
For https does your webserver 'require' sending a client certificate? If not make the ssl related selection boxes there on the backend server empty..
As for the frontend youve got it configured with http/https(offloading) but have not configured the server certificate on the frontend. That is required to be able to read the host-header for the "hdr_reg(host)" acl's youve used.

As for the 'redirect' backend it seems your pointing it to the pfSense localhost webgui that is then sending a redirect.. Better configure the redirect in haproxy itself as a 'action' if desired, or point it to the actual webserver:80 for testing.?

Also it might be that the overall webgui redirect in pfSense advanced settings is confusing you.. Or a cached HSTS header that the webgui sends.. To remove that HSTS redirect from browser cache special steps are needed in the browser used..

Offline uwscia

  • Newbie
  • *
  • Posts: 13
  • Karma: +0/-0
    • View Profile
Re: HAProxy + ACME Help!
« Reply #14 on: December 08, 2017, 06:09:37 pm »
So the certs get set in the frontend or backend?