pfSense Gold Subscription

Author Topic: HAProxy - SSL OffLoading Fine, Adding SSH to the Mix and Stuck  (Read 267 times)

0 Members and 1 Guest are viewing this topic.

Offline ak

  • Jr. Member
  • **
  • Posts: 68
  • Karma: +1/-0
    • View Profile
HAProxy - SSL OffLoading Fine, Adding SSH to the Mix and Stuck
« on: December 10, 2017, 04:11:58 am »
I had HAProxy working great as a reverse proxy with SSL Offloading.

FrontEnd:
  • Bound to 443
  • SSL Offloading setup.
  • 2 domains with SSL
  • Configured with SSL cert and 'Add ACL for certificate Subject Alternative Names
  • Mutliple ACL where if a path starts with particular value, it will go to relevant backend

This was working great and had minimal problems.

I then stumbled upon HAProxy's ability to inspect traffic and being able to then have SSH and SSL on the same 443 port. So I tried setting this up with

FrontEnd [Call this Front_SSL_SSH:443] set to listen to new 443.
  • Type is tcp
  • Set with inspect delay 5s
  • Set acl 'is_ssl' if req_ssl_ver 2:3.1
  • Configured to use the new back end if is_ssl true - BackEnd_SSL_SSH (see below)
  • User ssh_server backend (essentially to a ssh port on a server in my network)

BackEnd_SSL_SSH
  • This points to the original FrontEnd at the very top


SSH works fine, but the web requests fail. The HAProxy logs shows a 'SSL handshake failure' when I try and access the server via a browser. I suspect that the new front end that is doing the detection has done the SSL handshake already, so when it comes the web server, this fails as the browser does not expect a second SSL? Not sure - but would like to keep the SSL Offloading from the original if possible.

Any ideas? Or have I misunderstood the problem. I have googled but all examples I have found does not use the SSL Offloading for the web part.


Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy - SSL OffLoading Fine, Adding SSH to the Mix and Stuck
« Reply #1 on: December 10, 2017, 08:38:18 am »
Can you share the haproxy.conf from bottom of settings tab?

Offline ak

  • Jr. Member
  • **
  • Posts: 68
  • Karma: +1/-0
    • View Profile
Re: HAProxy - SSL OffLoading Fine, Adding SSH to the Mix and Stuck
« Reply #2 on: December 11, 2017, 01:06:39 am »
I have sanitised the config to the below and changed ip etc. Let me know if anything does not make sense.
Note that
  • 192.168.2.1 - this is the WAN address of pfsense
  • 192.168.1.1 - this is the LAN address of pfsense
  • 192.168.1.0/24 - this is the LAN network hosting the webservices


# Automaticaly generated, dont edit manually.
# Generated on: 2017-12-10 11:24
global
   maxconn         128
   log         /var/run/log   local0   info
   stats socket /tmp/haproxy.socket level admin
   uid         80
   gid         80
   nbproc         1
   chroot         /tmp/haproxy_chroot
   daemon
   tune.ssl.default-dh-param   2048
   server-state-file /tmp/haproxy_server_state
   # Time-to-first-Byte (TTFB) value needs to be optimized based on
   # the actual public certificate chain see
   # https://www.igvita.com/2013/10/24
   # /optimizing-tls-record-size-and-buffering-latency/
   tune.ssl.maxrecord 1370
   
   # Modern browser compatibility only as mentioned here:
   # https://wiki.mozilla.org/Security/Server_Side_TLS
   ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK

listen HAProxyLocalStats
   bind 127.0.0.1:2200 name localstats
   mode http
   stats enable
   stats admin if TRUE
   stats uri /haproxy/haproxy_stats.php?haproxystats=1
   timeout client 5000
   timeout connect 5000
   timeout server 5000

frontend personal-domain.com
   bind         192.168.2.1:443 name 192.168.2.1:443 ssl  crt /var/etc/haproxy/personal-domain.com.pem crt /var/etc/haproxy/personal-domain.com 
   mode         http
   log         global
   option         socket-stats
   option         http-keep-alive
   maxconn         10
   timeout client      30000
   redirect scheme https code 301 if !{ ssl_fc }
   acl         PersonalDomainBackend   hdr_end(host) -i personal-domain.com
   acl         AppOneBackend   path_beg -i /secure/appone
   acl         AppTwoBackend   path_beg -i /secure/apptwo
   acl         BusinessDomainBackend   hdr_end(host) -i business-domain.com
   use_backend appone_http_ipvANY  if  PersonalDomainBackend AppOneBackend aclcrt_personal-domain.com
   use_backend apptwo_http_ipvANY  if  PersonalDomainBackend AppTwoBackend aclcrt_personal-domain.com
   use_backend businessdomain_http_ipvANY  if  BusinessDomainBackend aclcrt_personal-domain.com
   use_backend personaldomain_http_ipvANY  if   aclcrt_personal-domain.com

frontend personal-dimain.com.http
   bind         192.168.2.1:80 name 192.168.2.1:80   
   mode         http
   log         global
   option         http-keep-alive
   timeout client      30000
   redirect scheme https code 301 if !{ ssl_fc }

frontend top_level_ssl_ssh_splitter
   bind         192.168.2.1:1443 name 192.168.2.1:1443 ssl  crt /var/etc/haproxy/top_level_ssl_ssh_splitter.pem crt /var/etc/haproxy/top_level_ssl_ssh_splitter 
   mode         tcp
   log         global
   option         log-separate-errors
   option         tcplog
   timeout client      30000
   tcp-request inspect-delay 5s
   tcp-request content accept if { req.ssl_hello_type 1 }
   acl is_ssh payload(1,7) -m bin 5353482d322e30
   tcp-request content accept if is_ssh
   use_backend main_ssl_web_to_frontend_tcp_ipvANY  if  !is_ssh
   use_backend ssh_server_tcp_ipvANY  if  is_ssh
   default_backend ssh_server_tcp_ipvANY

backend appone_http_ipvANY
   mode         http
   log         global
   timeout connect      30000
   timeout server      30000
   retries         3
   server         appone 192.168.1.9:9091 check inter 10000 

backend apptwo_http_ipvANY
   mode         http
   log         global
   timeout connect      30000
   timeout server      30000
   retries         3
   server         apptwo 192.168.1.9:8113 check inter 10000 

backend personaldomain_http_ipvANY
   mode         http
   log         global
   timeout connect      30000
   timeout server      30000
   retries         3
   server         personal-domain.com 192.168.1.9:10080 check inter 1000 

backend businessdomain_http_ipvANY
   mode         http
   log         global
   timeout connect      30000
   timeout server      30000
   retries         3
   acl         businessdomain   hdr_end(host) -i business-domain.com
   server         business-domain.com 192.168.1.9:10080 check inter 1000 

backend main_ssl_web_to_frontend_tcp_ipvANY
   mode         tcp
   log         global
   timeout connect      30000
   timeout server      30000
   retries         3
   server         ssl_web_server 192.168.2.1:443 

backend ssh_server_tcp_ipvANY
   mode         tcp
   log         global
   timeout connect      30000
   timeout server      30000
   retries         3
   server         ssh 192.168.1.9:22

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy - SSL OffLoading Fine, Adding SSH to the Mix and Stuck
« Reply #3 on: December 11, 2017, 07:39:23 am »
The https requests can only be decrypted once.. So after "top_level_ssl_ssh_splitter" decrypts the traffic with the configured certificate. The resulting traffic is plain ssh or http.

As such the "main_ssl_web_to_frontend_tcp_ipvANY" should probably be pointing to a internal/private/localhost:80 port on haproxy that does not decrypt ssl again and further processes the desired acl's to split domains for appone/apptwo/business/personal apart tp the backends.

Offline ak

  • Jr. Member
  • **
  • Posts: 68
  • Karma: +1/-0
    • View Profile
Re: HAProxy - SSL OffLoading Fine, Adding SSH to the Mix and Stuck
« Reply #4 on: December 13, 2017, 02:50:30 pm »
Not had an opportunity to test this out but just wanted to clarify the config

  • The entry point is port 1443 - as 443 was the original and I didn't want to change the setup to drastically. So the entry port via mapping is to port 1443, this then decides to hit the 443 or the ssh server
  • The https website is working fine. The SSH attempt shows in the haproxy.log an 'SSL handshake failure'
  • Tested via a ssh client on IOS

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy - SSL OffLoading Fine, Adding SSH to the Mix and Stuck
« Reply #5 on: December 13, 2017, 03:21:59 pm »
Seems to be the opposite of your previous problem? "SSH works fine, but the web requests fail." and yes i was assuming usage of the 1443.. So please check again which of the two is actually the problem.?.

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy - SSL OffLoading Fine, Adding SSH to the Mix and Stuck
« Reply #6 on: December 13, 2017, 03:34:40 pm »
As for the ssh connection make sure to wrap it inside a ssl layer.?

https://blog.chmd.fr/ssh-over-ssl-episode-4-a-haproxy-based-configuration.html
Code: [Select]
Connecting from an SSH client

To connect to your server from linux, just drop this in your ~/.ssh/config:

Host server.com
    ProxyCommand openssl s_client -connect server.com:443 -quiet

If you are on windows and you cannot install anything client side, there is also a solution for you.
Download socat and putty (none of them requires admin rights).
Then, with socat, run:

socat TCP-LISTEN:8888 OPENSSL:server.com:443,verify=0
And with putty, direct your client to 127.0.0.1 on the port 8888.

Offline ak

  • Jr. Member
  • **
  • Posts: 68
  • Karma: +1/-0
    • View Profile
Re: HAProxy - SSL OffLoading Fine, Adding SSH to the Mix and Stuck
« Reply #7 on: December 16, 2017, 03:50:21 am »
thanks - I can't remember what setting I had for 'ssh working but ssl not'. I had some different acl testing entries but as I needed the website up and running I reverted back to the current set of acl above.

I will try your suggestions. I think there is too much going on so will start from scratch over the holiday period and get basics working before moving up the level of complexity.

Just a quick question - if I have a splitter that decrypts SSL to divert between the web traffic and SSH traffice, will I be able to use SNI to run two websites on different URL's or have I lost that ability on the split as the SSL is decrypted by that time and its only HTTP sent to the web server/frontend?

Offline PiBa

  • Hero Member
  • *****
  • Posts: 819
  • Karma: +132/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: HAProxy - SSL OffLoading Fine, Adding SSH to the Mix and Stuck
« Reply #8 on: December 16, 2017, 09:52:15 am »
Using SNI does not need decryption of the traffic, so it should be possible to not configure any certificate on the 1443 frontend, and keep the is_ssh payload check and have it working like that.

Or perhaps this helps?: https://marc.info/?l=haproxy&m=132375969032305&w=2