Netgate SG-1000 microFirewall

Author Topic: Upgrade 2.4.0: firewall rule with alias and FQDN not working anymore  (Read 1293 times)

0 Members and 1 Guest are viewing this topic.

Offline ggzengel

  • Full Member
  • ***
  • Posts: 264
  • Karma: +3/-0
    • View Profile
Re: Upgrade 2.4.0: firewall rule with alias and FQDN not working anymore
« Reply #15 on: October 17, 2017, 10:05:51 am »
What says Status/System Logs/System/DNS Resolver?

Before update it was working:
Code: [Select]
Sep 22 22:47:42 filterdns adding entry 10.19.4.250 to table smtp_server on host smtp.domain.local
Sep 22 22:42:48 filterdns failed to resolve host smtp.domain.local will retry later again.
Sep 22 22:18:56 dnsmasq 43335 using nameserver 8.8.4.4#53
Sep 22 22:18:56 dnsmasq 43335 using nameserver 8.8.8.8#53
Sep 22 22:18:56 dnsmasq 43335 ignoring nameserver 127.0.0.1 - local interface

After update it was working to:
Code: [Select]
Oct 12 20:41:53 filterdns adding entry 10.19.4.250 to pf table smtp_server for host smtp.domain.local
Oct 12 20:41:53 filterdns clearing entry 10.19.4.250 from pf table smtp_server on host smtp.domain.local
Oct 12 20:41:46 filterdns adding entry 10.19.4.250 to pf table smtp_server for host smtp.domain.local
Oct 12 20:41:46 filterdns clearing entry 10.19.4.250 from pf table smtp_server on host smtp.domain.local
Oct 12 20:41:45 filterdns adding entry 10.19.4.250 to pf table smtp_server for host smtp.domain.local
Oct 12 20:41:45 filterdns failed to resolve host smtp.domain.local will retry later again.
Oct 12 20:26:06 dnsmasq 860 using nameserver 8.8.4.4#53
Oct 12 20:26:06 dnsmasq 860 using nameserver 8.8.8.8#53
Oct 12 20:26:06 dnsmasq 860 ignoring nameserver 127.0.0.1 - local interface

Suddently on Saturday it didn't update this entry any more:
Code: [Select]
Oct 17 13:30:37 filterdns adding entry 216.58.210.3 to pf table Host for host www.google.de
Oct 17 13:30:37 filterdns adding entry 10.19.4.250 to pf table Host for host smtp.domain.local
Oct 14 06:45:01 filterdns clearing entry 10.19.4.250 from pf table smtp_server on host smtp.domain.local
Oct 14 06:30:01 filterdns adding entry 10.19.4.250 to pf table smtp_server for host smtp.domain.local
Oct 14 06:30:01 filterdns clearing entry 10.19.4.250 from pf table smtp_server on host smtp.domain.local
Oct 14 06:15:01 filterdns adding entry 10.19.4.250 to pf table smtp_server for host smtp.domain.local
Oct 14 06:15:01 filterdns clearing entry 10.19.4.250 from pf table smtp_server on host smtp.domain.local
Oct 14 06:00:01 filterdns adding entry 10.19.4.250 to pf table smtp_server for host smtp.domain.local

It only worked today as I added google too.
Yesterday on OCT 16 I tried successfully to ping at smtp.domain.local. So why didn't he update? Did the job crash?

I think filterdns has a problem. I have 2 running and since the second one runs I have a fresh alias table:
Code: [Select]
ps aux | grep filterdns
root   19719   0.0  0.3  21492  3184  -  Is   13:30       0:00.03 /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1
root   58949   0.0  0.3  12784  2616  -  Is   Thu20       0:00.35 /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1
root   44060   0.0  0.2  14728  2444  0  S+   15:03       0:00.00 grep filterdns

Offline Gertjan

  • Hero Member
  • *****
  • Posts: 2425
  • Karma: +191/-9
    • View Profile
Re: Upgrade 2.4.0: firewall rule with alias and FQDN not working anymore
« Reply #16 on: October 17, 2017, 10:45:10 am »
Code: [Select]
Sep 22 22:47:42 filterdns adding entry 10.19.4.250 to table smtp_server on host smtp.domain.local
Sep 22 22:42:48 filterdns failed to resolve host smtp.domain.local will retry later again.
Sep 22 22:18:56 dnsmasq 43335 using nameserver 8.8.4.4#53
Sep 22 22:18:56 dnsmasq 43335 using nameserver 8.8.8.8#53
Sep 22 22:18:56 dnsmasq 43335 ignoring nameserver 127.0.0.1 - local interface
Wait ...
You're asking 8.8.8.8 - 8.8.4.4 (Also known as Google) info about "smtp.domain.local" ?
Well, yes, that will fail  ;D

If "smtp.domain.local" your has a static IP, add it to Services => DNS Forwarder => Host Overrides and you'll be fine.

Offline johnpoz

  • Hero Member
  • *****
  • Posts: 15153
  • Karma: +1413/-206
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Upgrade 2.4.0: firewall rule with alias and FQDN not working anymore
« Reply #17 on: October 17, 2017, 10:47:13 am »
yeah...

Why is it ignoring 127.0.0.1?

"Sep 22 22:18:56    dnsmasq    43335    ignoring nameserver 127.0.0.1 - local interface"

edit:  This is forwarder, going to have to forward somewhere ;)  I have not used the forwarder since they enabled unbound.. Well really before that when unbound was just a package.  A resolver is just so much better than a forwarder.  Not sure why anyone still uses it to be honest ;)

In a nutshell if you have an alias that is not working, you need to check the table.  If entries not in the table then you need to figure out why the resolution of whatever FQDN is not working is not in the table.  Pfsense needs to be able to resolve the FQDN you put in there for it to be able to put in the table..

So normally such problems just come down to name resolution troubleshooting.. Which doesn't look like any was done before bug report filed ;)

« Last Edit: October 17, 2017, 10:51:28 am by johnpoz »
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.4.2-RELEASE-p1 (work)
1x SG-4860 2.4.2-RELEASE-p1 (home)

Offline ggzengel

  • Full Member
  • ***
  • Posts: 264
  • Karma: +3/-0
    • View Profile
Re: Upgrade 2.4.0: firewall rule with alias and FQDN not working anymore
« Reply #18 on: October 17, 2017, 11:30:17 am »
Sep 22 22:18:56    dnsmasq    43335    using nameserver 8.8.4.4#53
Sep 22 22:18:56    dnsmasq    43335    using nameserver 8.8.8.8#53
Sep 22 22:18:56    dnsmasq    43335    ignoring nameserver 127.0.0.1 - local interface

What you see here is dnsmasq and not filterdns.
Dnsmasq works on localhost so it could not add itself. This would give a loop.

If filterdns is running it makes it good.
Code: [Select]
Oct 13 10:29:32 filterdns adding entry 10.19.4.250 to pf table smtp_server for host smtp.domain.local
Oct 13 10:29:32 filterdns clearing entry 10.19.4.250 from pf table smtp_server on host smtp.domain.local
Oct 13 10:15:01 filterdns adding entry 10.19.4.250 to pf table smtp_server for host smtp.domain.local
Oct 13 10:15:01 filterdns clearing entry 10.19.4.250 from pf table smtp_server on host smtp.domain.local
Oct 13 10:00:01 filterdns adding entry 10.19.4.250 to pf table smtp_server for host smtp.domain.local
Oct 13 10:00:01 filterdns clearing entry 10.19.4.250 from pf table smtp_server on host smtp.domain.local

But since update it talks too much:
Code: [Select]
Oct 10 01:47:52 filterdns failed to resolve host smtp.domain.local will retry later again.
Sep 22 22:47:42 filterdns adding entry 10.19.4.250 to table smtp_server on host smtp.domain.local
Sep 22 22:42:48 filterdns failed to resolve host smtp.domain.local will retry later again.
Sep 22 22:18:56 dnsmasq 43335 using nameserver 8.8.4.4#53
Sep 22 22:18:56 dnsmasq 43335 using nameserver 8.8.8.8#53
Sep 22 22:18:56 dnsmasq 43335 ignoring nameserver 127.0.0.1 - local interface

Offline pauby

  • Jr. Member
  • **
  • Posts: 43
  • Karma: +0/-0
    • View Profile
Re: Upgrade 2.4.0: firewall rule with alias and FQDN not working anymore
« Reply #19 on: October 17, 2017, 11:46:02 am »
What says Status/System Logs/System/DNS Resolver?

DNS Resolver only has the 'unbound' process. There is nothing of filterdns or dnsmasq in there. There is also nothing in System|General either for either filterdns or dnsmasq.

Are you not using DNS Forwarder service rather than DNS Resolver? I'm assuming there are different 'process' entries.

I'm happy to check anything else out to try and resolve this.

Offline ggzengel

  • Full Member
  • ***
  • Posts: 264
  • Karma: +3/-0
    • View Profile
Re: Upgrade 2.4.0: firewall rule with alias and FQDN not working anymore
« Reply #20 on: October 17, 2017, 11:51:52 am »

Quote
edit:  This is forwarder, going to have to forward somewhere ;)  I have not used the forwarder since they enabled unbound.. Well really before that when unbound was just a package.  A resolver is just so much better than a forwarder.  Not sure why anyone still uses it to be honest ;)

I tried to migrate to unbound last year but I had some problems: https://redmine.pfsense.org/issues/6065
Because I have more than 40 Overrides I don't like to try it again on this pfsense.
And there are still some unwanted effects with unbound: https://redmine.pfsense.org/issues/7884

Offline pauby

  • Jr. Member
  • **
  • Posts: 43
  • Karma: +0/-0
    • View Profile
Re: Upgrade 2.4.0: firewall rule with alias and FQDN not working anymore
« Reply #21 on: October 17, 2017, 11:54:14 am »
In a nutshell if you have an alias that is not working, you need to check the table.  If entries not in the table then you need to figure out why the resolution of whatever FQDN is not working is not in the table.  Pfsense needs to be able to resolve the FQDN you put in there for it to be able to put in the table..

So normally such problems just come down to name resolution troubleshooting.. Which doesn't look like any was done before bug report filed ;)

So what do you suggest beyond what has been done (by me)?

Offline ggzengel

  • Full Member
  • ***
  • Posts: 264
  • Karma: +3/-0
    • View Profile
Re: Upgrade 2.4.0: firewall rule with alias and FQDN not working anymore
« Reply #22 on: October 17, 2017, 11:58:21 am »
I have messages from filterdns in there. It's not dnsmasq and not unbound.

Even with unbound on an other pfsense I get this:
Code: [Select]
Oct 13 01:59:09 filterdns adding entry 79.1.2.3 to ipfw table for host dummy.dyndns.org
Oct 13 00:59:06 filterdns failed to resolve host dummy.dyndns.org will retry later again.
Oct 12 22:24:47 filterdns adding entry 87.1.2.3 to ipfw table for dummy.dyndns.org
Oct 12 22:24:45 unbound 74165:0 info: start of service (unbound 1.6.6).
Oct 12 22:24:45 unbound 74165:0 notice: init module 0: iterator

Offline ggzengel

  • Full Member
  • ***
  • Posts: 264
  • Karma: +3/-0
    • View Profile
Re: Upgrade 2.4.0: firewall rule with alias and FQDN not working anymore
« Reply #23 on: October 17, 2017, 12:19:39 pm »
I still had these two filterdns running.
I killed both. For the older one it was enough to send kill. The second one needed kill -9 to stop.

I removed the test entry with a FQDN inside and pfsense started a new filterdns.
Now it's not spamming any more. Only on changes (add/delete entries or changing IPs) I see filterdns entries in log.

Offline pauby

  • Jr. Member
  • **
  • Posts: 43
  • Karma: +0/-0
    • View Profile
Re: Upgrade 2.4.0: firewall rule with alias and FQDN not working anymore
« Reply #24 on: October 17, 2017, 01:40:07 pm »
I still had these two filterdns running.
I killed both. For the older one it was enough to send kill. The second one needed kill -9 to stop.

I removed the test entry with a FQDN inside and pfsense started a new filterdns.
Now it's not spamming any more. Only on changes (add/delete entries or changing IPs) I see filterdns entries in log.

I added a new alias and a new FQDN (www.barrymanilow.com) and on query it's table (Diagnostics -> Tables) it has no entries. I get no filterdns entries in System|General or System|DNS Resolver logs.

filterdns does exist and is running (Diagnostics -> Command -> ps -A | grep filterdns).

I do get these errors in the System|General log file (whcih could have been therefore prior to the upgrade and they are maybe a red herring):

Oct 17 19:29:39    dhcpleases       kqueue error: unkown
Oct 17 19:29:38    dhcpleases       Could not deliver signal HUP to process because its pidfile (/var/run/unbound.pid) does not exist, No such process.
Oct 17 19:29:38    dhcpleases       /etc/hosts changed size from original!

The /var/run/unbound.pid does exist.

I also did a 'cat /etc/hosts' and the nas.fqdn.private entry is in there. I think we can discount the 'if pfsense cannot resolve it it won't be in a table' issue as pfSense can not only resolve it, it's put it into it's Hosts file.

So the issue is that Aliases have no table entries.

And I'll say again what I said earlier the upgrade to 2.4 broke this.

Offline pauby

  • Jr. Member
  • **
  • Posts: 43
  • Karma: +0/-0
    • View Profile
[SOLVED] Upgrade 2.4.0: firewall rule with alias and FQDN not working anymore
« Reply #25 on: October 17, 2017, 02:03:33 pm »
So I'll update the post with the fix for this.

The first DNS server listed in System -> General Setup was dead. It was working fine as there were anotehr 3 DNS servers in there. As soon as I replaced this one server with a working one I started seeing 'filterdns' entries in the System|DNS Resolver log. I checked the rules and they have started working now as well.

What I don't understand is:

1. The majority of the aliases are for internal IP's and therefore don't need external DNS resolution;
2. The majority of the aliases are for DHCP leases and are therefore registered by the DHCP service and appear in the pfSense Hosts file so again don't need external resolution;
3. If you have 4 listed DNS servers and one breaks then why should this stop aliases working;
4. What has changed that this issue did not appear before the upgrade to v2.4;

If I have a working system and I upgrade it and parts of it stops working then that's a problem. It's a bug. A bug in the upgrade. A bug in the way something works. But it's a bug. Something that should work doesn't. That's clear from this.
« Last Edit: October 17, 2017, 02:12:15 pm by pauby »

Offline ggzengel

  • Full Member
  • ***
  • Posts: 264
  • Karma: +3/-0
    • View Profile
Re: Upgrade 2.4.0: firewall rule with alias and FQDN not working anymore
« Reply #26 on: October 17, 2017, 02:21:41 pm »
If I understand it right?

You have local DNS entries which appear in /etc/hosts?
And now the first of the external DNS servers is not responding and local IPs (from /etc/hosts) are not resolved in alias table?

Normally nsswitch.conf looks like:
hosts: files dns

What shows yours?

Offline pauby

  • Jr. Member
  • **
  • Posts: 43
  • Karma: +0/-0
    • View Profile
Re: Upgrade 2.4.0: firewall rule with alias and FQDN not working anymore
« Reply #27 on: October 17, 2017, 02:28:59 pm »
If I understand it right?

You have local DNS entries which appear in /etc/hosts?

Yes. All of them were static DHCP leases. All of them resolved using Diagnostics -> Ping  and Disagnostics -> DNS Lookup in pfSense.

Quote
And now the first of the external DNS servers is not responding and local IPs (from /etc/hosts) are not resolved in alias table?

They don't appear in the aliases tables. And filterdns entries were missing from the System|DNS Resolver logs.

Quote
Normally nsswitch.conf looks like:
hosts: files dns

What shows yours?

It is:

hosts: files dns

Exactly the same.

Offline ggzengel

  • Full Member
  • ***
  • Posts: 264
  • Karma: +3/-0
    • View Profile
Re: Upgrade 2.4.0: firewall rule with alias and FQDN not working anymore
« Reply #28 on: October 17, 2017, 02:32:59 pm »
I think you should open a bug in redmine.

Offline pauby

  • Jr. Member
  • **
  • Posts: 43
  • Karma: +0/-0
    • View Profile
Re: Upgrade 2.4.0: firewall rule with alias and FQDN not working anymore
« Reply #29 on: October 17, 2017, 02:38:19 pm »
I think you should open a bug in redmine.

I would do normally. But there is already a bug open for this. The way it was also completely dismissed without waiting for further information and pushed back to the forum means I'm not going to waste my time going through the hoops to do it. I appreciate that there was not a lot of information given on the issue raised but the way it was handled was poor. Pre-empting an issue as not a bug 'because we don't see it here' is a naive viewpoint and a does not encourage people to feedback on their project.

But I do appreciate your help in this ggzengel. Between the pair of us it lead me to find what I did. It's been much appreciated.