The pfSense Store

Author Topic: pkg.pfsense.org appears to be dead  (Read 3245 times)

0 Members and 1 Guest are viewing this topic.

Offline basic612

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
pkg.pfsense.org appears to be dead
« on: May 17, 2017, 03:24:43 am »
The package manager in pfsense has been failing for me of late.

Code: [Select]
[2.3.3-RELEASE][root@lab-firewall.admin.lab]/root: pkg upgrade
Updating pfSense-core repository catalogue...
pkg: https://pkg.pfsense.org/pfSense_v2_3_4_amd64-core/meta.txz: Service Unavailable
repository pfSense-core has no meta file, using default settings
pkg: https://pkg.pfsense.org/pfSense_v2_3_4_amd64-core/packagesite.txz: Service Unavailable
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
pkg: https://pkg.pfsense.org/pfSense_v2_3_4_amd64-pfSense_v2_3_4/meta.txz: Service Unavailable
repository pfSense has no meta file, using default settings
pkg: https://pkg.pfsense.org/pfSense_v2_3_4_amd64-pfSense_v2_3_4/packagesite.txz: Service Unavailable
Unable to update repository pfSense
All repositories are up-to-date.
pkg: Repository pfSense-core cannot be opened. 'pkg update' required
pkg: Repository pfSense cannot be opened. 'pkg update' required
Checking for upgrades (0 candidates): 100%
Processing candidates (0 candidates): 100%
Checking integrity... done (0 conflicting)
Your packages are up to date.

I thought it was something to do with my local proxy setup but am now realising that pkg.pfsense.org seems to have been retired without me realising. Is this the case?

My squid cache is unable to return results:

Quote
The following error was encountered while trying to retrieve the URL: http://pkg.pfsense.org/

Unable to determine IP address from host name pkg.pfsense.org

The DNS server returned:

No DNS records
This means that the cache was not able to resolve the hostname presented in the URL. Check if the address is correct.


Checking dnsstuff it seems that they can't resolve the hostname pkg.pfsense.org either

Quote
No A records exist for pkg.pfsense.org
Tool requires an IP address and pkg.pfsense.org is not a valid IP Address

MXtoolbox is similarly in the dark

Quote
a:pkg.pfsense.orgFind Problems    a 
Test   Result   
   DNS Record Published   DNS Record not found

AFAIK nobody can resolve pkg.pfsense.org at the moment.

Ugh

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21387
  • Karma: +1432/-26
    • View Profile
Re: pkg.pfsense.org appears to be dead
« Reply #1 on: May 17, 2017, 10:21:40 am »
pkg does not use A/AAAA records. It uses service records. It is not meant to be accessed using a browser.

Code: [Select]
$ host -t srv _https._tcp.pkg.pfsense.org
_https._tcp.pkg.pfsense.org has SRV record 10 10 443 files01.netgate.com.
_https._tcp.pkg.pfsense.org has SRV record 10 10 443 files00.netgate.com.
$ host files01.netgate.com.
files01.netgate.com has address 162.208.119.40
files01.netgate.com has IPv6 address 2610:1c1:0:6::40
$ host files00.netgate.com.
files00.netgate.com has address 162.208.119.41
files00.netgate.com has IPv6 address 2610:1c1:0:6::41

Accessing it using its real hostname works fine.

Code: [Select]
$curl --output /dev/null --silent --head --fail "https://files00.netgate.com/pfSense_v2_3_4_amd64-core/meta.txz"
$ echo $?
0

Whatever caused your failure to access pkg.pfsense.org services, it was not DNS, and it does not appear to be a problem at the moment either.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline basic612

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: pkg.pfsense.org appears to be dead
« Reply #2 on: May 17, 2017, 05:16:44 pm »
Many thanks for clarification on that, makes sense now that our upstream squid proxy is failing, not knowing to ask for service records.

Would it be possible to set up A/AAAA records or cname for pkg.pfsense.org.

For now I have fixed as follows:

Quote
[2.3.3-RELEASE][root@lab-firewall.admin.lab]/root: pkg update
Updating pfSense-core repository catalogue...
pkg: https://pkg.pfsense.org/pfSense_v2_3_4_amd64-core/meta.txz: Service Unavailable
repository pfSense-core has no meta file, using default settings
pkg: https://pkg.pfsense.org/pfSense_v2_3_4_amd64-core/packagesite.txz: Service Unavailable
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
pkg: https://pkg.pfsense.org/pfSense_v2_3_4_amd64-pfSense_v2_3_4/meta.txz: Service Unavailable
repository pfSense has no meta file, using default settings
pkg: https://pkg.pfsense.org/pfSense_v2_3_4_amd64-pfSense_v2_3_4/packagesite.txz: Service Unavailable
Unable to update repository pfSense

[2.3.3-RELEASE][root@lab-firewall.admin.lab]/root: sed -i '' 's/pkg.pfsense.org/files00.netgate.com/' /usr/local/etc/pkg/repos/pfSense.conf

[2.3.3-RELEASE][root@lab-firewall.admin.lab]/root: pkg update
Updating pfSense-core repository catalogue...
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01   
Fetching packagesite.txz: 100%    2 KiB   1.8kB/s    00:01   
Processing entries: 100%
pfSense-core repository update completed. 8 packages processed.
Updating pfSense repository catalogue...
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01   
Fetching packagesite.txz: 100%  120 KiB  24.6kB/s    00:05   
Processing entries: 100%
pfSense repository update completed. 439 packages processed.

Offline Alex Atkin UK

  • Jr. Member
  • **
  • Posts: 44
  • Karma: +4/-0
    • View Profile
Re: pkg.pfsense.org appears to be dead
« Reply #3 on: May 19, 2017, 04:50:17 pm »
Surely the point is the package manager asks for the SRV records and then uses the real hostnames?

So Squid should only ever see a request from the package manager for the REAL hostnames?

If DNS via Squid or otherwise is failing to process SRV records would that not be considered a misconfiguration of the network?

Offline basic612

  • Newbie
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: pkg.pfsense.org appears to be dead
« Reply #4 on: May 21, 2017, 04:25:54 pm »
Due to the firewall being internal and not having real world DNS resolution ability - ie. Internet access is only available via the upstream Squid proxy - the package manager will never get a response to the SRV records

Having our internal (LAN) dns servers not return SRV (or any other) records for external Internet connected hosts is not a misconfiguration of our network.

Offline Alex Atkin UK

  • Jr. Member
  • **
  • Posts: 44
  • Karma: +4/-0
    • View Profile
Re: pkg.pfsense.org appears to be dead
« Reply #5 on: May 26, 2017, 11:24:21 pm »
My point is that if Squid is not supporting modern DNS standards then its not the fault of pfSense who are using DNS correctly.

Online johnpoz

  • Hero Member
  • *****
  • Posts: 14431
  • Karma: +1336/-200
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: pkg.pfsense.org appears to be dead
« Reply #6 on: May 27, 2017, 05:02:14 am »
I have not played with squid in a long time.. But pretty sure there was a SRV redirector many years ago that was integrated..

http://wiki.squid-cache.org/Features/SrvDnsOriginServerLocation
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.3.4_p1 (work)
1x SG-4860 2.4.2-RELEASE (home)

Offline techbee

  • Jr. Member
  • **
  • Posts: 88
  • Karma: +0/-2
    • View Profile

Offline techbee

  • Jr. Member
  • **
  • Posts: 88
  • Karma: +0/-2
    • View Profile
Re: pkg.pfsense.org appears to be dead
« Reply #8 on: June 07, 2017, 09:27:31 am »
I am confused. If pkg is not meant to be accessed by browser then why not use files00.netgate.com directly instead of pkg.pfsense.org in the url that causes error.

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21387
  • Karma: +1432/-26
    • View Profile
Re: pkg.pfsense.org appears to be dead
« Reply #9 on: June 07, 2017, 09:34:37 am »
Because that is not how pkg works. It may not have been able to find files00 if there was a DNS error, for example.

If you think it should operate some other way, take it up with FreeBSD.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline pfAnlosa

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: pkg.pfsense.org appears to be dead
« Reply #10 on: June 11, 2017, 09:45:33 pm »
Howdy,

Just though I might jump in and help a little.  I had a similar problem when trying to upgrade from 2.2.5 to 2.3.4:

Code: [Select]
/root: pkg update
Updating pfSense-core repository catalogue...
pkg: Repository pfSense-core load error: access repo file(/var/db/pkg/repo-pfSense-core.sqlite) failed: No such file or directory
pkg: https://pkg.pfsense.org/pfSense_v2_3_4_amd64-core/meta.txz: No address record
repository pfSense-core has no meta file, using default settings
pkg: https://pkg.pfsense.org/pfSense_v2_3_4_amd64-core/packagesite.txz: No address record
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
pkg: Repository pfSense load error: access repo file(/var/db/pkg/repo-pfSense.sqlite) failed: No such file or directory
pkg: https://pkg.pfsense.org/pfSense_v2_3_4_amd64-pfSense_v2_3_4/meta.txz: No address record
repository pfSense has no meta file, using default settings
pkg: https://pkg.pfsense.org/pfSense_v2_3_4_amd64-pfSense_v2_3_4/packagesite.txz: No address record
Unable to update repository pfSense
Error updating repositories!

After reading this thread and monitoring my DNS requests I could see the request going to _https._tcp.pkg.pfsense.org.  Manually looking up the SRV record returns:

Code: [Select]
$ nslookup
> set type=srv
> _https._tcp.pkg.pfsense.org
Server:         MyDnsServer
Address:        MyDnsServer#53

Non-authoritative answer:
_https._tcp.pkg.pfsense.org     service = 10 10 443 files00.netgate.com.
_https._tcp.pkg.pfsense.org     service = 10 10 443 files01.netgate.com.


So the Package Manager needs to be able to look up the SRV record and it also needs to be able to look up the servers in the netgate.com domain.  My DNS server is set up with a limited number of domains that are allowed to be forwarded. pfsense.org was in the list but netgate.com was not. Once I added netgate.com things started working.

Code: [Select]
/root: pkg update
Updating pfSense-core repository catalogue...
pkg: Repository pfSense-core load error: access repo file(/var/db/pkg/repo-pfSense-core.sqlite) failed: No such file or directory
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01   
Fetching packagesite.txz: 100%    2 KiB   1.8kB/s    00:01   
Processing entries: 100%
pfSense-core repository update completed. 8 packages processed.
Updating pfSense repository catalogue...
pkg: Repository pfSense load error: access repo file(/var/db/pkg/repo-pfSense.sqlite) failed: No such file or directory
Fetching meta.txz: 100%    944 B   0.9kB/s    00:01   
Fetching packagesite.txz: 100%  121 KiB 124.0kB/s    00:01   
Processing entries: 100%
pfSense repository update completed. 443 packages processed.
All repositories are up to date.
/root: pkg update
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.

I removed netgate.com from my DNS forwarder afterward and the Installed Packages link stopped working (which I didn't expect) along with the Available Packages link (which I did expect).  Allowing netgate.com queries to be forwarded seems to be required to use the Package Manager links no matter what.

Wikipedia has some information on how SRV records work and why they exist.  The https lines in the error output are confusing because they contain a host name that doesn't exist with an A record and will return no answer.  What you really need to be able to retrieve is the SRV records to get the hosts that need to be reached.  I'm not sure how to do that through a Squid proxy since this is a DNS option and not a HTTP option. IMHO, your proxy/DNS configuration is not broken but you do need to find a way to pass on the SRV record to the firewall so it can send the appropriate HTTPS request to Squid to retrieve the package files. Or you'll need to manually update your package manager config files with the servers in the SRV record before starting an upgrade or package maintenance should they change in the future.

This seems to be a departure from the way previous versions of pfSense worked.  I'm not sure why this changed but it had me stumped for a few hours.  Almost rolled back to 2.2.5 but I really wanted my Snort to work again and google wasn't real helpful.  The SRV record hint from this thread helped guide me in the right direction.

Thank you.  Hope this helps others understand what's happening.

Offline bonnie222

  • Newbie
  • *
  • Posts: 1
  • Karma: +0/-0
    • View Profile
Re: pkg.pfsense.org appears to be dead
« Reply #11 on: August 15, 2017, 03:50:18 am »
I am having the same problem. I have followed the instructions from pfAlnso, but seems its not working on my box. Anyone help please

Code: [Select]
[2.3.3-RELEASE][root@pfSense.igss-africa.org]/root: pkg update -f
Updating pfSense-core repository catalogue...
pkg: Repository pfSense-core load error: access repo file(/var/db/pkg/repo-pfSense-core.sqlite) failed: No such file or directory
pkg: https://pkg.pfsense.org/pfSense_v2_3_3_amd60-core/meta.txz: No address record
repository pfSense-core has no meta file, using default settings
pkg: https://pkg.pfsense.org/pfSense_v2_3_3_amd60-core/packagesite.txz: No address record
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
pkg: Repository pfSense load error: access repo file(/var/db/pkg/repo-pfSense.sqlite) failed: No such file or directory
pkg: https://pkg.pfsense.org/pfSense_v2_3_3_amd60-pfSense_v2_3_3/meta.txz: No address record
repository pfSense has no meta file, using default settings
pkg: https://pkg.pfsense.org/pfSense_v2_3_3_amd60-pfSense_v2_3_3/packagesite.txz: No address record
Unable to update repository pfSense
Error updating repositories!
[2.3.3-RELEASE][root@pfSense.igss-africa.org]/root: nslookup
> set type=srv
> https._tcp.pkg.pfsense.org
Server:      127.0.0.1
Address:   127.0.0.1#53

** server can't find https._tcp.pkg.pfsense.org: REFUSED
> _https._tcp.pkg.pfsense.org
Server:      127.0.0.1
Address:   127.0.0.1#53

** server can't find _https._tcp.pkg.pfsense.org: REFUSED
> _https._tcp.pkg.pfsense.org
Server:      127.0.0.1
Address:   127.0.0.1#53

Online johnpoz

  • Hero Member
  • *****
  • Posts: 14431
  • Karma: +1336/-200
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: pkg.pfsense.org appears to be dead
« Reply #12 on: August 15, 2017, 05:09:34 am »
And REFUSED means your dns is not working.. So yeah going to have a problem!  Basically told you to F off..  Not that it couldn't find it - its not going to answer you.. You mess with your dns settings?  Are you using resolver or forwarder?  Your getting refused - not nx..

Your not even asking for the correct FQDN

_https._tcp.pkg.pfsense.org

there is _ in the front!!
« Last Edit: August 15, 2017, 05:15:31 am by johnpoz »
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.3.4_p1 (work)
1x SG-4860 2.4.2-RELEASE (home)

Offline Gertjan

  • Hero Member
  • *****
  • Posts: 2147
  • Karma: +165/-9
    • View Profile
Re: pkg.pfsense.org appears to be dead
« Reply #13 on: August 15, 2017, 05:30:48 am »
I am having the same problem. .....
Same code ....
Different setup for sure (among others : I never even touched the DNS settings because its works out of the box)
Works fine using the same commands:

Code: [Select]
[2.3.4-RELEASE][admin@pfsense.brit-hotel-fumel.net]/root: nslookup
> set type=srv
> _https._tcp.pkg.pfsense.org
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
_https._tcp.pkg.pfsense.org     service = 10 10 443 files00.netgate.com.
_https._tcp.pkg.pfsense.org     service = 10 10 443 files01.netgate.com.

Authoritative answers can be found from:
pfsense.org     nameserver = ns2.netgate.com.
pfsense.org     nameserver = ns1.netgate.com.
>

Offline pfAnlosa

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: pkg.pfsense.org appears to be dead
« Reply #14 on: August 19, 2017, 12:12:21 am »
bonnie222,

I have to agree with johnpoz.  You have something going on with your DNS configuration.  Have you checked your DNS settings to make sure you have a DNS server configured?  Can you successfully look up A records for sites like www.pfsense.org and files00.netgate.com.  If you can't do A record lookups you certainly aren't going to be able to do SRV record lookups.

There is an option under 'System -> General Setup -> DNS Server Settings' that allows your ISP DHCP settings to override the local DNS server settings.  If you are using DHCP from your ISP and using their DNS servers make sure this is checked. This should not be unchecked unless you have your own DNS servers, you want to use different DNS servers (like google), or your ISP doesn't provide DNS via DHCP.  If the box is unchecked you have to manually configure DNS servers.

If your box has Internet access to google's DNS servers you can test by using 8.8.8.8  as a DNS Server, uncheck 'DNS Server Override" and check the "Disable DNS Forwarder'.  This will skip using the pfSense box for DNS and go directly to Google's DNS server.  If this works then either your ISP isn't providing a DNS server via DHCP and you need to configure a DNS server manually or your own DNS server is filtering what you are allowed to resolve. 

Hope this helps.