Netgate SG-1000 microFirewall

Author Topic: [SOLVED] Update 2.3.4 to 2.4 failed ("Unable to check for updates", ...)  (Read 5748 times)

0 Members and 1 Guest are viewing this topic.

Offline felix.wolfsteller

  • Newbie
  • *
  • Posts: 17
  • Karma: +2/-0
    • View Profile
Hey there

we converted a NanoBSD into a full 64b installation using the procedure from the wiki (boot slices, script).
We ran the update, which failed due to timeouts during package download 2 or 3 times.
After reboot, the GUI still shows version "2.3.4-RELEASE-p1 (amd64)".

Afterwards we
  • disabled the RAM Disk feature (shouldnt this be proposed in the nanobsd->full-wiki), as logging behaviour and other stuff seemed weird.
  • deleted a /var/db/pkg symlink mess as found in another thread
  • switched System->Update->Update Settings branch back and forth (no difference)
  • Tried to update via pfSense-upgrade and pkg

So far the update attempts fail.
I will post some command output which seems helpful to debug the situation (unfortunately I have no deeper BSD knowledge)

Code: [Select]
$ pkg info
pkg: Warning: Major OS version upgrade detected.  Running "pkg-static install -f pkg" recommended

Code: [Select]
$ pkg-static install -f pkg
pkg-static install -f pkg
pkg-static: Warning: Major OS version upgrade detected.  Running "pkg-static install -f pkg" recommended
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
pkg: 1.10.1_1 [pfSense]
# .... we hit 'y' and 'Enter', nothing happens, process seem to hang



Code: [Select]
$  pfSense-upgrade -d
pkg-static: Warning: Major OS version upgrade detected.  Running "pkg-static install -f pkg" recommended
pkg-static: No package(s) matching pfSense-kernel-*
pkg-static: Warning: Major OS version upgrade detected.  Running "pkg-static install -f pkg" recommended
ERROR: It was not possible to identify which pfSense kernel is installed



Code: [Select]
$ ls -lha /var/db/pkg
total 1559
drwxr-xr-x   2 root  wheel   512B Oct 21 16:32 .
drwxr-xr-x  13 root  wheel   1.5K Oct 21 15:46 ..
-rw-r--r--   1 root  wheel   352K Oct 21 16:32 local.sqlite
-rw-r--r--   1 root  wheel   246B Oct 10 14:19 pfSense-core.meta
-rw-r--r--   1 root  wheel   246B Oct 21 12:57 pfSense.meta
lrwxr-xr-x   1 root  wheel    21B Oct 21 02:27 pkg -> ../../root/var/db/pkg
-rw-r--r--   1 root  wheel   208K Oct 10 14:19 repo-pfSense-core.sqlite
-rw-r--r--   1 root  wheel   984K Oct 21 12:57 repo-pfSense.sqlite

Any idea how to debug this further?

Steps to resolve

1. Cleaned up link-mess in /var/cache/pkg
Code: [Select]
$ ls -lha /var/cache/pkg
lrwxr-xr-x  1 root  wheel    24B Oct 21 02:27 /var/cache/pkg -> ../../root/var/cache/pkg
[/li][/list]

2. Delete an ".empty" file in /usr/local/share/pfSense/keys/pkg/revoked/ (check below for other key-issues)

3. Installed packages via pkg-static and upgrade
Code: [Select]
pkg-static update -f
pkg-static install -f pkg
pkg-static install -y pfSense pfSense-base pfSense-repo pfSense-kernel-pfSense pfSense-rc pfSense-repo pfSense-upgrade

pkg-static unlock pfSense-kernel-pfSense
pkg-static upgrade -f
pkg-static lock pfSense-kernel-pfSense

pfSense-upgrade -d

4. Installed and Removed acme package via GUI to get rid of crash reports

Thanks everybody who helped.
Btw update to 2.4.1 went smooth afterwards, too.
« Last Edit: October 26, 2017, 01:39:41 am by felix.wolfsteller »

Offline discostur

  • Newbie
  • *
  • Posts: 12
  • Karma: +0/-0
    • View Profile
Re: Update 2.3.4 to 2.4 failed ("Unable to check for updates", ...)
« Reply #1 on: October 21, 2017, 10:08:30 am »
Hi,

having exact the same problem!

Offline PiBa

  • Hero Member
  • *****
  • Posts: 871
  • Karma: +140/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: Update 2.3.4 to 2.4 failed ("Unable to check for updates", ...)
« Reply #2 on: October 21, 2017, 02:39:59 pm »
Internet connectivity from pfSense box itself works fine?

Check if you have any 0 byte packages in the /var/pkg/cache , and perhaps a file called 'empty' in the

If you see any zero byte files in the directories below, such as a file named ".empty", remove them.
/usr/local/share/pfSense/keys/pkg/trusted/
/usr/local/share/pfSense/keys/pkg/revoked/
https://forum.pfsense.org/index.php?topic=136950.msg749972#msg749972


If above doesn't help, perhaps run "truss -f -o /root/pkg_truss.log pkg-static install -f pkg" and attach the file that gets created. It might tell what pkg is trying to do..

Offline JohnnyBeGood

  • Full Member
  • ***
  • Posts: 161
  • Karma: +0/-0
    • View Profile
Re: Update 2.3.4 to 2.4 failed ("Unable to check for updates", ...)
« Reply #3 on: October 21, 2017, 03:44:26 pm »
Hi all,

Had same issue while trying to upgrade using web interface. Rebooted pfSense using web interface and tried again and it worked!
Here's the successful upgrade log https://pastebin.com/b0j8JEzy
I like to fill my tub up with water, then turn the shower on and act like I'm in a submarine that's been hit!

Offline felix.wolfsteller

  • Newbie
  • *
  • Posts: 17
  • Karma: +2/-0
    • View Profile
Re: Update 2.3.4 to 2.4 failed ("Unable to check for updates", ...)
« Reply #4 on: October 22, 2017, 05:49:11 am »
Internet connectivity from pfSense box itself works fine?
Yes, it did.

Check if you have any 0 byte packages in the /var/pkg/cache , and perhaps a file called 'empty' in the

You mean /var/cache/pkg:
Code: [Select]
$ ls -lha /var/cache/pkg
lrwxr-xr-x  1 root  wheel    24B Oct 21 02:27 /var/cache/pkg -> ../../root/var/cache/pkg

Probably again such a symlink thing (target does not exist). I deleted (and mkdir'd) /var/cache/pkg; afterwards `pkg-static install -f pkg` succeeded.

Now /var/cache/pkg looks like this:
Quote
$ ls -lha /var/cache/pkg/
total 2917
drwxr-xr-x  2 root  wheel   512B Oct 22 12:37 .
drwxr-xr-x  3 root  wheel   512B Oct 22 12:37 ..
-rw-r--r--  1 root  wheel   2.8M Oct  8 23:44 pkg-1.10.1_1-5a32648e84.txz
lrwxr-xr-x  1 root  wheel    27B Oct 22 12:37 pkg-1.10.1_1.txz -> pkg-1.10.1_1-5a32648e84.txz

If you see any zero byte files in the directories below, such as a file named ".empty", remove them.
/usr/local/share/pfSense/keys/pkg/trusted/
/usr/local/share/pfSense/keys/pkg/revoked/
https://forum.pfsense.org/index.php?topic=136950.msg749972#msg749972
In "trusted" there are two 95B files, in "revoked" one .empty, which i deleted.

If above doesn't help, perhaps run "truss -f -o /root/pkg_truss.log pkg-static install -f pkg" and attach the file that gets created. It might tell what pkg is trying to do..
I did not do this because "pkg-static install -f pkg" succeeded after the cache-symlink-cleanup (see above)

Thanks a lot PiBa! At least this allowed to go one step further. The GUI still shows "unable to check for updates", but at least the pkg thing seems to be clean now. I guess I could start the update via shell (pfSense-upgrade?) now - will try this when a slightly more relaxed time slot falls into my life. soon.

Offline felix.wolfsteller

  • Newbie
  • *
  • Posts: 17
  • Karma: +2/-0
    • View Profile
Re: Update 2.3.4 to 2.4 failed ("Unable to check for updates", ...)
« Reply #5 on: October 22, 2017, 06:01:26 am »
I though I'd at least check if pfSense-upgrade would show me available package updates. Following happens:

Code: [Select]
$ pfSense-upgrade
pkg-static: No package(s) matching pfSense-kernel-*
>>> Updating repositories metadata...
Updating pfSense-core repository catalogue...
pkg-static: Repository pfSense-core has a wrong packagesite, need to re-create database
Fetching meta.txz: . done
Fetching packagesite.txz: . done
Processing entries: . done
pfSense-core repository update completed. 7 packages processed.
Updating pfSense repository catalogue...
pkg-static: Repository pfSense has a wrong packagesite, need to re-create database
Fetching meta.txz: . done
Fetching packagesite.txz: .......... done
Processing entries: .......... done
pfSense repository update completed. 485 packages processed.
All repositories are up to date.
ERROR: Unable to compare version of pfSense-repo

That seems not so dramatic, but following scares me quite a lot:

Code: [Select]
$ pkg info
Shared object "libssl.so.8" not found, required by "pkg"
[/quote]

[code]
$ find / -name "libssl.so*"
/usr/lib/libssl.so.7
/usr/lib/libssl.so

Code: [Select]
$ pkg update
Shared object "libssl.so.8" not found, required by "pkg"

I guess I should not reboot that box now ... ? :)
Any help appreciated. The cheapest try would be to symlink `ln -s /usr/lib/libssl.so /usr/lib/libssl.so.8` - but this really gives me goosebumps.

Offline Grimson

  • Sr. Member
  • ****
  • Posts: 429
  • Karma: +63/-7
    • View Profile
Re: Update 2.3.4 to 2.4 failed ("Unable to check for updates", ...)
« Reply #6 on: October 22, 2017, 06:55:57 am »
Just do a fresh 2.4 install and recover/restore the config backup, saves a lot of time and you are sure the system is in a clean state.

Offline PiBa

  • Hero Member
  • *****
  • Posts: 871
  • Karma: +140/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: Update 2.3.4 to 2.4 failed ("Unable to check for updates", ...)
« Reply #7 on: October 22, 2017, 07:49:39 am »
A reinstall would probably be best indeed.

If you want to keep trying to salvage the current running installation then some commands to try below. But be prepared to reinstall anyhow as it might break functionality further..

Code: [Select]
pkg-static update -f
pkg-static install -f pkg
pkg-static install -y pfSense pfSense-base pfSense-repo pfSense-kernel-pfSense pfSense-rc pfSense-repo pfSense-upgrade

pkg-static unlock pfSense-kernel-pfSense
pkg-static upgrade -f
pkg-static lock pfSense-kernel-pfSense

pfSense-upgrade -d

Offline felix.wolfsteller

  • Newbie
  • *
  • Posts: 17
  • Karma: +2/-0
    • View Profile
Re: Update 2.3.4 to 2.4 failed ("Unable to check for updates", ...)
« Reply #8 on: October 22, 2017, 08:08:45 am »
A reinstall would probably be best indeed.

If you want to keep trying to salvage the current running installation then some commands to try below. But be prepared to reinstall anyhow as it might break functionality further..

Code: [Select]
pkg-static update -f
pkg-static install -f pkg
pkg-static install -y pfSense pfSense-base pfSense-repo pfSense-kernel-pfSense pfSense-rc pfSense-repo pfSense-upgrade

pkg-static unlock pfSense-kernel-pfSense
pkg-static upgrade -f
pkg-static lock pfSense-kernel-pfSense

pfSense-upgrade -d

Okay. Our users are allergic to downtimes, so this needs a bit of preparation. Will post the outcome and edit original question if solution is found within PiBas Wisdom (thanks a lot!).

Offline felix.wolfsteller

  • Newbie
  • *
  • Posts: 17
  • Karma: +2/-0
    • View Profile
Re: Update 2.3.4 to 2.4 failed ("Unable to check for updates", ...)
« Reply #9 on: October 23, 2017, 12:19:58 am »
Awesome PiBa, that worked (I executed all of the commands). Got quite some further warnings about libssl, libcrypto etc pp but the box rebooted and the logs look fine to me! The GU shows "version 2.4.0" and "latest version". Is there anything to do to clean thing up?

Once I gained confidence that the box behaves stable, I will update the original Post.

Offline felix.wolfsteller

  • Newbie
  • *
  • Posts: 17
  • Karma: +2/-0
    • View Profile
Re: Update 2.3.4 to 2.4 failed ("Unable to check for updates", ...)
« Reply #10 on: October 23, 2017, 12:46:28 am »
This is the relevant crash info (appears on every reboot). Creating a symlink to libssl.so.{8,7} leads to similar issues (I think then libcrypto.so is missing).
I found various other threads about this issue (https://forum.pfsense.org/index.php?topic=137941 / ACME, https://forum.pfsense.org/index.php?topic=137939) but did not dare to dive into it - for now the system seems okay. I cross fingers for a 2.4.1 update moving everything back into place.

Code: [Select]
Crash report begins.  Anonymous machine information:

amd64
11.1-RELEASE-p1
FreeBSD 11.1-RELEASE-p1 #83 r313908+d77c47fe50c(RELENG_2_4_0): Tue Oct 10 06:48:42 CDT 2017     root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense

Crash report details:

PHP Errors:
[23-Oct-2017 07:25:06 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:06 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:08 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:34 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:34 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:34 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:36 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:36 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:37 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:37 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0
[23-Oct-2017 07:25:43 Europe/Berlin] PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/local/lib/php/20131226/ssh2.so' - Shared object "libssl.so.7" not found, required by "libssh2.so.1" in Unknown on line 0

No FreeBSD crash data found.

Offline PiBa

  • Hero Member
  • *****
  • Posts: 871
  • Karma: +140/-1
  • PiBa-NL(on IRC)
    • View Profile
Re: Update 2.3.4 to 2.4 failed ("Unable to check for updates", ...)
« Reply #11 on: October 23, 2017, 10:17:39 am »
Installing acme package from system\packagemanager and then removing it 'should' be a safe action to take..

Offline echu2016

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: Update 2.3.4 to 2.4 failed ("Unable to check for updates", ...)
« Reply #12 on: October 24, 2017, 12:37:40 pm »
I got crazy to with 4 PFSense VMs.....
After a day, I realized the way to go...
The problem begins when following the suggestion "Major OS blahblah..." and installing pkg again.... Because, it installs FreeBSD 11 pkg package.
So, to make the upgrade work again, and not to deal with a manual upgrade what I did (worked) was:
Code: [Select]
cat /usr/local/sbin/pkg-static > /usr/sbin/pkgThis is, switching back (manually) to pkg-static version.
After issuing this command, upgrade should be done automatically as usual.
Try to issue a pkg -v before and after the command I wrote here, notice de difference.
Regards,

BlueKobold

  • Guest
Re: Update 2.3.4 to 2.4 failed ("Unable to check for updates", ...)
« Reply #13 on: October 25, 2017, 02:59:08 pm »
Quote
we converted a NanoBSD into a full 64b installation using the procedure from the wiki (boot slices, script).
Netgates Blog - pfSense version 2.4.0 is here

Important Information
A few noteworthy items about running or upgrading to pfSense 2.4:

* 32-bit x86 and NanoBSD have been deprecated and are not supported on pfSense 2.4.
Hardware capable of running 64-bit images should be reinstalled with a 64-bit version.
* 32-bit x86 hardware can continue to run pfSense software version 2.3.x, which will receive
security updates for at least a year after pfSense 2.4.0-RELEASE.
* NanoBSD installs on 64-bit hardware must be changed to run a full installation. This can be
accomplished by reinstalling or by following the manual upgrade procedure to convert NanoBSD to a full installation.
* To use ZFS, a reinstall of the operating system is required. It is not possible to upgrade in-place from UFS to ZFS at this time.
* Wireless interfaces must be created on the Wireless tab under Interfaces > Assignments before they are
available for assignment
* Some hardware devices may not boot pfSense 2.4.0 installation images, for example, due to UEFI compatibility changes.
These are primarily BIOS issues and not issues with the installer images. Upgrading in place from pfSense 2.3.x typically
allows affected hardware to run version pfSense 2.4.x. In some cases, manually adjusting the installer images can allow
the hardware to boot.


A clean, fresh and full install was causing only ~30 minutes with a configuration play back!
I was choosing to install and use ZFS and so it was a "must be" with the fresh install but
without any problems, the upgrade from 2.4.0 to 2.4.1 was also not a nice trip here,
and I mean this really, so once more again I was installing fresh, and full and now I
am staying here on 2.4.1 so long as I can do it!

Quote
We ran the update, which failed due to timeouts during package download 2 or 3 times.
Deinstalling all packages and reinstalling all packages will be needing the same time to do a clean, fresh and full
install and play back the config.

Quote
After reboot, the GUI still shows version "2.3.4-RELEASE-p1 (amd64)".
Try out a fresh and full install against all other things, it not really popular but faster then all other methods
brings you fully ZFS support and you get not into trouble with IPSec VPN too likes much other users.

Offline felix.wolfsteller

  • Newbie
  • *
  • Posts: 17
  • Karma: +2/-0
    • View Profile
Re: [SOLVED] Update 2.3.4 to 2.4 failed ("Unable to check for updates", ...)
« Reply #14 on: October 26, 2017, 01:44:01 am »
Thanks everybody. Updated the issue and marked as solved. I was able to have a (for now) stable box without a fresh install.
I think it remains very important that updates are possible (think of remote hardware) and the devs and users should know where updates are tricky - to solve issues and improve behaviour in the future.
Of course fresh installations might be easier and faster and thanks god export and import of the configuration works.