Netgate SG-1000 microFirewall

Author Topic: Admin password changed itself. Twice. Yes it did.  (Read 4427 times)

0 Members and 1 Guest are viewing this topic.

Offline mhvmhv

  • Newbie
  • *
  • Posts: 8
  • Karma: +4/-6
    • View Profile
Admin password changed itself. Twice. Yes it did.
« on: January 23, 2018, 12:32:09 pm »
[I should know better]

This situation is alarming and I need some serious expertise and guidance from the heavy hitters here:

I am configuring a new setup, dual HA firewalls, dual WAN, full LACP/CARP redundancy to a stacked Cisco L3 switch. Using this hardware, which came pre-installed with PFSense: https://www.amazon.ca/gp/product/B0733H75TB/ref=oh_aui_detailpage_o01_s00?ie=UTF8&psc=1

I upgraded (using the console, using menu item 13) to 2.4.2.

Yesterday, I successfully configured IPSec for remote access. When I returned to the system after working on something else, the admin password no longer worked. After trying the default and other user passwords (to eliminate the possibility of me having accidentally changing the password), I changed the password form the console and moved on to refine the VPN config. I ended up not making any changes, however, being called away to a meeting.

This morning the admin password has changed again. No, Caps Lock is not on. No I don't sleepwalk. Nobody else has access to the gear. I actually suspect something rather sinister is going on...

I can run paranoid scenarios with the best of them, but I want help to gather some facts to target the investigation of WTH is going on here. Please ask your questions; I will answer them as best I can. Given that this is a time-sensitive issue, I would respectfully ask that advice which is neither constructive nor expert be saved for the recrimination phase after the issue is solved!

Help.
« Last Edit: January 24, 2018, 01:18:14 pm by mhvmhv »

Online johnpoz

  • Hero Member
  • *****
  • Posts: 15172
  • Karma: +1413/-206
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Admin password changed itself. Twice. Yes it did.
« Reply #1 on: January 23, 2018, 01:02:15 pm »
well lets see the hash of the root password, and then after it changes on its own lets see the hash again.

So you can use vipw to view the hash or look in /etc/master.passwd with cat or something, etc..
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.4.2-RELEASE-p1 (work)
1x SG-4860 2.4.2-RELEASE-p1 (home)

Offline ivor

  • Administrator
  • Hero Member
  • *****
  • Posts: 730
  • Karma: +154/-135
    • View Profile
    • Netgate
Re: Admin password changed itself. Twice. Yes it did.
« Reply #2 on: January 23, 2018, 01:11:58 pm »
I upgraded (using the console, using menu item 13) to 2.4.2.

That's your problem right there. You purchased firewall hardware from a 3rd party which had pfSense pre-installed. It is strictly against our policy to use our trademarks to sell 3rd party hardware and to pre-install / sell pfSense. One of many reasons we have that policy is also because one cannot trust a 3rd party to deliver unmodified or vanilla pfSense.

I suggest you do a clean re-install but if there is a malicious factor in place here, you cannot trust that hardware considering the events which have occurred.
Need help fast? Commercial support: https://www.netgate.com/support/

Online johnpoz

  • Hero Member
  • *****
  • Posts: 15172
  • Karma: +1413/-206
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Admin password changed itself. Twice. Yes it did.
« Reply #3 on: January 23, 2018, 01:14:21 pm »
I think you quoted the wrong section ivor ;)  You prob wanted to quote the part where they state came preloaded... Which I missed on my first read thru.. nice catch

Quote
which came pre-installed with PFSense:
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.4.2-RELEASE-p1 (work)
1x SG-4860 2.4.2-RELEASE-p1 (home)

Offline ivor

  • Administrator
  • Hero Member
  • *****
  • Posts: 730
  • Karma: +154/-135
    • View Profile
    • Netgate
Re: Admin password changed itself. Twice. Yes it did.
« Reply #4 on: January 23, 2018, 01:18:04 pm »
No, not in this case :) I was pointing out the issue on OP's end.
Need help fast? Commercial support: https://www.netgate.com/support/

Online johnpoz

  • Hero Member
  • *****
  • Posts: 15172
  • Karma: +1413/-206
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Admin password changed itself. Twice. Yes it did.
« Reply #5 on: January 23, 2018, 01:23:44 pm »
Not sure why anyone would buy such a box, for a few dollars more you can get a sg-3100 direct from netgate..
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.4.2-RELEASE-p1 (work)
1x SG-4860 2.4.2-RELEASE-p1 (home)

Offline mhvmhv

  • Newbie
  • *
  • Posts: 8
  • Karma: +4/-6
    • View Profile
Re: Admin password changed itself. Twice. Yes it did.
« Reply #6 on: January 23, 2018, 01:29:46 pm »
I upgraded (using the console, using menu item 13) to 2.4.2.

That's your problem right there. You purchased firewall hardware from a 3rd party which had pfSense pre-installed. It is strictly against our policy to use our trademarks to sell 3rd party hardware and to pre-install / sell pfSense. One of many reasons we have that policy is also because one cannot trust a 3rd party to deliver unmodified or vanilla pfSense.

I suggest you do a clean re-install but if there is a malicious factor in place here, you cannot trust that hardware considering the events which have occurred.

Ok, first of all thank you for the reply.
Second, I had no idea of the policy when I purchased hardware in good faith. I see the issue you identify now, and I understand the policy. That said, here I am and I need to resolve this.

What are the chances that the hardware itself is compromised? That seems less likely than a modified PFSense install.
Is there a possibility that my backups could be compromised? If I do a clean install and restore from backup, is there a risk still?


Online johnpoz

  • Hero Member
  • *****
  • Posts: 15172
  • Karma: +1413/-206
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: Admin password changed itself. Twice. Yes it did.
« Reply #7 on: January 23, 2018, 01:31:04 pm »
Backups are just XML files - you could look through it to see exactly what in there.
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Please don't PM me for personal help
- if you want to say thanks applaud or https://www.freebsdfoundation.org/donate/
1x SG-2440 2.4.2-RELEASE-p1 (work)
1x SG-4860 2.4.2-RELEASE-p1 (home)

Offline mhvmhv

  • Newbie
  • *
  • Posts: 8
  • Karma: +4/-6
    • View Profile
Re: Admin password changed itself. Twice. Yes it did.
« Reply #8 on: January 23, 2018, 01:35:02 pm »
Not sure why anyone would buy such a box, for a few dollars more you can get a sg-3100 direct from netgate..

Simple: I didn't know such devices were for sale by NetGate, and even if I had, I was unaware of the restriction on selling devices preloaded. I just went looking for a 4-port device with adequate hardware. It was actually the purchase of these devices that led me to PFSense; I was originally going to install a different firewall product.

Moving forward here: This is still a time-sensitive install. Is there a chance that Netgate can get two devices to me within a couple days?

Offline ivor

  • Administrator
  • Hero Member
  • *****
  • Posts: 730
  • Karma: +154/-135
    • View Profile
    • Netgate
Re: Admin password changed itself. Twice. Yes it did.
« Reply #9 on: January 23, 2018, 01:35:21 pm »
What are the chances that the hardware itself is compromised? That seems less likely than a modified PFSense install.
Is there a possibility that my backups could be compromised? If I do a clean install and restore from backup, is there a risk still?

OS might be compromised but we don't know if hardware is also. Considering what happened I don't see how one can trust the hardware. I am not sure you will find a BIOS for appliance board because it's a generic Aliexpress J900 board (so you can reflash it). As Johnpoz noticed, backups are in XML and one can analyze them but I would do a clean install and configure from scratch. You might want to talk to your Amazon seller about it, they're the ones responsible. I would want a refund.
Need help fast? Commercial support: https://www.netgate.com/support/

Offline ivor

  • Administrator
  • Hero Member
  • *****
  • Posts: 730
  • Karma: +154/-135
    • View Profile
    • Netgate
Re: Admin password changed itself. Twice. Yes it did.
« Reply #10 on: January 23, 2018, 01:36:16 pm »
[Moving forward here: This is still a time-sensitive install. Is there a chance that Netgate can get two devices to me within a couple days?

Yes, we have several build time and shipping options. http://store.netgate.com/
Need help fast? Commercial support: https://www.netgate.com/support/

Offline mhvmhv

  • Newbie
  • *
  • Posts: 8
  • Karma: +4/-6
    • View Profile
Re: Admin password changed itself. Twice. Yes it did.
« Reply #11 on: January 23, 2018, 01:53:33 pm »
[Moving forward here: This is still a time-sensitive install. Is there a chance that Netgate can get two devices to me within a couple days?

Yes, we have several build time and shipping options. http://store.netgate.com/

Ok. Working on it now.

Thanks very much. Always fun to learn a lesson... see you guys around here, I'm sure.

Offline ivor

  • Administrator
  • Hero Member
  • *****
  • Posts: 730
  • Karma: +154/-135
    • View Profile
    • Netgate
Re: Admin password changed itself. Twice. Yes it did.
« Reply #12 on: January 23, 2018, 01:55:27 pm »
You're welcome. Please feel free to contact our sales if you need assistance with your order! https://www.netgate.com/company/contact-us.html
Need help fast? Commercial support: https://www.netgate.com/support/

Offline jwt

  • Administrator
  • Sr. Member
  • *****
  • Posts: 369
  • Karma: +104/-34
    • View Profile
Re: Admin password changed itself. Twice. Yes it did.
« Reply #13 on: January 23, 2018, 02:45:44 pm »
Not sure why anyone would buy such a box, for a few dollars more you can get a sg-3100 direct from netgate..

Simple: I didn't know such devices were for sale by NetGate, and even if I had, I was unaware of the restriction on selling devices preloaded. I just went looking for a 4-port device with adequate hardware. It was actually the purchase of these devices that led me to PFSense; I was originally going to install a different firewall product.


At some point an event like this will occur that will be the proverbial straw that breaks the camel's back.

Why would *anyone* think it's OK to sell a device with pfsense preloaded?   There is only one answer: Greed.   These vendors take what we engineer, package and test and use it to sell their crapware.

In the entire history of the pfSense project there is a single company that has always paid to pre-load pfSense.  That company is Netgate.

I'm still OK with members of the community building their own firewall using pfSense software.

But when will the community learn that hardware sales are what pays for the engineering time and talent, the testing, the documentation, and the infrastructure (on-line and offline) that goes into making pfSense software available to them?



Offline mhvmhv

  • Newbie
  • *
  • Posts: 8
  • Karma: +4/-6
    • View Profile
Re: Admin password changed itself. Twice. Yes it did.
« Reply #14 on: January 23, 2018, 03:07:46 pm »

At some point an event like this will occur that will be the proverbial straw that breaks the camel's back.

Why would *anyone* think it's OK to sell a device with pfsense preloaded?   There is only one answer: Greed.   These vendors take what we engineer, package and test and use it to sell their crapware.

In the entire history of the pfSense project there is a single company that has always paid to pre-load pfSense.  That company is Netgate.

I'm still OK with members of the community building their own firewall using pfSense software.

But when will the community learn that hardware sales are what pays for the engineering time and talent, the testing, the documentation, and the infrastructure (on-line and offline) that goes into making pfSense software available to them?

There seems to be something more sinister than simple greed at play in this case. Resetting the Admin password? Oh and by the way, disabling all logging - I thought I just wasn't configuring it right, but it wouldn't log anything, so I couldn't see the malicious traffic. My first clue should really have been when the "Enable IPSec" checkbox wasn't there. I assumed just a version difference between the docs and my install, but I would suggest it's more likely that there was a VPN built-in as an exploit vector, and the absence of the checkbox prevented it from being disabled.

I got lucky, by some measure at least. Lucky to know enough to set it up on a discreet WAN segment so the attack was mitigated. Lucky I didn't config it and just let it run unobserved - much less install it at the production site. Lucky the bad actors were not especially subtle or clever.

I must say, this spooked me. It's been a long time since an attack got that far in.

My order (including support) should help prop up the product environment a bit.