pfSense Support Subscription

Author Topic: No clue what happened... unable to make any changes via GUI or SSH to box  (Read 229 times)

0 Members and 1 Guest are viewing this topic.

Offline dalequalls

  • Newbie
  • *
  • Posts: 12
  • Karma: +0/-0
    • View Profile
It's been a long day...

This started out with some folks from Germany not being able to reach a server that I have here at my office.  I use pfBlockerNG and figured it was that since I block Germany.  Using geoscreenshot.com I could see that a connection from Germany wasn't getting in.  I turned off pfBlockerNG (and Snort) and I was still having the issue.  I could see the IP of the German folks getting blocked.

Couple this with the problem that I constant get the 502 Bad Gateway error and my crap day just got crappier.

WTH, I'll reboot...  Bad idea.  It took no less than 4 reboots to get the system running again (one of them forced a fsck which took a while).

I'm now functional but just barely.  I can't make any gui changes at all.  I've also lost the ability to SSH into the box.  Anytime I try to make a change I receive this:

<error>
Fatal error: Cannot create references to/from string offsets nor overloaded objects in /etc/inc/xmlparse.inc on line 103 Call Stack: 0.0002 124668 1. {main}() /usr/local/www/system_advanced_admin.php:0 0.0341 850152 2. write_config() /usr/local/www/system_advanced_admin.php:296 0.0948 979640 3. cleanup_backupcache() /etc/inc/config.lib.inc:593 1.9734 1371640 4. parse_xml_config() /etc/inc/config.lib.inc:859 1.9734 1371668 5. parse_xml_config_raw() /etc/inc/xmlparse.inc:178 2.0390 1737880 6. xml_parse() /etc/inc/xmlparse.inc:217 2.0390 1738072 7. startElement() /etc/inc/xmlparse.inc:217 PHP ERROR: Type: 1, File: /etc/inc/xmlparse.inc, Line: 103, Message: Cannot create references to/from string offsets nor overloaded objects
</error>

I try to enable SSH from the console and it says it's already enabled so it prompts me to disable, I figure what the heck, disable and then re-enable.  Trying this gets me a similar error.

<error>
Fatal error: Cannot create references to/from string offsets nor overloaded objects in /etc/inc/xmlparse.inc on line 103...
</error>

Thoughts?  Corrupted config maybe?  I have a pretty recent backup but I'm afraid to apply it.  Will it interrupt internet access for my folks?

Thanks in advance for any guidance

Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9046
  • Karma: +1031/-306
    • View Profile
I would guess failing storage.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline dalequalls

  • Newbie
  • *
  • Posts: 12
  • Karma: +0/-0
    • View Profile
Thanks for the prompt reply!

I'm using GEOM but I can't add the widget to the dashboard to see if the array is healthy.  Is there a way to from the console?

Offline dalequalls

  • Newbie
  • *
  • Posts: 12
  • Karma: +0/-0
    • View Profile
They look fine in Diagnostics - GEOM Mirrors...


Offline Derelict

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 9046
  • Karma: +1031/-306
    • View Profile
Hmmm. Something else must have gotten corrupted. About all I can think of doing is dropping to single user mode and running a few /sbin/fsck -y / commands to see if that helps and if not a reinstall / restore.

I would take a new config backup if that is working.
Las Vegas, Nevada, USA
Use this diagram to describe your issue.
The pfSense Book is now available for just $24.70!
Do Not PM For Help! NO_WAN_EGRESSTM

Offline dalequalls

  • Newbie
  • *
  • Posts: 12
  • Karma: +0/-0
    • View Profile
Thanks, I'll give that a try!

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 21284
  • Karma: +1415/-26
    • View Profile
It still sounds like failing storage hardware.

All of the symptoms line up, and though it may have been able to rebuild the mirror, it's probably a ticking time bomb.

Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline dalequalls

  • Newbie
  • *
  • Posts: 12
  • Karma: +0/-0
    • View Profile
Re: No clue what happened... unable to make any changes via GUI or SSH to box
« Reply #7 on: September 01, 2017, 09:13:23 am »
Thanks, I'll replace the drives and report back.

Offline dalequalls

  • Newbie
  • *
  • Posts: 12
  • Karma: +0/-0
    • View Profile
Re: No clue what happened... unable to make any changes via GUI or SSH to box
« Reply #8 on: September 12, 2017, 01:36:16 pm »
I broke the mirror and swapped out ada1.  It finished the rebuild so I broke the mirror again and replaced ada0.  After a rebuild I'm still unable to make changes to anything.  Here's what I receive:

Fatal error: Cannot create references to/from string offsets nor overloaded objects in /etc/inc/xmlparse.inc on line 103 Call Stack: 0.0002 128780 1. {main}() /usr/local/www/status_logs_settings.php:0 0.0340 852800 2. write_config() /usr/local/www/status_logs_settings.php:195 0.0948 982232 3. cleanup_backupcache() /etc/inc/config.lib.inc:593 2.0669 1374200 4. parse_xml_config() /etc/inc/config.lib.inc:859 2.0669 1374228 5. parse_xml_config_raw() /etc/inc/xmlparse.inc:178 2.1324 1740504 6. xml_parse() /etc/inc/xmlparse.inc:217 2.1324 1740696 7. startElement() /etc/inc/xmlparse.inc:217 PHP ERROR: Type: 1, File: /etc/inc/xmlparse.inc, Line: 103, Message: Cannot create references to/from string offsets nor overloaded objects

Any additional thoughts?

TIA!

Dale

Offline doktornotor

  • Hero Member
  • *****
  • Posts: 8553
  • Karma: +956/-278
  • Not a pfSense employee, they cannot fire me...
    • View Profile
Re: No clue what happened... unable to make any changes via GUI or SSH to box
« Reply #9 on: September 12, 2017, 01:43:20 pm »
Yeah, reinstall the thing and restore a config backup. Rebuilding corrupted mirrors does no good.
Do NOT PM for help!

Offline dalequalls

  • Newbie
  • *
  • Posts: 12
  • Karma: +0/-0
    • View Profile
Re: No clue what happened... unable to make any changes via GUI or SSH to box
« Reply #10 on: September 12, 2017, 01:45:58 pm »
Dang, I was afraid that was going to be the next step...

Thanks!