The pfSense Store

Author Topic: SG-2220 Boot Issue (No Console)  (Read 782 times)

0 Members and 1 Guest are viewing this topic.

Offline Carboxyl

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
SG-2220 Boot Issue (No Console)
« on: July 29, 2017, 09:51:52 am »
Hi there,

I have an SG-2220 running up-to-date pfSense. Today I noticed low speed over the WAN. I rebooted the device from the web GUI to see if that would help. The SG-2220 didn't restart properly and seemed to get stuck. I waited for a bit, but nothing improved. I tried physically power cycling the device and still no luck.

At that point I realised something odd was going on. Here are the symptoms:
  • After every power cycle the status light lights up orange, and both Ethernet interfaces' green LEDs all go on. (So an orange status LED and all four green Ethernet LEDs go on solidly).
  • Doing a hardware factory reset (holding the reset button through a power-up for at least a minute) does nothing. The LEDs respond in exactly the same way.
  • There is no sign of the console. I tried PuTTY on Windows, screen on Linux, different PCs, with the correct settings (8N1 115,200 bps), etc.
  • Leaving the device on for an extended period, or off for an extended period does nothing.
Any ideas?

Offline ivor

  • Administrator
  • Sr. Member
  • *****
  • Posts: 583
  • Karma: +134/-125
    • View Profile
    • Netgate
Re: SG-2220 Boot Issue (No Console)
« Reply #1 on: July 30, 2017, 04:10:32 am »
Hi,

Please contact our support at https://customercare.netgate.com

Please reference this thread.

Thanks!
Need help fast? Commercial support: https://www.netgate.com/support/

Offline Carboxyl

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: SG-2220 Boot Issue (No Console)
« Reply #2 on: August 01, 2017, 06:27:58 am »
Thanks, Ivor. I'll keep the forum posted on any resolution.

Offline Carboxyl

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: SG-2220 Boot Issue (No Console)
« Reply #3 on: August 01, 2017, 06:20:34 pm »
I contacted support and got told to use the console to recover. :P

After explaining the situation a second time, I got a "well your warranty's up, so sucks to be you" sort of response. (I'm paraphrasing, of course.)

I'm concerned about committing more money to the 300 or so already spent, just to be told it's dead.

Is there no way for an end user to try to re-write the flash storage, without the console? (Is there, perhaps, a JTAG header inside the device, or something like that?)

Offline ivor

  • Administrator
  • Sr. Member
  • *****
  • Posts: 583
  • Karma: +134/-125
    • View Profile
    • Netgate
Re: SG-2220 Boot Issue (No Console)
« Reply #4 on: August 02, 2017, 04:04:48 am »
I contacted support and got told to use the console to recover. :P

After explaining the situation a second time, I got a "well your warranty's up, so sucks to be you" sort of response. (I'm paraphrasing, of course.)

I'm concerned about committing more money to the 300 or so already spent, just to be told it's dead.

Is there no way for an end user to try to re-write the flash storage, without the console? (Is there, perhaps, a JTAG header inside the device, or something like that?)

Well I can understand the warranty part, it's just how it works. However, accessing console is a valid suggestion. You don't need a JTAG header, there's USB console port in the front of the device. It's the easiest and quickest way to figure out what's happening.

Can you please confirm you attempted to connect to console like described here: https://www.netgate.com/docs/sg-2220/connect-to-console.html

Very important, make sure you have console drivers installed https://www.netgate.com/docs/sg-2220/connect-to-console.html#advanced-configuration

If you followed those instructions correctly, you should have access to the console.

Try that and please report back. Most likely you will have to reinstall pfSense, which is pretty simple.
Need help fast? Commercial support: https://www.netgate.com/support/

Offline Carboxyl

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: SG-2220 Boot Issue (No Console)
« Reply #5 on: August 19, 2017, 11:40:02 am »
OK. I've spent a few hours now going through every procedure you've suggested in detail.

I've confirmed that:
  • I have the correct CP210x USB to UART Bridge VCP driver installed correctly.
  • I have used the correct settings in PuTTY, including trying 115,200, 34,800 and 9,600 baud.
  • I have the cables plugged in correctly.
There is absolutely no output in PuTTY when I power up the SG-2220. Nothing.

As for the LEDs on the front of the device, the STATUS LED goes on solid orange and all four of the Ethernet port LEDs go solid green. Nothing blinks. There is no visible activity... Which is a clue. I don't think the clock is running. More on that in a moment.

I have also created a pfSense image on a USB drive (using the install image for the Netgate ADI platform) and tried powering up the device with the USB drive plugged in. The result is exactly the same. Absolutely no output in PuTTY and all LEDs stuck on solidly. (I tried this same procedure with an old PC using the AMD64 platform and it worked correctly on the old PC, so I know I'm getting this procedure right.)

I did a really deep dive on this problem and I can now see that this is almost certainly a hardware fault, and its symptoms exactly match the Intel Atom C2xxx LPC problem described in a few places:

https://www.netgate.com/blog/clock-signal-component-issue.html
https://www.theregister.co.uk/2017/02/06/cisco_intel_decline_to_link_product_warning_to_faulty_chip/
https://forum.pfsense.org/index.php?topic=125105.msg691024#msg691024
https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/atom-c2000-family-spec-update.pdf (AVR54)

Per the Netgate commitment in the first link, I assume I can get this device repaired or replaced. I'll report back on Netgate's response.
« Last Edit: September 06, 2017, 08:06:00 pm by Carboxyl »

Offline ivor

  • Administrator
  • Sr. Member
  • *****
  • Posts: 583
  • Karma: +134/-125
    • View Profile
    • Netgate
Re: SG-2220 Boot Issue (No Console)
« Reply #6 on: August 19, 2017, 04:11:40 pm »
I was just informed that the RMA was approved. I'm glad we confirmed the issue. Please coordinate with our support the rest of RMA process.

Thank you!
Need help fast? Commercial support: https://www.netgate.com/support/

Offline Carboxyl

  • Newbie
  • *
  • Posts: 5
  • Karma: +0/-0
    • View Profile
Re: SG-2220 Boot Issue (No Console)
« Reply #7 on: September 06, 2017, 08:04:04 pm »
I'm very happy to say that Netgate replaced the motherboard in my device and it's working like new!  :D

Offline ivor

  • Administrator
  • Sr. Member
  • *****
  • Posts: 583
  • Karma: +134/-125
    • View Profile
    • Netgate
Re: SG-2220 Boot Issue (No Console)
« Reply #8 on: September 07, 2017, 05:13:02 am »
Thanks for the follow up! :)
Need help fast? Commercial support: https://www.netgate.com/support/

Offline dennypage

  • Hero Member
  • *****
  • Posts: 690
  • Karma: +132/-0
    • View Profile
Re: SG-2220 Boot Issue (No Console)
« Reply #9 on: September 16, 2017, 11:23:27 am »
My SG-2220 just failed with the same symptoms. Reaching out to support.

Offline shull

  • Newbie
  • *
  • Posts: 19
  • Karma: +0/-0
    • View Profile
Re: SG-2220 Boot Issue (No Console)
« Reply #10 on: November 02, 2017, 11:29:23 pm »
My SG-2220 just died with the same symptoms. One minute everything was fine, then it just stopped and everything was lit up like a Christmas tree.  I cannot communicate with the box using the console cable.  I will be contacting Netgate to see what my options are. 

Offline dennypage

  • Hero Member
  • *****
  • Posts: 690
  • Karma: +132/-0
    • View Profile
Re: SG-2220 Boot Issue (No Console)
« Reply #11 on: November 03, 2017, 01:25:50 am »
Sorry, forgot to do a follow up post earlier...

As with the original poster, Netgate resolved my issue by replacing the board.

Offline bcruze

  • Newbie
  • *
  • Posts: 24
  • Karma: +1/-0
    • View Profile
Re: SG-2220 Boot Issue (No Console)
« Reply #12 on: November 03, 2017, 07:00:49 am »
i have the same box.  did you perform the firmware update to fix the processor bug?

if i remember right because of the bug didn't they extended the warranty of this device as well?

i do NOT look forward to the day i have to replace this box.   it has been 100% reliable until i did the 2.4 beta upgrades and it disabled the network ports... 

   the 2 port minnowboard will probably be my next replacement device from Netgate when that day comes

« Last Edit: November 03, 2017, 07:04:02 am by bcruze »