The pfSense Store

Author Topic: Emerging Threats Pro rules file download failed. Bad MD5 checksum.  (Read 334 times)

0 Members and 1 Guest are viewing this topic.

Offline ntct

  • Jr. Member
  • **
  • Posts: 64
  • Karma: +8/-0
    • View Profile
Starting rules update...  Time: 2017-11-28 13:16:15
   Downloading Emerging Threats Pro rules md5 file etpro.rules.tar.gz.md5...
   Checking Emerging Threats Pro rules md5 file...
   There is a new set of Emerging Threats Pro rules posted.
   Downloading file 'etpro.rules.tar.gz'...
   Done downloading rules file.
   Emerging Threats Pro rules file download failed.  Bad MD5 checksum.
   Downloaded Emerging Threats Pro rules file MD5: a1f22307529dd1a1cfb1ea9dd0c2e158
   Expected Emerging Threats Pro rules file MD5:

   Emerging Threats Pro rules file download failed.  Emerging Threats Pro rules will not be updated.
The Rules update has finished.  Time: 2017-11-28 13:16:17
« Last Edit: November 27, 2017, 11:38:36 pm by ntct »

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3148
  • Karma: +817/-0
    • View Profile
Re: Emerging Threats Pro rules file download failed. Bad MD5 checksum.
« Reply #1 on: November 28, 2017, 08:18:28 am »
This is not uncommon.  It's a problem on the web site hosting the files.  It will self-correct.  It's not a problem with the pfSense package.

Edit:  to add some additional explanation ---

The Long Version ...
For when this issue comes up again (and I'm sure it will), here are some details.  When the Snort or Suricata packages on pfSense download new rules update archives from the vendor websites, the pfSense package calculates the MD5 checksum of the downloaded rules archive.  It also downloads the separate MD5 file posted by the rules vendor on the vendor's web site.  Snort or Suricata on pfSense tests the MD5 it calcuated from the downloaded gzip archive against the value posted on the vendor's web site.  If they do not match, the MD5 checksum error is printed and the updating of those rules is skipped on the assumption the downloaded gzip archive is corrupt.

The file might have really been corrupted during the download or, more likely, the matching MD5 file on the vendor's web site did not get updated to reflect the correct value for the updated rules gzip archive.  This latter case is more frequent (the MD5 file didn't get updated by the rules vendor).  The vendor usually catches the error fairly quickly and fixes the posted MD5 file.  It can also be a replication issue if the vendor's rules files are hosted on a CDN.  It could be that the replication of the new rules gzip archive succeeded, but the matching MD5 file replication failed or else is behind schedule.  In either case you are left with a new gzip rules archive but an older MD5 file.  When Snort or Suricata downloads the gzip archive and tests its calculated MD5 against the value posted in the vendor's MD5 file you get a mismatch and the error is printed.

Generally by the time of the next update cycle this is fixed and the rules update.  Obviously you need enough time for the vendor or the CDN replication process to get the MD5 file fixed up.  So if you try to run updates back-to-back, then yeah you might get the same MD5 error repeatedly.  If you instead wait for say 12 hours or even until the next day, things will usually be good.

The final possibility, but it is really rare, is that something is wrong on your side and your downloaded copy of the gzip archive truly is corrupt.  One thing that can cause this is running Snort or Suricata on NanoBSD or by having the /tmp partition on a RAM disk.  If there is not enough free space in /tmp to download all the rules, then the MD5 checksum can fail because only a partial copy of the rules file got stored on your RAM disk.  Running Snort or Suricata on a RAM disk is highly discouraged!  If you do that, then you are on your own as that is not a supported configuration.

Bill
« Last Edit: November 28, 2017, 01:24:53 pm by bmeeks »

Offline ntct

  • Jr. Member
  • **
  • Posts: 64
  • Karma: +8/-0
    • View Profile
Re: Emerging Threats Pro rules file download failed. Bad MD5 checksum.
« Reply #2 on: November 29, 2017, 07:36:12 pm »
Hi.

I test Snort can download Emerging Threats Pro Rules correctly, but Suricata can't. :-\


« Last Edit: November 29, 2017, 09:33:52 pm by ntct »

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3148
  • Karma: +817/-0
    • View Profile
Re: Emerging Threats Pro rules file download failed. Bad MD5 checksum.
« Reply #3 on: November 30, 2017, 07:01:14 am »
Hi.

I test Snort can download Emerging Threats Pro Rules correctly, but Suricata can't. :-\

Is this a suddenly new problem, is this the first use of Suricata on the box having the problem?  I'm asking because one possibility is the ET Pro code is incorrect on the Suricata box.  Both packages use the same baseline URL to download the vendor rules.  In the case of ET Pro and Snort VRT, you ET Pro code or Snort Oinkcode is added into the baseline URL to generate the complete URL.

Are the two boxes located in separate geographical locations?  Could be a temporary DNS or CDN issue at the Suricata site if the boxes are widely separated.

Take a look in the Update Log file on the UPDATES tab in Suricata to see what error message is being printed about the download.  Since you're posting in this MD5 thread, does that mean you are seeing an MD5 error on the Suricata box?

Bill

Offline sboyle

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: Emerging Threats Pro rules file download failed. Bad MD5 checksum.
« Reply #4 on: November 30, 2017, 10:08:19 am »
I'm experiencing the same issue.  I have an ET pro ruleset subscription.  I have a single instance of pfSense that runs both Snort and Suricata.  Snort is able to successfully download and use the ET Pro ruleset.  Suricata gives an MD5 checksum error when downloading the ET Pro ruleset:

   Emerging Threats Pro rules file download failed.  Bad MD5 checksum.
   Downloaded Emerging Threats Pro rules file MD5: a1f22307529dd1a1cfb1ea9dd0c2e158
   Expected Emerging Threats Pro rules file MD5:
   Emerging Threats Pro rules file download failed.  Emerging Threats Pro rules will not be updated.

Since Snort is working, I suspect the issue is not with the way the files are being served.  I verified that I have the same ET PRO ruleset code configured for Snort and Suricata (copy/paste after all). Any suggestions?

I'm running the latest version of pfSense, 2.4.2.

Thanks,
Steve
« Last Edit: November 30, 2017, 10:11:59 am by sboyle »

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3148
  • Karma: +817/-0
    • View Profile
Re: Emerging Threats Pro rules file download failed. Bad MD5 checksum.
« Reply #5 on: November 30, 2017, 05:54:50 pm »
I'm experiencing the same issue.  I have an ET pro ruleset subscription.  I have a single instance of pfSense that runs both Snort and Suricata.  Snort is able to successfully download and use the ET Pro ruleset.  Suricata gives an MD5 checksum error when downloading the ET Pro ruleset:

   Emerging Threats Pro rules file download failed.  Bad MD5 checksum.
   Downloaded Emerging Threats Pro rules file MD5: a1f22307529dd1a1cfb1ea9dd0c2e158
   Expected Emerging Threats Pro rules file MD5:
   Emerging Threats Pro rules file download failed.  Emerging Threats Pro rules will not be updated.

Since Snort is working, I suspect the issue is not with the way the files are being served.  I verified that I have the same ET PRO ruleset code configured for Snort and Suricata (copy/paste after all). Any suggestions?

I'm running the latest version of pfSense, 2.4.2.

Thanks,
Steve

I will need to test in a Suricata virtual machine.  I use Snort for my personal home network (just because that's what I started with before I created the Suricata package).  Is this something that just started happening?  If so, can you give me a time table for when it began?

Bill

Offline sboyle

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: Emerging Threats Pro rules file download failed. Bad MD5 checksum.
« Reply #6 on: December 01, 2017, 10:21:44 am »
For me this is a new install.  New hardware and a fresh install of pfSense 2.4.2.  This is the first time I've ever had an ET Pro subscription.  Unfortunately, I can't tell you if this is a new problem or was preexisting, its a new issue to me.  That said, its working today.  Sometime between midnight and 8am (PST) it started working:

Starting rules update...  Time: 2017-12-01 00:30:00
   Downloading Emerging Threats Pro rules md5 file etpro.rules.tar.gz.md5...
   Checking Emerging Threats Pro rules md5 file...
   There is a new set of Emerging Threats Pro rules posted.
   Downloading file 'etpro.rules.tar.gz'...
   Done downloading rules file.
   Emerging Threats Pro rules file download failed.  Bad MD5 checksum.
   Downloaded Emerging Threats Pro rules file MD5: a1f22307529dd1a1cfb1ea9dd0c2e158
   Expected Emerging Threats Pro rules file MD5:
   Emerging Threats Pro rules file download failed.  Emerging Threats Pro rules will not be updated.
   Downloading Snort VRT rules md5 file snortrules-snapshot-2990.tar.gz.md5...
   Checking Snort VRT rules md5 file...
   Snort VRT rules are up to date.
The Rules update has finished.  Time: 2017-12-01 00:30:07

Starting rules update...  Time: 2017-12-01 08:18:15
   Downloading Emerging Threats Pro rules md5 file etpro.rules.tar.gz.md5...
   Checking Emerging Threats Pro rules md5 file...
   There is a new set of Emerging Threats Pro rules posted.
   Downloading file 'etpro.rules.tar.gz'...
   Done downloading rules file.
   Downloading Snort VRT rules md5 file snortrules-snapshot-2990.tar.gz.md5...
   Checking Snort VRT rules md5 file...
   Snort VRT rules are up to date.
   Extracting and installing Emerging Threats Pro rules...
   Installation of Emerging Threats Pro rules completed.
   Copying new config and map files...
   Updating rules configuration for: WAN ...
   Updating rules configuration for: LAN ...
   Live-Reload of updated rules is enabled...
   Live swap of updated rules requested for WAN.
   Live swap of updated rules requested for LAN.
   Live-Reload of the updated rules is complete.
The Rules update has finished.  Time: 2017-12-01 08:18:29


I guess we'll have to blame Proofpoint in this case.  Thanks for the help.

Offline ntct

  • Jr. Member
  • **
  • Posts: 64
  • Karma: +8/-0
    • View Profile
Re: Emerging Threats Pro rules file download failed. Bad MD5 checksum.
« Reply #7 on: December 01, 2017, 05:44:24 pm »
It sucess download yesterday.  :D

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3148
  • Karma: +817/-0
    • View Profile
Re: Emerging Threats Pro rules file download failed. Bad MD5 checksum.
« Reply #8 on: December 02, 2017, 02:35:42 pm »
Thanks for the feedback.  Sometimes the rules vendors have hiccups in the distribution networks.

Bill

Offline gsiemon

  • Jr. Member
  • **
  • Posts: 79
  • Karma: +13/-0
    • View Profile
Re: Emerging Threats Pro rules file download failed. Bad MD5 checksum.
« Reply #9 on: December 03, 2017, 05:21:02 pm »
Bill,

I have been seeing the same issue since I installed pfSense 2.4.2 on 23 November.  Until now, I haven't had any significant periods of failed updates in the three years I have been running pfSense and Suricata.  I haven't made any changes to configuration in months.

Code: [Select]
Starting rules update...  Time: 2017-11-23 12:30:00
Downloading Emerging Threats Pro rules md5 file etpro.rules.tar.gz.md5...
Checking Emerging Threats Pro rules md5 file...
There is a new set of Emerging Threats Pro rules posted.
Downloading file 'etpro.rules.tar.gz'...
Done downloading rules file.
Emerging Threats Pro rules file download failed.  Bad MD5 checksum.
Downloaded Emerging Threats Pro rules file MD5: a1f22307529dd1a1cfb1ea9dd0c2e158
Expected Emerging Threats Pro rules file MD5:
Emerging Threats Pro rules file download failed.  Emerging Threats Pro rules will not be updated.
The Rules update has finished.  Time: 2017-11-23 12:30:08

A few things to note:

The MD5 checksum for the rules file quoted in that error message does not change ie it is the same today as it was on the November 23 (and also seems to match the other log entries posted here).  The expected MD5 checksum is never populated.  Is it even being read correctly?  If the MD5 of the downloaded rules isn't changing it is either not downloading the rules correctly or it is downloading the same unchanging file each time.

Around the same time, Emerging Threats enabled a new ruleset for Suricata 4.0.  I don't know if they modified their folder structure for older engines or not.  I can't see exactly where the updater is looking for the files so its hard for me to troubleshoot this by downloading the original ET rules files and manually computing the MD5 ie is it trying to pull 2.0 rules or 3.2 rules or 4.0 rules?

https://lists.emergingthreats.net/pipermail/emerging-sigs/2017-October/028424.html

I uninstalled and then reinstalled Suricata (keeping settings) over the weekend.  Immediately after install I saw the following successful log entry:

Code: [Select]
Starting rules update...  Time: 2017-12-02 06:30:00
Downloading Emerging Threats Pro rules md5 file etpro.rules.tar.gz.md5...
Checking Emerging Threats Pro rules md5 file...
Emerging Threats Pro rules are up to date.
The Rules update has finished.  Time: 2017-12-02 06:30:06

However, subsequent updates continue to fail with exactly the same MD5 checksum error as above:

Code: [Select]
Starting rules update...  Time: 2017-12-04 06:30:00
Downloading Emerging Threats Pro rules md5 file etpro.rules.tar.gz.md5...
Checking Emerging Threats Pro rules md5 file...
There is a new set of Emerging Threats Pro rules posted.
Downloading file 'etpro.rules.tar.gz'...
Done downloading rules file.
Emerging Threats Pro rules file download failed.  Bad MD5 checksum.
Downloaded Emerging Threats Pro rules file MD5: a1f22307529dd1a1cfb1ea9dd0c2e158
Expected Emerging Threats Pro rules file MD5:
Emerging Threats Pro rules file download failed.  Emerging Threats Pro rules will not be updated.
The Rules update has finished.  Time: 2017-12-04 06:30:06

I'd like to do some more troubleshooting.  I'd like to check the timestamps and MD5 of the etpro.rules.tar.gz and the contents of etpro.rules.tar.gz.md5 and see what is going on.  Can you tell me where the rules files are downloaded to on pfSense?  Can you also confirm the download URL that I should be looking at for Suricata 4.0.1?


Greg



Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3148
  • Karma: +817/-0
    • View Profile
Re: Emerging Threats Pro rules file download failed. Bad MD5 checksum.
« Reply #10 on: December 03, 2017, 09:04:54 pm »
Bill,

I have been seeing the same issue since I installed pfSense 2.4.2 on 23 November.  Until now, I haven't had any significant periods of failed updates in the three years I have been running pfSense and Suricata.  I haven't made any changes to configuration in months.


I'd like to do some more troubleshooting.  I'd like to check the timestamps and MD5 of the etpro.rules.tar.gz and the contents of etpro.rules.tar.gz.md5 and see what is going on.  Can you tell me where the rules files are downloaded to on pfSense?  Can you also confirm the download URL that I should be looking at for Suricata 4.0.1?


Greg

I have not touched the ET-Pro rules download URL since the Suricata package was created.  I once was given an ET-Pro code for testing.  I will need to see if it still works and use the access to check the directory structure.  Could be some changes have happened with the new owners.

Bill

Offline gsiemon

  • Jr. Member
  • **
  • Posts: 79
  • Karma: +13/-0
    • View Profile
Re: Emerging Threats Pro rules file download failed. Bad MD5 checksum.
« Reply #11 on: December 04, 2017, 04:17:25 pm »
Bill,

Further troubleshooting at my end...

I found the relevant code that defines the URL for downloading files in the pfSense Suricata package.

The base download URL is defined in the variable ETPRO_BASE_DNLD_URL in the default config file as "https://rules.emergingthreatspro.com/".

In suricata_check_for_rule_updates.php this base URL is extended on line 71 by appending "{$etproid}/suricata/"

Code: [Select]
/* Set up Emerging Threats rules filenames and URL */
if ($etpro == "on") {
$emergingthreats_filename = ETPRO_DNLD_FILENAME;
$emergingthreats_filename_md5 = ETPRO_DNLD_FILENAME . ".md5";
$emergingthreats_url = ETPRO_BASE_DNLD_URL;
---> $emergingthreats_url .= "{$etproid}/suricata/";
$et_name = "Emerging Threats Pro";
$et_md5_remove = ET_DNLD_FILENAME . ".md5";
unlink_if_exists("{$suricatadir}{$et_md5_remove}");
}
else {

For me, that complete URL https://rules.emergingthreatspro.com/{$etproid}/suricata/ where I substitute my Subscription code for {$etproid} is now an error page:

Code: [Select]
Emerging Threats Pro suricata-1.3-enhanced
Sorry, I wasn't able to find your subscription to this service. Please contact support@emergingthreats.net for help.
Name
Last Modified
Size

If I change line 71 of suricata_check_for_rule_updates.php to append "{$etproid}/suricata-4.0/" then everything starts updating correctly:

Code: [Select]
Starting rules update...  Time: 2017-12-05 08:01:37
Downloading Emerging Threats Pro rules md5 file etpro.rules.tar.gz.md5...
Checking Emerging Threats Pro rules md5 file...
There is a new set of Emerging Threats Pro rules posted.
Downloading file 'etpro.rules.tar.gz'...
Done downloading rules file.
Extracting and installing Emerging Threats Pro rules...
Installation of Emerging Threats Pro rules completed.
Copying new config and map files...
Updating rules configuration for: LAN ...
Live-Reload of updated rules is enabled...
Live swap of updated rules requested for LAN.
Live-Reload of the updated rules is complete.
The Rules update has finished.  Time: 2017-12-05 08:02:17

It looks like the root cause of the update problems is Emerging Threats reorganising their folder structure in the third week of November.  The link I previously posted now states that updates should be retrieved using the following URL format:
Code: [Select]
https://rules.emergingthreatspro.com/$oinkcode/$engine-$version/
Hope that helps.

Greg

Offline ntct

  • Jr. Member
  • **
  • Posts: 64
  • Karma: +8/-0
    • View Profile
Re: Emerging Threats Pro rules file download failed. Bad MD5 checksum.
« Reply #12 on: December 04, 2017, 06:27:23 pm »
Quote
https://rules.emergingthreatspro.com/$oinkcode/$engine-$version/

Yes, I can download again. :D

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3148
  • Karma: +817/-0
    • View Profile
Re: Emerging Threats Pro rules file download failed. Bad MD5 checksum.
« Reply #13 on: December 04, 2017, 09:08:03 pm »
Bill,

Further troubleshooting at my end...

I found the relevant code that defines the URL for downloading files in the pfSense Suricata package.

The base download URL is defined in the variable ETPRO_BASE_DNLD_URL in the default config file as "https://rules.emergingthreatspro.com/".

In suricata_check_for_rule_updates.php this base URL is extended on line 71 by appending "{$etproid}/suricata/"

Code: [Select]
/* Set up Emerging Threats rules filenames and URL */
if ($etpro == "on") {
$emergingthreats_filename = ETPRO_DNLD_FILENAME;
$emergingthreats_filename_md5 = ETPRO_DNLD_FILENAME . ".md5";
$emergingthreats_url = ETPRO_BASE_DNLD_URL;
---> $emergingthreats_url .= "{$etproid}/suricata/";
$et_name = "Emerging Threats Pro";
$et_md5_remove = ET_DNLD_FILENAME . ".md5";
unlink_if_exists("{$suricatadir}{$et_md5_remove}");
}
else {

For me, that complete URL https://rules.emergingthreatspro.com/{$etproid}/suricata/ where I substitute my Subscription code for {$etproid} is now an error page:

Code: [Select]
Emerging Threats Pro suricata-1.3-enhanced
Sorry, I wasn't able to find your subscription to this service. Please contact support@emergingthreats.net for help.
Name
Last Modified
Size

If I change line 71 of suricata_check_for_rule_updates.php to append "{$etproid}/suricata-4.0/" then everything starts updating correctly:

Code: [Select]
Starting rules update...  Time: 2017-12-05 08:01:37
Downloading Emerging Threats Pro rules md5 file etpro.rules.tar.gz.md5...
Checking Emerging Threats Pro rules md5 file...
There is a new set of Emerging Threats Pro rules posted.
Downloading file 'etpro.rules.tar.gz'...
Done downloading rules file.
Extracting and installing Emerging Threats Pro rules...
Installation of Emerging Threats Pro rules completed.
Copying new config and map files...
Updating rules configuration for: LAN ...
Live-Reload of updated rules is enabled...
Live swap of updated rules requested for LAN.
Live-Reload of the updated rules is complete.
The Rules update has finished.  Time: 2017-12-05 08:02:17

It looks like the root cause of the update problems is Emerging Threats reorganising their folder structure in the third week of November.  The link I previously posted now states that updates should be retrieved using the following URL format:
Code: [Select]
https://rules.emergingthreatspro.com/$oinkcode/$engine-$version/
Hope that helps.

Greg

Thanks for the research into this.  I will need to update the URL used in the package.

Edit:  I have submitted the patch for this to the pfSense for review and merging into production.  Here is a link to the Github Pull Request: https://github.com/pfsense/FreeBSD-ports/pull/486.  Look for a new Suricata package version to appear in the next few days.  The only change in the new package is this ET-Pro and ET-Open rules URL update.

Bill
« Last Edit: December 04, 2017, 09:43:12 pm by bmeeks »

Offline gsiemon

  • Jr. Member
  • **
  • Posts: 79
  • Karma: +13/-0
    • View Profile
Re: Emerging Threats Pro rules file download failed. Bad MD5 checksum.
« Reply #14 on: December 05, 2017, 01:16:34 am »
Bill,  Thankyou for the quick response. 

While appending suricata-4.0 seems to work, on closer inspection of the ET Mailing list entry I think it would be better to base the Rules URL on the full Suricata version number.  They give the following examples:

Code: [Select]
Suricata 4.0: https://rules.emergingthreatspro.com/$oinkcode/suricata-4.0.0/
Suricata 3.2.3: https://rules.emergingthreatspro.com/$oinkcode/suricata-3.2.3/
Suricata 2.0.11: https://rules.emergingthreatspro.com/$oinkcode/suricata-2.0.11/

Reference: https://lists.emergingthreats.net/pipermail/emerging-sigs/2017-October/028424.html

Perhaps a longer term fix is to append the current package version number to the URL?

Greg
« Last Edit: December 05, 2017, 01:20:46 am by gsiemon »

Offline bmeeks

  • Hero Member
  • *****
  • Posts: 3148
  • Karma: +817/-0
    • View Profile
Re: Emerging Threats Pro rules file download failed. Bad MD5 checksum.
« Reply #15 on: December 05, 2017, 08:14:03 am »
Bill,  Thankyou for the quick response. 

While appending suricata-4.0 seems to work, on closer inspection of the ET Mailing list entry I think it would be better to base the Rules URL on the full Suricata version number.  They give the following examples:

Code: [Select]
Suricata 4.0: https://rules.emergingthreatspro.com/$oinkcode/suricata-4.0.0/
Suricata 3.2.3: https://rules.emergingthreatspro.com/$oinkcode/suricata-3.2.3/
Suricata 2.0.11: https://rules.emergingthreatspro.com/$oinkcode/suricata-2.0.11/

Reference: https://lists.emergingthreats.net/pipermail/emerging-sigs/2017-October/028424.html

Perhaps a longer term fix is to append the current package version number to the URL?

Greg

Thank you for the update and the link to the mailing list.  I will look into this.  For now, the issue should be fixed with the new package update released today.

Bill