pfSense Gold Subscription

Author Topic: NAT not always working  (Read 2133 times)

0 Members and 1 Guest are viewing this topic.

Offline samstre

  • Newbie
  • *
  • Posts: 7
    • View Profile
NAT not always working
« on: December 15, 2011, 07:28:18 pm »
i've got a strange problem with my amd64 pfsense 2.0.
i've got several external ip's (using ip aliases) and i'm forwarding the required ports to my servers. it all works quite well.

however i've got a problem with "fileuploads". if i try to upload a file larger than ~55kb, the upload fails.
in my example i try to upload a file onto a webserver. i found out about the problem because i had troubles with attachments in emails. same situation there. if the attachment gets to big, i cannot send / receive it.

i've already tried to modify the firewall optimization (changed to conservative) and i also tried to disable scrubbing.

the thing that freaks me out, is that according to the logs everything is fine (no packets dropped or blocked).

any ideas on my problem / anyone else with the same problem?

greetings

Offline cmb

  • Administrator
  • Hero Member
  • *****
  • Posts: 6326
    • LinkedIn
    • Twitter
    • View Profile
    • Chris Buechler
Re: NAT not always working
« Reply #1 on: December 16, 2011, 11:31:58 am »
First guess, you need to lower your MSS on WAN. Try 1400.

Offline samstre

  • Newbie
  • *
  • Posts: 7
    • View Profile
Re: NAT not always working
« Reply #2 on: December 17, 2011, 08:26:29 am »
I tried it, but it didn't work. I also tried lowering the MTU - but also with no success.

I did 2 tcpdumps. The first with a "small" file (i think it was 52KB big) that was working and a second one with a "big" file (56KB or something) that failed.

I've uploaded the dumps. Maybe someone could have a look. I would really appreciate it.
http://dl.dropbox.com/u/5448701/pfsense_working_small_file
http://dl.dropbox.com/u/5448701/pfsense_notworking_big_file


I did some more dumps.
Tried to upload a file from an external client to a http server behind fw.

WORKING 52KB upload
------------------------------
52KB Fileupload captured on LAN interface (tcpdump -nN -i re0_vlan3 net INTERNAL_SERVER)
http://dl.dropbox.com/u/5448701/52KB_LAN_INTERFACE
52KB Fileupload captured on WAN interface (tcpdump -nN -i em0 net EXTERNAL_CLIENT and not port 1194)
http://dl.dropbox.com/u/5448701/52KB_WAN_INTERFACE


NOT WORKING 66KB upload
------------------------------------
66KB Fileupload captured on LAN interface (tcpdump -nN -i re0_vlan3 net INTERNAL_SERVER)
http://dl.dropbox.com/u/5448701/66KB_LAN_INTERFACE
66KB Fileupload captured on WAN interface (tcpdump -nN -i em0 net EXTERNAL_CLIENT and not port 1194)
http://dl.dropbox.com/u/5448701/66KB_WAN_INTERFACE


btw. i'm using following nics:
WAN: em0: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0xdc00-0xdc1f mem 0xfeae0000-0xfeafffff,0xfea00000-0xfea7ffff,0xfeadc000-0xfeadffff irq 16 at device 0.0 on pci1
LAN (re0): rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
« Last Edit: December 17, 2011, 10:52:11 am by samstre »

Offline cmb

  • Administrator
  • Hero Member
  • *****
  • Posts: 6326
    • LinkedIn
    • Twitter
    • View Profile
    • Chris Buechler
Re: NAT not always working
« Reply #3 on: December 17, 2011, 12:29:22 pm »
The text output isn't really adequate to analyze that particular scenario, if you can save that to a pcap (-w <filename>) and upload it, that will be much more telling.

Offline samstre

  • Newbie
  • *
  • Posts: 7
    • View Profile
Re: NAT not always working
« Reply #4 on: December 19, 2011, 07:11:15 am »
Ok... I did so. Sorry it took quite a while cause I was not at home.

Here are the 2 files.
working.cap (53.93KB JPG) http://dl.dropbox.com/u/5448701/working.cap
not_working.cap (67.9KB JPG) http://dl.dropbox.com/u/5448701/not_working.cap

Offline cmb

  • Administrator
  • Hero Member
  • *****
  • Posts: 6326
    • LinkedIn
    • Twitter
    • View Profile
    • Chris Buechler
Re: NAT not always working
« Reply #5 on: December 19, 2011, 07:36:53 am »
It's exactly what I alluded to in my first reply, large packets aren't getting through somewhere. Guessing that capture was on WAN since it sounds like you're NATing, what does it look like from LAN at the same time?

Offline samstre

  • Newbie
  • *
  • Posts: 7
    • View Profile
Re: NAT not always working
« Reply #6 on: December 19, 2011, 08:04:46 am »
Now it's getting weird... Almost all captured packets on the lan-if have invalid header checksums. Can sombody explain that to me please. Shouldn't the nat rewrite the checksums to match the translated ip??? (RFC1631 or something)

Working file (53.93KB)
WAN http://dl.dropbox.com/u/5448701/working_wan.cap
LAN http://dl.dropbox.com/u/5448701/working_lan.cap

Not working file (67.9KB)
WAN http://dl.dropbox.com/u/5448701/not_working_wan.cap
LAN http://dl.dropbox.com/u/5448701/not_working_lan.cap

Another strange thing I noticed is that I can receive e-Mails (from external) with big attachments (just tried a 5MB mail). But I still cannot send an large email (from outside via my internal mailserver)
« Last Edit: December 19, 2011, 08:14:28 am by samstre »

Offline trunglam

  • Jr. Member
  • **
  • Posts: 53
    • View Profile
Re: NAT not always working
« Reply #7 on: December 19, 2011, 09:57:38 am »
did you check limited attach size on your mail server?
« Last Edit: December 19, 2011, 09:59:32 am by trunglam »

Offline samstre

  • Newbie
  • *
  • Posts: 7
    • View Profile
Re: NAT not always working
« Reply #8 on: December 19, 2011, 10:51:52 am »
The Mailserver is working fine. (It works from inside the lan and with the old fw)
But thanks :)

I did some other tests with my old freebsd fw (ipfw + natd). The old firewall is rewriting the header checksum (after rewriting the Dst), while the new Pfsense seems to forget about it. I really hope someone of you guys could explain that to me / help me. Another strange thing is that the MSS seems to change (from 1460 on WAN to 1360 on LAN) on the Pfsense.

I really appreciate all of you help!
Tell me what infos you need i'd be happy to provide them.

Offline johnpoz

  • Hero Member
  • *****
  • Posts: 2889
    • View Profile
Re: NAT not always working
« Reply #9 on: December 19, 2011, 11:13:52 am »
You truncated the capture to 96Bytes, Im like 99% sure you can not validate checksums when you truncate.

Also if your offloading to your nic, checks sums can be wacky -- I normally turn off checksum validation in wireshark because of this.

I would redo you captures without truncating.
- An intelligent man is sometimes forced to be drunk to spend time with his fools.
- Have I helped you, want to say thanks?  Donate to pfsense the cost of a beer http://pfsense.org/donate.html

Offline cmb

  • Administrator
  • Hero Member
  • *****
  • Posts: 6326
    • LinkedIn
    • Twitter
    • View Profile
    • Chris Buechler
Re: NAT not always working
« Reply #10 on: December 19, 2011, 04:56:39 pm »
You won't have checksums at all when capturing on any host with hardware checksum offloading enabled. That's added by the NIC right before it hits the wire. It's normal (Wireshark even gives you a note as such). You do have the correct checksum there, 0x0000 is what it really is at that stage. With a 96 byte snap length you will get the full IP and TCP headers (plus some), which will have the full checksums.

So the second capture shows the firewall is sending all those packets out of LAN that disappear somewhere. Short of the checksum that gets added by the hardware, what you're capturing on LAN is what is on the wire. It's passing it to the internal host.

Those captures show it's more than just large packets not getting through, SYNs are even getting retransmitted multiple times. Unless you had some odd filter on that capture, you have some broken routing somewhere. Note there is no traffic going from 192.168.200.16 back to any outside IP on that LAN capture. Which probably means that host's default gateway is set to something else, which isn't going to work correctly.

Offline samstre

  • Newbie
  • *
  • Posts: 7
    • View Profile
Re: NAT not always working
« Reply #11 on: December 20, 2011, 08:00:27 am »
OMFG!

THANK YOU CMB!!!!!!

What a $!%%#**! I simply screwed up my routing. Now it is working like charm!
Thank you a million times. I was just to stupid :)

You guys are awesome! Thank you all for your help!

Offline pk-oso

  • Newbie
  • *
  • Posts: 2
    • View Profile
Re: NAT not always working
« Reply #12 on: January 20, 2012, 12:38:09 pm »
Hi all.. What do you mean with screewed my routing"? Thank You

Offline cmb

  • Administrator
  • Hero Member
  • *****
  • Posts: 6326
    • LinkedIn
    • Twitter
    • View Profile
    • Chris Buechler
Re: NAT not always working
« Reply #13 on: January 20, 2012, 12:59:58 pm »
Hi all.. What do you mean with screewed my routing"? Thank You

There are lots of ways to screw routing. If yours is screwed, it's likely screwed in different ways than this one. :) Start a new thread to describe your issue.

In this case, it came down to:
Note there is no traffic going from 192.168.200.16 back to any outside IP on that LAN capture. Which probably means that host's default gateway is set to something else, which isn't going to work correctly.