Netgate m1n1wall

Author Topic: Updated today to latest snap - very slow  (Read 16310 times)

0 Members and 1 Guest are viewing this topic.

Offline eweri

  • Jr. Member
  • **
  • Posts: 45
  • Karma: +0/-0
    • View Profile
Re: Updated today to latest snap - very slow
« Reply #45 on: April 26, 2012, 11:25:43 am »
Hello!

Just installed pfSense-2.1-DEVELOPMENT-4g-i386-nanobsd-20120425-2326.img fresh (no update) on my 8GB Kingston CF with my ALIX board and it is still very slow.
Well it is more like it is sleeping for minutes than does its job until it is sleeping for minutes again.

For example: I am connected to the serial console and the first sleep happens when it tries to configure the WAN interface - boot process hangs for at least 2 minutes. After pfsense is completely up and running I select "Assign interfaces" from the menu and try to assign the interfaces in the right order. Well when I tried to detect interfaces automatically it takes very long, until I get any response.
After every interface is assigned it takes very long "writing configuration" but after that my WAN got no IP-address (should use DHCP).
Than I selected "Reboot system" and once again it takes ages ;-) until the system reboots.

But one thing is better than the snapshots before - now after reboot my WAN-interface got an IP-address from my DHCP server.

Hope this helps to solve this problem, bye,
eweri

Offline moullas

  • Newbie
  • *
  • Posts: 12
  • Karma: +0/-0
    • View Profile
Re: Updated today to latest snap - very slow
« Reply #46 on: April 26, 2012, 03:37:44 pm »
re-flashed today to latest snap... still extremely slow, when re-mounting as ro

Btw, my CF is a Lexar 4GB if it matters.. are there any logs that may shed some light?

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 14990
  • Karma: +4/-0
    • View Profile
Re: Updated today to latest snap - very slow
« Reply #47 on: April 27, 2012, 01:11:14 pm »
To add a couple more data points:
* Enabling TRIM on the slice - no effect, still slow
* Enabling softupdates on the slice - no effect, still slow
* Running sync;sync;sync; before mounting ro - no effect, still slow
* The mount process sticks in biowr state, which is disk i/o
* While mount is stuck, the system is somewhat unresponsive - can't run a process or connect with ssh, but does appear to continue routing.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 14990
  • Karma: +4/-0
    • View Profile
Re: Updated today to latest snap - very slow
« Reply #48 on: April 27, 2012, 02:01:29 pm »
I went ahead and opened a redmine ticket for this:
https://redmine.pfsense.org/issues/2401
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline markky

  • Newbie
  • *
  • Posts: 9
  • Karma: +0/-0
    • View Profile
Re: Updated today to latest snap - very slow
« Reply #49 on: April 28, 2012, 02:29:18 am »
...

Many have also suggested that there's a difference in behaviour between an upgrade an a fresh install of the flash card.  I doubt it actually has any impact, but I'll run that test tomorrow on the cheapo card to eliminate/confirm it.

 - Mark

As expected, the slow CF card still exhibits stalls in mount rw -> ro irrespective of whether the image is an upgrade or a complete reimage of the flash.

As far as I can remember, my slow CF card can be written at just under 4 MB/s from the dev virtual machine.  The sandisk card by comparison writes at over 5MB/s.

 - Mark


Within half an hour of putting the 4801 in production with a sandisk 8G card instead of the previous slower 4G card, the system started exhibiting "stall" conditions.

Unfortunately, my production placement makes serial access a pain (and hence kernel debugger) so I haven't garnered much since.

I'm at the point where I'm thinking I'll just rebuild the kernel and use unionfs to reduce flash wear and tear and do the occasional sync to either underlaying fs or a persistent overlay.

 - Mark

Offline markky

  • Newbie
  • *
  • Posts: 9
  • Karma: +0/-0
    • View Profile
Re: Updated today to latest snap - very slow
« Reply #50 on: April 28, 2012, 03:00:47 am »
I went ahead and opened a redmine ticket for this:
https://redmine.pfsense.org/issues/2401

Here's a bit more detail for what you described.  The fact that flushfiles is flushing out so much on a quiescent filesystem is rather suspect (as a counter point, run lsof and look for writeable files).


The next line is a tty interrogation with ^T
load: 0.22  cmd: mount 2123 [biowr] 5.82r 0.02u 0.08s 7% 1044k

This is the alternate ddb entry, "~^b"

~KDB: enter: Break sequence on console
[thread pid 10 tid 64003 ]
Stopped at      kdb_enter+0x3a: movl    $0,kdb_why

# Just what is the hung mount doing...
db> bt 2123
Tracing pid 2123 tid 64074 td 0xc25612e0
sched_switch(c25612e0,0,104,191,10e78686,...) at sched_switch+0x399
mi_switch(104,0,c0c8be83,1eb,4c,...) at mi_switch+0x1da
sleepq_switch(c25612e0,0,c0c8be83,260,0,...) at sleepq_switch+0x15f
sleepq_wait(cf240fd0,4c,c0c92fe4,0,0,...) at sleepq_wait+0x63
_sleep(cf240fd0,c212c524,4c,c0c92fe4,0,...) at _sleep+0x372
bwait(cf240fd0,4c,c0c92fe4,cf240fd0,d412a69c,...) at bwait+0x6f
bufwait(cf240fd0,cf240fd0,df,cf240fd0,c22fb5e4,...) at bufwait+0x48
bufwrite(cf240fd0,0,c0cb0aac,772,0,...) at bufwrite+0x165
ffs_bufwrite(cf240fd0,c22fd600,80,1000,0,...) at ffs_bufwrite+0x27b
ffs_update(c238610c,1,c0cb10b8,162,d412a798,...) at ffs_update+0x2ac
ffs_syncvnode(c238610c,1,c25612e0,c2320110,c23861b4,...) at ffs_syncvnode+0x48f
ffs_fsync(d412a858,c238610c,0,c238610c,d412a880,...) at ffs_fsync+0x27
VOP_FSYNC_APV(c11dcc20,d412a858,c0c955dc,9cb,0,...) at VOP_FSYNC_APV+0xa5
vflush(c22f2c94,0,6,c25612e0,c22e0200,...) at vflush+0x2b4
ffs_flushfiles(c22f2c94,6,c25612e0,e6,c095ca37,...) at ffs_flushfiles+0x7b
ffs_mount(c22f2c94,0,c0c94deb,3ed,0,...) at ffs_mount+0x452
vfs_donmount(c25612e0,10090003,c286aa00,c286aa00,bfbfdca9,...) at vfs_donmount+0x1012
nmount(c25612e0,d412acec,c25612e0,c1214c80,207,...) at nmount+0x64
syscall(d412ad28) at syscall+0x24e
--More--^M        ^MXint0x80_syscall() at Xint0x80_syscall+0x21
--- syscall (378, FreeBSD ELF32, nmount), eip = 0x280e746b, esp = 0xbfbfdc7c, ebp = 0xbfbfe1d8 ---

db> show mount
0xc22f2c94 /dev/ufs/pfsense1 on / (ufs)
0xc22f3000 devfs on /dev (devfs)
0xc22f2a10 /dev/md0 on /tmp (ufs)
0xc22f278c /dev/md1 on /var (ufs)
0xc22f2508 /dev/ufs/cf on /cf (ufs)
0xc22f2000 devfs on /var/dhcpd/dev (devfs)

Offline moullas

  • Newbie
  • *
  • Posts: 12
  • Karma: +0/-0
    • View Profile
Re: Updated today to latest snap - very slow
« Reply #51 on: May 04, 2012, 04:35:33 am »
So.... has this been abandoned or still actively looked into?

Weekend coming up, have time for testing :-D

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 14990
  • Karma: +4/-0
    • View Profile
Re: Updated today to latest snap - very slow
« Reply #52 on: May 04, 2012, 06:42:13 pm »
It hasn't been forgotten, but I haven't had any new leads to try. Ermal thought it might be something that was recently fixed on FreeBSD 8-STABLE, but I wasn't able to make usable patches out of the commits I found so for me that was a dead end (someone else might give it a try)
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline Beerman

  • Jr. Member
  • **
  • Posts: 65
  • Karma: +0/-0
    • View Profile
Re: Updated today to latest snap - very slow
« Reply #53 on: May 16, 2012, 09:33:51 am »
Any (good) News?  :)

Offline databeestje

  • Administrator
  • Hero Member
  • *****
  • Posts: 1048
  • Karma: +0/-0
  • It just might be your luck day, if you only knew.
    • View Profile
Re: Updated today to latest snap - very slow
« Reply #54 on: May 16, 2012, 11:42:09 am »
None yet

Offline tritron

  • Jr. Member
  • **
  • Posts: 85
  • Karma: +0/-0
    • View Profile
Re: Updated today to latest snap - very slow
« Reply #55 on: June 09, 2012, 11:00:48 pm »
It seems that the bug still exist interface is very slow unless I execute /etc/rc.conf_mount_rw  in ssh. So workaround is to run /etc/rc.conf_mount_rw  before making any changes

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 14990
  • Karma: +4/-0
    • View Profile
Re: Updated today to latest snap - very slow
« Reply #56 on: June 09, 2012, 11:22:38 pm »
We're still aware it's an issue, there is an open bug in redmine for it.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline phil.davis

  • Hero Member
  • *****
  • Posts: 2366
  • Karma: +8/-0
    • View Profile
    • International Nepal Fellowship
Re: Updated today to latest snap - very slow
« Reply #57 on: June 20, 2012, 06:17:09 am »
I noticed that Seth Mos has committed "Hopefully tackle the slow nanobsd writes. Redmine ticket #2401" on Github. I guess this is probably built in the version I just updated to:
2.1-BETA0 (i386)
built on Tue Jun 19 20:53:01 EDT 2012
FreeBSD 8.3-RELEASE-p3

Is there any particular sequence of operations that we can/should test to help to see if the issue is resolved or improved?
(I must say, I haven't really noticed the issue on various test nanoBSD installs on my Alix 2D3 in recent weeks!)
If I helped you, then help someone else - buy someone a gift from the INF catalog http://secure.inf.org/gifts/usd/

Offline jimp

  • Administrator
  • Hero Member
  • *****
  • Posts: 14990
  • Karma: +4/-0
    • View Profile
Re: Updated today to latest snap - very slow
« Reply #58 on: June 20, 2012, 06:20:22 am »
It didn't help, unfortunately.

Timing the mount rw then ro calls is sufficient, see the existing redmine ticket for it.
Need help fast? Commercial Support!

Co-Author of pfSense: The Definitive Guide. - Check the Doc Wiki for FAQs.

Do not PM for help!

Offline eweri

  • Jr. Member
  • **
  • Posts: 45
  • Karma: +0/-0
    • View Profile
Re: Updated today to latest snap - very slow
« Reply #59 on: July 03, 2012, 06:24:54 am »
 Just installed snapshot pfSense-2.1-DEVELOPMENT-2g-i386-nanobsd-20120629-1927
still the same problem