IO broken on 2.4

[Guest]

[deadmalc]

After a bit more digging I'm convinced that postfix is broken in 2.4
I use it with authentication to the primary mail server, and after an email with an attachment the system hangs.
Really bizarre, I can ssh to the server but if i try and vi a file vi hangs, anything with file I/O seems to be broken.
Think I will migrate that off to another dmz box, and see if the problem still occurs.
Anyone else seen this issue?

[deadmalc]

Nope issue is still reocurring without postfix, what a strange crash.
No logs, nothing just frozen IO on the box

[nico.prenzel]

I've got almost the same error (postifx isn't involved here).
The EFW 2.4 stops doing anything IO related. After a  of time it resumes as nothing had happened.
One good thing about that. It happens rarely. I can't specify it more exactly.
Crazy.

[deadmalc]

It doesn't happen to me all the time, but about once a day. Which means if I'm not there to forcefully shut the box off (a reboot will not work)
then we can't send email (postfix), or use the web (squid) but the firewall still processes traffic.
If I try to use the web interface, that jams as well.

Anyone have any ideas how to debug this?

[nico.prenzel]

Today, the Endian Team released another bunch of upgrades to the EFW 2.4 community version. Perhaps, these changes does have effect at the IO freeze bug.

I do also remember that the freeze IO error happened at the very first usage of the EFW 2.4. It has been reproduceable then some services (for me OpenVPN) had been restarted by the web GUI. But this behaviour is also gone for now.

Does your IO freeze happen each day at the same time (pherhaps some cron action)?

[deadmalc]

Cheers for the note about updates, I've updated and running 2.4 again to see what happens.
The main way it hung was sending an attachment (word document) through postfix. That seemed to kill it, not every time.
Sometimes, it will just randomly freeze - most of the time it's because email has gone and killed it.

Still getting that dodgy segfault though 

kernel: [  221.707812] fetchipac[6972]: segfault at 1 ip b75f7e16 sp bf9902e8 error 4 in libc-2.3.4.so[b75a4000+114000]

Well we shall see if that is any better! Lets hope so.

I've also noticed pdflush using a hell a lot of cpu every so often (possibly nothing as this is running of a file storage rather than dedicated storage)

[nico.prenzel]

Today, the Endian Team also fixed the segfault of ipac. At least there bugfix database shows that. Hopefully there would be another upgrade!?

I've searched for pdflush and found that:

pdflush is for flushing out dirty buffers to disk. If however you have
little RAM and lots of swap it may indeed thrash for a little while while
doing this task.

Pherhaps you should increase your RAM. I'll also look at my EFW and look at the pdflush performance. If i do remember correctly my EFW does have 512 MB RAM assigned.

[nico.prenzel]

On more... pherhaps your machine runs out of free memory because of a potential memory leak (pherhaps, again related to the segfault of ipac)!

One goal of pdflush is to write dirty pages to disk under low memory conditions.

Read there:
freeweb.hu/linuxkernel2/ch15lev1sec4.html

[deadmalc]

Plenty of memory free (2Gb assigned), 0 swap used (I only have 100MB of swap, due to a cockup in my calculation with fdisk)
Think this is just my dodgy on filesystem partition in kvm I use for testing

[nico.prenzel]

pity, it seemed to be such a nice explanation.

[deadmalc]

Day one finished after the efw-upgrade and no problems so far....cross fingers!

[deadmalc]

Seems the postfix problem is still lurking there :-(

Restarting postfix causes the hash on /etc/aliases to hang and amavisd will not shutdown

After shutting postfix down fine, and attempting to restart it....

root      9239  9212  0 08:03 pts/0    00:00:00 /bin/bash /etc/init.d/postfix
start
root      9244  9239  0 08:03 pts/0    00:00:00 /usr/sbin/postalias hash
/etc/aliases


root@saaflanfirewall:/var/log # /etc/init.d/amavisd stop
Shutting down Mail Virus Scanner (amavisd):                [  OK  ]
ps -efroot@saaflanfirewall:/var/log # ps -ef|fgrep amavis
amavis    5416     1  0 07:07 ?        00:00:00 amavisd (ch1-finish)
amavis    5417     1  0 07:07 ?        00:00:00 amavisd (ch0-finish)
amavis    5418     1  0 07:07 ?        00:00:00 amavisd (ch0-finish)
amavis    5419     1  0 07:07 ?        00:00:00 amavisd (ch0-finish)
amavis    5421     1  0 07:07 ?        00:00:00 amavisd (ch0-finish)
amavis    5423     1  0 07:07 ?        00:00:00 amavisd (ch0-finish)
amavis    5424     1  0 07:07 ?        00:00:00 amavisd (ch0-finish)
amavis    5427     1  0 07:07 ?        00:00:00 amavisd (ch0-finish)
amavis    5429     1  0 07:07 ?        00:00:00 amavisd (ch0-finish)
amavis    5430     1  0 07:07 ?        00:00:00 amavisd (ch0-finish)
amavis    5431     1  0 07:07 ?        00:00:00 amavisd (ch0-finish)
root      9456  9212  0 08:03 pts/0    00:00:00 fgrep amavis


root@saaflanfirewall:/var/log # killall amavisd
root@saaflanfirewall:/var/log # killall amavisd
root@saaflanfirewall:/var/log # killall amavisd
root@saaflanfirewall:/var/log # killall -9 amavisd
root@saaflanfirewall:/var/log # killall -9 amavisd
root@saaflanfirewall:/var/log # killall -9 amavisd
root@saaflanfirewall:/var/log # killall -9 amavisd
root@saaflanfirewall:/var/log # ps -ef|fgrep amavis
amavis    5416     1  0 07:06 ?        00:00:00 amavisd (ch1-finish)
amavis    5417     1  0 07:06 ?        00:00:00 amavisd (ch0-finish)
amavis    5418     1  0 07:06 ?        00:00:00 amavisd (ch0-finish)
amavis    5419     1  0 07:06 ?        00:00:00 amavisd (ch0-finish)
amavis    5421     1  0 07:06 ?        00:00:00 amavisd (ch0-finish)
amavis    5423     1  0 07:06 ?        00:00:00 amavisd (ch0-finish)
amavis    5424     1  0 07:06 ?        00:00:00 amavisd (ch0-finish)
amavis    5427     1  0 07:06 ?        00:00:00 amavisd (ch0-finish)
amavis    5429     1  0 07:06 ?        00:00:00 amavisd (ch0-finish)
amavis    5430     1  0 07:06 ?        00:00:00 amavisd (ch0-finish)
amavis    5431     1  0 07:06 ?        00:00:00 amavisd (ch0-finish)

root      9239  9212  0 07:56 pts/0    00:00:00 /bin/bash /etc/init.d/postfix
start
root      9244  9239  0 07:56 pts/0    00:00:00 /usr/sbin/postalias hash
/etc/aliases
root      9468  9212  0 08:03 pts/0    00:00:00 ps -ef
root@saaflanfirewall:/var/log # kill 9244
root@saaflanfirewall:/var/log # kill 9244
root@saaflanfirewall:/var/log # kill 9244
root@saaflanfirewall:/var/log # kill -9 9244
root@saaflanfirewall:/var/log # kill -9 9244
root@saaflanfirewall:/var/log # kill -9 9244
root@saaflanfirewall:/var/log # ps -ef|fgrep postalias
root      9244  9239  0 07:56 pts/0    00:00:00 /usr/sbin/postalias hash
/etc/aliases
root      9470  9212  0 08:03 pts/0    00:00:00 fgrep postalias

Bug filed http://bugs.endian.com/view.php?id=2997

[deadmalc]

I've turned off virus mail scanning, see if that is the problem....
Still battling with it :-(