Endian 2.3 High CPU usage

[Guest]

[acecombat]

I'm running Endian 2.3 and it constantly has the CPU running at close to 100%.  I've checked top and there is no process using that amount but the system usage is high.

Tasks:  59 total,   4 running,  55 sleeping,   0 stopped,   0 zombie
Cpu(s): 12.0%us, 76.4%sy,  0.0%ni,  0.0%id, 10.3%wa,  0.4%hi,  0.9%si,  0.0%st

Anyone else having the same problem?
Any idea what would be causing the system usage to be so high?

[davvidde]

Check with "top" what process got the near 100% CPU, it is impossible that is no processes gain cpu time.

[acecombat]

Check with "top" what process got the near 100% CPU, it is impossible that is no processes gain cpu time.

None :S  I don't get it...Again, if you look the CPU usage is sitting at over 90% but the processes only add up to 12% CPU utilization...

top - 18:47:14 up 8 days,  3:27,  1 user,  load average: 2.11, 1.85, 1.60
Tasks:  59 total,   3 running,  56 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.7%us, 86.3%sy,  0.0%ni,  0.0%id, 10.7%wa,  0.3%hi,  0.0%si,  0.0%st
Mem:    254908k total,   250876k used,     4032k free,    31364k buffers
Swap:   514072k total,       72k used,   514000k free,   152668k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+    TIME COMMAND
26168 root      25   0  2096  560  432 R 10.3  0.2   1:13.49   1:13 rrdfix.sh
26167 root      18   0  1532  512  412 S  0.7  0.2   0:05.95   0:05 find
    1 root      16   0  1512  568  480 S  0.0  0.2   0:18.65   0:18 init
    2 root      11  -5     0    0    0 S  0.0  0.0   0:00.01   0:00 kthreadd
    3 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00   0:00 migration/0
    4 root      34  19     0    0    0 S  0.0  0.0   0:09.47   0:09 ksoftirqd/0
    5 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00   0:00 watchdog/0
    6 root      10  -5     0    0    0 S  0.0  0.0   0:00.97   0:00 events/0
    7 root      11  -5     0    0    0 S  0.0  0.0   0:00.10   0:00 khelper
   42 root      10  -5     0    0    0 S  0.0  0.0   0:16.74   0:16 kblockd/0
   43 root      20  -5     0    0    0 S  0.0  0.0   0:00.00   0:00 kacpid
   44 root      20  -5     0    0    0 S  0.0  0.0   0:00.00   0:00 kacpi_notify
  210 root      16  -5     0    0    0 S  0.0  0.0   0:00.00   0:00 cqueue/0
  211 root      16  -5     0    0    0 S  0.0  0.0   0:00.00   0:00 ksuspend_usb
  214 root      16  -5     0    0    0 S  0.0  0.0   0:00.00   0:00 khubd
  216 root      11  -5     0    0    0 S  0.0  0.0   0:00.00   0:00 kseriod
  237 root      20   0     0    0    0 S  0.0  0.0   0:00.00   0:00 pdflush

[acecombat]

Anyone have any ideas what could be doing this???

[acecombat]

*Bump
Surely someone out there knows??
It's starting to cause problems for me with the CPU usage running high constantly.  I also want to enable proxy and cache but with the CPU already maxed out I fear it will reduce my performance rather than increase it

[Steve]

It could be something to do with the power management of your CPU

Try this in putty:
echo 1 > /sys/module/processor/parameters/max_cstate


( The default max_cstate is 8 )

[Steve]

Actually, it looks like you don't have enough RAM

256MB is hardly enough to run everything Endian has.
Watch your swap usage, you should never be using swap space.
If you start getting into swap, your system performance is going to go down the drain even if you have super-fast hard disks.

[mrkroket]

It seems to me the old classic rrdfix.sh error. It was on 2.2, and probably will be in 2.3.

of rrdfix.sh error:
-High CPU usage, dure to rrdfix.sh
-rrdfix.sh execution takes more than 10 seconds.
-Broken graphs on Status tab

Causes:
-rrdfix.sh takes too long to execute. It should be only a few millisecs. But if you have Traffic Monitoring (ntop) enabled, some configurations can hang up the server.

Is your case?
On console, run rrdfix.sh. If it takes too long to finish, you have the problem.


Solutions?:
1- Disable Traffic monitoring (ugh!)
2- Configure Traffic Monitoring to only catch data of your local hosts. eg 192.168.0.0/16
3- Recheck the rrdfix script to find the problem:

This is the rrdfix.sh script:
#!/bin/sh

DIRS="/var/log/rrd/ /var/ntop/ /var/lib/collectd/rrd/"

rm /home/httpd/html/graphs/*.png &>/dev/null

find $DIRS -name "*.rrd" | \
    while read F; do
    NOW=$(date "+%s")
    LAST=$(rrdtool last $F)
    if [ "$LAST" -gt "$NOW" ]; then
        echo "rrd file $F contains timestamps in future. Remove the file!"
        rm -f $F
    fi
done


In my case I have the problem on one of the dirs (maybe /var/ntop?). It has way a lot of dirs and files, takes even minutes to delete all files.

[acecombat]

I tried "echo 1 > /sys/module/processor/parameters/max_cstate" was I meant to see something happen?
I thought maybe it changed a system parameter and wouldn't show anything so I rebooted the console and still have high system usage.

I know 256mb isn't a huge amount of RAM, but the swap usage is showing 72k only.  That's not really using it   If I did add extra memory however how would that reduce CPU usage which is my main concern? 

I already had traffic monitoring disabled as I read that it can cause issues with high CPU, but I ran rrdfix anyhow and it hangs after showing:
# rrdfix.sh
ERROR: opening '/var/ntop/rrd/flows/Host': No such file or directory
ERROR: opening '/var/ntop/rrd/flows/Host': No such file or directory

And after all these Top is still showing 60-80% CPU usage and I'm still pulling at my hair   

[mrkroket]

How much time rrdfix takes to execute?

Mine takes about 2-3 seconds.

[acecombat]

How much time rrdfix takes to execute?

Mine takes about 2-3 seconds.

I ran it and it comes up with:

Code:

# rrdfix.sh
ERROR: opening '/var/ntop/rrd/flows/Host': No such file or directory
ERROR: opening '/var/ntop/rrd/flows/Host': No such file or directory

At this point it doesn't return to the command line and just hangs for at least 15mins.  I had to CTRL+C in Putty to return back to command line.

[mrkroket]

At this point it doesn't return to the command line and just hangs for at least 15mins.  I had to CTRL+C in Putty to return back to command line.

You have it. This is what is bloating your system. Edit the /usr/local/bin/rrdfix.sh script and remove the /var/ntop from the DIRS variable.
Run again the script and see if that reduces your execution time.

[acecombat]

You have it. This is what is bloating your system. Edit the /usr/local/bin/rrdfix.sh script and remove the /var/ntop from the DIRS variable.
Run again the script and see if that reduces your execution time.

Genius!!!  Now execution time is down to 2-3 secs.  CPU usage appears ok at present, I'll monitor it and hopefully it'll work fine

[acecombat]

Genius!!!  Now execution time is down to 2-3 secs.  CPU usage appears ok at present, I'll monitor it and hopefully it'll work fine

24 hours and server is still happy   IOU 1 Beer

[acecombat]

Just wanted to update you kind sir.  It has now been 5 days and CPU usage has been normal.  Thanks for your great help, I can continue to use Endian and be happy