Issue with RRD Graphs and Traffic Monitoring

[Guest]

[mrkroket]

I recently started to test endian firewall.

But a week ago the System & Traffic Graphs started to fail, And I can't see any graph on the web (Proxy graphs keep working).
After searching and googleing I found out that the problem was in /usr/local/bin/rrdfix.sh. With top it appears to take several minutes and more that 10% CPU time to finish.
That script tries to delete all RRD from some dirs, so it shouldn't take too much time:

Code:

#!/bin/sh

DIRS="/var/log/rrd/ /var/ntop/"

rm /home/httpd/html/graphs/*.png &>/dev/null

find $DIRS -name "*.rrd" | \
    while read F; do
    NOW=$(date "+%s")
    LAST=$(rrdtool last $F)
    if [ "$LAST" -gt "$NOW" ]; then
	echo "rrd file $F contains timestamps in future. Remove the file!"
	rm -f $F
    fi
done

The problem is that if you enable some options on Traffic Monitoring, you'll have thousands of rrd files on /var/ntop/rrd/interfaces/br0/hosts/. And that much files takes a lot of CPU time, a lot of HDD usage and a lot of time. That will "time out" the graph refresh. So the main point is to limit the time used with rrdfix.sh.

My dirty solution was to remove the /var/ntop/ from the previous script. System & traffic graphs are in /var/log/rrd/, so I suppose that Traffic Monitoring will handle rrd's in  /var/ntop/, and therefore we don't need to delete rrd's from traffic monitoring on that script. Am I wrong with that?

Code:

#!/bin/sh

DIRS="/var/log/rrd/"
# Really needed? /var/ntop/"

rm /home/httpd/html/graphs/*.png &>/dev/null

find $DIRS -name "*.rrd" | \
    while read F; do
    NOW=$(date "+%s")
    LAST=$(rrdtool last $F)
    if [ "$LAST" -gt "$NOW" ]; then
	echo "rrd file $F contains timestamps in future. Remove the file!"
	rm -f $F
    fi
done

Also it's important  to limit rrd's on Traffic Monitoring to the hosts you need (https://EFW_IP:3001/editPrefs.html , rrd.hostsFilter option), to reduce the rrd files on /hosts/ folder.

 Now the script takes <1 second to finish, CPU usage drop and graph are again displayed.

Check the difference:

From 33.36% CPU time on TUE it drops to 0.39% on FRI. The white bands are when graphs are unavailable due to the rrdfix.sh timing out.

[jpgillivan]

I haven't experienced any problems like that. But we just implemented EFW a  weeks ago.  What are your system specs?

My specs:
Processor: P4 2.8GHZ
Memory:
                   total       used       free     shared    buffers     cached
Mem:        895592     849716      45876          0     151804     188844
-/+ buffers/cache:     509068     386524
Swap:      1790200      34528    1755672

My memory usage was alot lower but then I changed the cache management to:
Cache management
Harddisk cache size (MB) * 500         
Memory cache size (MB) *    200   
Max object size (KB) *    1024
Min object size (KB) *    0

Maximum swap space used is 3.6%

[mrkroket]

I have nothing to do with system specs, it is a bug that time out the rrd graphs.

My specs are a virtualized EFW:
Virtual Guest: EFW
 Dual Core
 2 GB Ram
 40GB HDD

host:
  AMD Phenom 8450 (3x Core at 2.10Ghz)
  4 GB RAM
  600GB HDD

[inteq]

I post here, so not to start another topic, with related subject.
I do not know what might cause the RRD problem above. Maybe too many clients behind the FW?

My problem is:

I have a constant 4 MB/s upload and most of the time a download of 7-9 MB/s (megabytes) but the graphs show me a max of 550K for one day graph.
Any reason why?
I have to specify that the traffic is made on a forwarded port to a GREEN IP

[mrkroket]

I post here, so not to start another topic, with related subject.
I do not know what might cause the RRD problem above. Maybe too many clients behind the FW?

My problem is:

I have a constant 4 MB/s upload and most of the time a download of 7-9 MB/s (megabytes) but the graphs show me a max of 550K for one day graph.
Any reason why?
I have to specify that the traffic is made on a forwarded port to a GREEN IP

Forwarded inside GREEN? it's an strange setup.  The normal Firewall use is traffic going from Green to RED. What you have is a GREEN to GREEN Traffic, that is not graphed.

[odtabien]

I post here, so not to start another topic, with related subject.
I do not know what might cause the RRD problem above. Maybe too many clients behind the FW?

My problem is:

I have a constant 4 MB/s upload and most of the time a download of 7-9 MB/s (megabytes) but the graphs show me a max of 550K for one day graph.
Any reason why?
I have to specify that the traffic is made on a forwarded port to a GREEN IP

I have the same problem with the network traffic graphs.  we have a 6Mbps connection but the graph peaks at 600Kbps and i know the usage actually goes at 5.98Mbps during peak hours.

i tried to look for the MRTG configuration but didnt find any, maybe they have used another tool for the graph?