Welcome, Guest. Please login or register.
Did you miss your activation email?
Saturday 14 December 2024, 01:16:42 am

Login with username, password and session length

Get the new Updates directly from Endian  HERE
14261 Posts in 4377 Topics by 6517 Members
Latest Member: Sandro
Search:     Advanced search
+  EFW Support
|-+  Support
| |-+  General Support
| | |-+  Issue with RRD Graphs and Traffic Monitoring
0 Members and 0 Guests are viewing this topic. « previous next »
Pages: [1] Go Down Print
Author Topic: Issue with RRD Graphs and Traffic Monitoring  (Read 20278 times)
mrkroket
Hero Member
*****
Offline Offline

Posts: 495


« on: Saturday 21 March 2009, 02:07:13 am »

I recently started to test endian firewall.

But a week ago the System & Traffic Graphs started to fail, And I can't see any graph on the web (Proxy graphs keep working).
After searching and googleing I found out that the problem was in /usr/local/bin/rrdfix.sh. With top it appears to take several minutes and more that 10% CPU time to finish.
That script tries to delete all RRD from some dirs, so it shouldn't take too much time:
Code:
#!/bin/sh

DIRS="/var/log/rrd/ /var/ntop/"

rm /home/httpd/html/graphs/*.png &>/dev/null

find $DIRS -name "*.rrd" | \
    while read F; do
    NOW=$(date "+%s")
    LAST=$(rrdtool last $F)
    if [ "$LAST" -gt "$NOW" ]; then
echo "rrd file $F contains timestamps in future. Remove the file!"
rm -f $F
    fi
done
The problem is that if you enable some options on Traffic Monitoring, you'll have thousands of rrd files on /var/ntop/rrd/interfaces/br0/hosts/. And that much files takes a lot of CPU time, a lot of HDD usage and a lot of time. That will "time out" the graph refresh. So the main point is to limit the time used with rrdfix.sh.

My dirty solution was to remove the /var/ntop/ from the previous script. System & traffic graphs are in /var/log/rrd/, so I suppose that Traffic Monitoring will handle rrd's in  /var/ntop/, and therefore we don't need to delete rrd's from traffic monitoring on that script. Am I wrong with that?
Code:
#!/bin/sh

DIRS="/var/log/rrd/"
# Really needed? /var/ntop/"

rm /home/httpd/html/graphs/*.png &>/dev/null

find $DIRS -name "*.rrd" | \
    while read F; do
    NOW=$(date "+%s")
    LAST=$(rrdtool last $F)
    if [ "$LAST" -gt "$NOW" ]; then
echo "rrd file $F contains timestamps in future. Remove the file!"
rm -f $F
    fi
done


Also it's important  to limit rrd's on Traffic Monitoring to the hosts you need (https://EFW_IP:3001/editPrefs.html , rrd.hostsFilter option), to reduce the rrd files on /hosts/ folder.

 Now the script takes <1 second to finish, CPU usage drop and graph are again displayed.

Check the difference:

From 33.36% CPU time on TUE it drops to 0.39% on FRI. The white bands are when graphs are unavailable due to the rrdfix.sh timing out.

Logged
jpgillivan
Full Member
***
Offline Offline

Posts: 31


« Reply #1 on: Wednesday 06 May 2009, 12:13:15 am »

I haven't experienced any problems like that. But we just implemented EFW a  weeks ago.  What are your system specs?

My specs:
Processor: P4 2.8GHZ
Memory:
                   total       used       free     shared    buffers     cached
Mem:        895592     849716      45876          0     151804     188844
-/+ buffers/cache:     509068     386524
Swap:      1790200      34528    1755672

My memory usage was alot lower but then I changed the cache management to:
Cache management
Harddisk cache size (MB) * 500         
Memory cache size (MB) *    200   
Max object size (KB) *    1024
Min object size (KB) *    0

Maximum swap space used is 3.6%
Logged
mrkroket
Hero Member
*****
Offline Offline

Posts: 495


« Reply #2 on: Thursday 14 May 2009, 11:35:08 pm »

I have nothing to do with system specs, it is a bug that time out the rrd graphs.

My specs are a virtualized EFW:
Virtual Guest: EFW
 Dual Core
 2 GB Ram
 40GB HDD

host:
  AMD Phenom 8450 (3x Core at 2.10Ghz)
  4 GB RAM
  600GB HDD
Logged
inteq
Full Member
***
Offline Offline

Posts: 12


« Reply #3 on: Tuesday 26 May 2009, 11:57:36 am »

I post here, so not to start another topic, with related subject.
I do not know what might cause the RRD problem above. Maybe too many clients behind the FW?

My problem is:

I have a constant 4 MB/s upload and most of the time a download of 7-9 MB/s (megabytes) but the graphs show me a max of 550K for one day graph.
Any reason why?
I have to specify that the traffic is made on a forwarded port to a GREEN IP
Logged
mrkroket
Hero Member
*****
Offline Offline

Posts: 495


« Reply #4 on: Friday 19 June 2009, 08:18:20 am »

I post here, so not to start another topic, with related subject.
I do not know what might cause the RRD problem above. Maybe too many clients behind the FW?

My problem is:

I have a constant 4 MB/s upload and most of the time a download of 7-9 MB/s (megabytes) but the graphs show me a max of 550K for one day graph.
Any reason why?
I have to specify that the traffic is made on a forwarded port to a GREEN IP
Forwarded inside GREEN? it's an strange setup.  The normal Firewall use is traffic going from Green to RED. What you have is a GREEN to GREEN Traffic, that is not graphed.
Logged
odtabien
Full Member
***
Offline Offline

Posts: 15


« Reply #5 on: Wednesday 02 September 2009, 07:20:54 pm »

I post here, so not to start another topic, with related subject.
I do not know what might cause the RRD problem above. Maybe too many clients behind the FW?

My problem is:

I have a constant 4 MB/s upload and most of the time a download of 7-9 MB/s (megabytes) but the graphs show me a max of 550K for one day graph.
Any reason why?
I have to specify that the traffic is made on a forwarded port to a GREEN IP

I have the same problem with the network traffic graphs.  we have a 6Mbps connection but the graph peaks at 600Kbps and i know the usage actually goes at 5.98Mbps during peak hours.

i tried to look for the MRTG configuration but didnt find any, maybe they have used another tool for the graph?
Logged
Pages: [1] Go Up Print 
« previous next »
Jump to:  

Page created in 0.109 seconds with 19 queries.
Powered by SMF 1.1 RC2 | SMF © 2001-2005, Lewis Media Design by 7dana.com