Welcome, Guest. Please login or register.
Did you miss your activation email?
Thursday 28 March 2024, 08:03:41 pm

Login with username, password and session length

Visit the official Endian Community Mailinglist  HERE
14247 Posts in 4376 Topics by 6489 Members
Latest Member: GB-gattoboy
Search:     Advanced search
+  EFW Support
|-+  Support
| |-+  EFW SMTP, HTTP, SIP, FTP Proxy Support
| | |-+  Squint Reports
0 Members and 1 Guest are viewing this topic. « previous next »
Pages: 1 2 [All] Go Down Print
Author Topic: Squint Reports  (Read 35437 times)
nmatese
Full Member
***
Offline Offline

Posts: 27


« on: Tuesday 23 February 2010, 02:16:46 am »

Has anyone been able to integrate Squint with Endian?  I am having trouble with it, I have it installed and running every night but for some reason it only keeps 1 day of history reports.  Is there anyone who knows how to keep it accumulating data more than 1 day period.

Thanks,
Nick
Logged
entourage
Full Member
***
Offline Offline

Posts: 48


« Reply #1 on: Saturday 27 March 2010, 02:32:24 am »

I just installed it also because I liked the depth of info much better than other log parsers, but I noticed that it was only getting 1 day.
Basically what I found was the the access.log is being reset/archived each night (as it's supposed to)  That's why you have so many .gz files in your /var/log/squid folder.
So unless there's a way to turn off archiving (which would probably break, or render useless, the native log viewer) or have it create a separate log file that isn't reset each evening.

Any ideas anyone?
Logged
entourage
Full Member
***
Offline Offline

Posts: 48


« Reply #2 on: Tuesday 06 April 2010, 01:40:20 am »

Ok, so I'm getting close.  The command necessary to complete this is:

Code:
cat /var/log/squid/access.log >> /var/log/squid/squint/access.log

Then in your squint.cron.sh you can change the LOGDIR="/var/log/squid/squint"

That way it's looking in the ../squint directory for the new access.log file which is compiling each night.

What I haven't figured out is how to get this to run as a cron job. I'd like to have it run right before the logrotate which happens at 11:57pm each evening.  The code appends the current access.log to the end of the new access.log and then squint reads that file.

Hope that helps someone.  Feel free to correct anything I've gotten wrong to this point.
Logged
entourage
Full Member
***
Offline Offline

Posts: 48


« Reply #3 on: Wednesday 07 April 2010, 05:22:13 am »

Ok, bit of an update...
After much testing I figured out that squint is looking at the 'access.log_short' file instead of 'access.log'

First I created a new folder called 'squint' in the /var/log/squid directory
Next I modified my /usr/local/bin/squint.cron.sh file to reflect the new LOGDIR value:
Code:
LOGDIR="/var/log/squid/squint"
Then I created a file in /etc/cron.d called 'logsquint' (Doesn't matter the name) that contains:
Code:
30 23 * * * /bin/cat /var/log/squid/access.log_short >> /var/log/squid/squint/access.log_short

What this will do is at 11:30pm it will append the current access.log_short to the end of the new access.log_short file in the new squint folder.

Hopefully, and this has yet to be tested, the daily, weekly and monthly cron jobs will run approprately since they were generated by the squint install.  Here's the output of my fcrontab -l:
Code:
15:19:45 listing root's fcrontab
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=
HOME=/

#
&nolog */1 * * * *      [ -x /bin/run-parts ] && run-parts --report /etc/cron.minutely
&nolog */5 * * * *      [ -x /bin/run-parts ] && run-parts --report /etc/cron.cyclic
01  * * * *     [ -x /bin/run-parts ] && run-parts --report /etc/cron.hourly
25  1 * * *     [ -x /bin/run-parts ] && run-parts --report /etc/cron.daily
47  2 * * 0     [ -x /bin/run-parts ] && run-parts --report /etc/cron.weekly
52  3 1 * *     [ -x /bin/run-parts ] && run-parts --report /etc/cron.monthly

#
@nolog 5        [ -x /bin/run-parts ] && run-parts --report /etc/anacron.cyclic
@ 1h    [ -x /bin/run-parts ] && run-parts --report /etc/anacron.hourly
@ 1d    [ -x /bin/run-parts ] && run-parts --report /etc/anacron.daily
@ 1w    [ -x /bin/run-parts ] && run-parts --report /etc/anacron.weekly
@ 1m    [ -x /bin/run-parts ] && run-parts --report /etc/anacron.monthly

# Summarize ip accounting info:
2 1 * * *       /usr/sbin/ipacsum -r -H `/bin/hostname` -t "the day 2 days ago" >/dev/null
3 1 * * 0       /usr/sbin/ipacsum -r -H `/bin/hostname` -t "the week 4 weeks ago" >/dev/null
4 1 1 2 *       /usr/sbin/ipacsum -r -H `/bin/hostname` -t "the year 2 years ago" >/dev/null
# squint squid reports
# Weekly, on Mondays
00 01 * * Monday  root /usr/local/bin/squint.cron.sh weekly
# Monthly, on the first day of the month
00 02 1 * *    root /usr/local/bin/squint.cron.sh monthly
# Daily at 3am
00 03 * * *    root /usr/local/bin/squint.cron.sh daily
57 23 * * *     [ -x /usr/lib/cron/logrotate.cron ] && /usr/lib/cron/logrotate.cron
00 15 * * * /bin/cat /var/log/squid/access.log_short >> /var/log/squid/squint/access.log_short
00 23 * * *     [ -x /usr/lib/cron/sargdaily.cron ] && /usr/lib/cron/sargdaily.cron
I'll post back tomorrow with hopefully positive results!
Logged
gyp_the_cat
Full Member
***
Offline Offline

Posts: 81



WWW
« Reply #4 on: Wednesday 07 April 2010, 05:29:09 am »

Hi Entourage, wow Smiley  Looking forward to what you manage to get working Smiley
Logged
nmatese
Full Member
***
Offline Offline

Posts: 27


« Reply #5 on: Wednesday 07 April 2010, 05:30:27 am »

This does appear to be working, I am also testing it this evening. 
Logged
gyp_the_cat
Full Member
***
Offline Offline

Posts: 81



WWW
« Reply #6 on: Wednesday 07 April 2010, 05:32:40 am »

In that case nmatese I'm looking forward to both your findings Smiley
Logged
entourage
Full Member
***
Offline Offline

Posts: 48


« Reply #7 on: Wednesday 07 April 2010, 06:58:32 am »

The only downside to this (so far) is that there isn't a live report.
You can create a once-off report to view current stats:
Code:
cat /var/log/squid/access.log_short | /usr/local/bin/squint.pl /home/httpd/html/instant 0 1270590423

Then browse to http://<ip>/instant

*Note: The 1270590423 is unix time.  I haven't experimented with other time formats, but you have to have some sort of date range in there (in my case 0 which is Jan 1, 1970 to 1270590423 which is April 6, 2010 @ 4:47pm.
Logged
entourage
Full Member
***
Offline Offline

Posts: 48


« Reply #8 on: Wednesday 07 April 2010, 11:01:03 pm »

Well...it appears that everything worked last night, except I forgot the change my cat cron job to 11:30pm.  So it didn't copy over my access.log_short.  I just needed one more reboot.  DANG

Oh well...try it again tonight!

How did things fair for you nmatese?
Logged
nmatese
Full Member
***
Offline Offline

Posts: 27


« Reply #9 on: Thursday 08 April 2010, 04:27:43 am »

Everything seems to have worked on my end, the best test will be this evening when a second days worth of data is added.  That is where we ran into trouble before, so I will report back again tomorrow AM.

Thanks,
Nick
Logged
quickben
Jr. Member
*
Offline Offline

Posts: 8


« Reply #10 on: Thursday 08 April 2010, 05:21:15 am »

This sounds exactly like what I need! Any chance of a step by step walk through on installing and configuring?
Logged
entourage
Full Member
***
Offline Offline

Posts: 48


« Reply #11 on: Thursday 08 April 2010, 06:20:30 am »

The basic installation of squint is relatively easy and included in the INSTALL file included with the gz file.
First, grab the squint.tar.gz file from the site:
http://www.ledge.co.za/software/squint/squint.tar.gz

If you don't already have it, I recommend getting WinSCP to transfer the squint.tar.gz file to a location on your Endian.  For me, I just copied it to the /root folder.
Here's a copy of the Installation text:
Quote
INSTALLATION

 # tar -zxvf squint.tar.gz
   I hope this part is easy enough

 # cp squint.pl squint.cron.sh /usr/local/bin
    Install the binaries in a nice place

 # squint.cron.sh init
   This creates a place for the reports in
   /usr/local/httpd/htdocs (or whereever the htdocs
   directory is found).  It installs a crontab entry to run
   the report daily, weekly and monthly.  You may wish to
   edit the crontab entry first -- the system has not been
   optimised for security, and runs (unnecessarily) as the
   user root, rather than the user squid.

   Note that if you run squint.cron.sh at all, it is likely to make
   crontab entries for you (!)

 # squint.cron.sh all
   Create current daily, weekly and monthly reports -- don't
   do this twice! -- the reports get rotated every time you
   run the task, so things can get a little messy.

 # squint.cron.sh regen
   Recreate all the reports for your initial system, and
   also for when the program manges to mess things up.  You
   need to run this to maintain weekly and monthly reports
   if your server happens to miss a cron job.

Once you've followed those initial instructions, find my post above about creating the squint folder and then what to modify in  the /usr/local/bin/squint.cron.sh file

All in all, it's not bad.  Let me know if I missed something.
Logged
quickben
Jr. Member
*
Offline Offline

Posts: 8


« Reply #12 on: Thursday 08 April 2010, 08:55:00 am »

Brilliant! Thanks for that and will post back if it goes pear shaped. I'm also looking at bandwidthd as a solution and should get to try it this weekend on a  VM's. Will start a new thread on that topic.
Logged
nmatese
Full Member
***
Offline Offline

Posts: 27


« Reply #13 on: Thursday 08 April 2010, 11:42:30 pm »

Im happy to report that this is working as intended for me.  It is keeping track of multiple days of logs.  Thanks so much for your investigation entourage!
Logged
entourage
Full Member
***
Offline Offline

Posts: 48


« Reply #14 on: Thursday 08 April 2010, 11:46:23 pm »

Excellent!  Glad to have helped!  Mine seems to be working well also!
Logged
entourage
Full Member
***
Offline Offline

Posts: 48


« Reply #15 on: Tuesday 13 April 2010, 01:40:32 am »

Ok, might have found a glitch...
Was hoping to see the new 'weekly' report show up which was scheduled to run Monday morning at 1:00am, however I think there's an error in fcrontab:
Code:
00 01 * * Monday  root /usr/local/bin/squint.cron.sh weekly
I don't know if it matters or not, but I think that instead of 'Monday' it was supposed to be '1' so the whole line should have read:
Code:
00 01 * * 1  root /usr/local/bin/squint.cron.sh weekly
If anyone knows how to change it, let me know because my attempts to edit fcrontab have yet-so-far failed.
Logged
entourage
Full Member
***
Offline Offline

Posts: 48


« Reply #16 on: Thursday 28 October 2010, 02:52:57 am »

So everything was going along nicely, until my root partition was filled up with a 10GB log file.

Yes, I knew that could happen eventually, but didn't know how fast...

Anyway, what I would like to do is purge anything from the log that has a timestamp older than Aug 1st 2010 - Unix timestamp: 1280638801

Anyone have ANY ideas how to do that??
Logged
Pages: 1 2 [All] Go Up Print 
« previous next »
Jump to:  

Page created in 0.094 seconds with 18 queries.
Powered by SMF 1.1 RC2 | SMF © 2001-2005, Lewis Media Design by 7dana.com