Squint Reports

[Guest]

[nmatese]

Has anyone been able to integrate Squint with Endian?  I am having trouble with it, I have it installed and running every night but for some reason it only keeps 1 day of history reports.  Is there anyone who knows how to keep it accumulating data more than 1 day period.

Thanks,
Nick

[entourage]

I just installed it also because I liked the depth of info much better than other log parsers, but I noticed that it was only getting 1 day.
Basically what I found was the the access.log is being reset/archived each night (as it's supposed to)  That's why you have so many .gz files in your /var/log/squid folder.
So unless there's a way to turn off archiving (which would probably break, or render useless, the native log viewer) or have it create a separate log file that isn't reset each evening.

Any ideas anyone?

[entourage]

Ok, so I'm getting close.  The command necessary to complete this is:

Code:

cat /var/log/squid/access.log >> /var/log/squid/squint/access.log

Then in your squint.cron.sh you can change the LOGDIR="/var/log/squid/squint"

That way it's looking in the ../squint directory for the new access.log file which is compiling each night.

What I haven't figured out is how to get this to run as a cron job. I'd like to have it run right before the logrotate which happens at 11:57pm each evening.  The code appends the current access.log to the end of the new access.log and then squint reads that file.

Hope that helps someone.  Feel free to correct anything I've gotten wrong to this point.

[entourage]

Ok, bit of an update...
After much testing I figured out that squint is looking at the 'access.log_short' file instead of 'access.log'

First I created a new folder called 'squint' in the /var/log/squid directory
Next I modified my /usr/local/bin/squint.cron.sh file to reflect the new LOGDIR value:

Code:

LOGDIR="/var/log/squid/squint"

Then I created a file in /etc/cron.d called 'logsquint' (Doesn't matter the name) that contains:

Code:

30 23 * * * /bin/cat /var/log/squid/access.log_short >> /var/log/squid/squint/access.log_short

What this will do is at 11:30pm it will append the current access.log_short to the end of the new access.log_short file in the new squint folder.

Hopefully, and this has yet to be tested, the daily, weekly and monthly cron jobs will run approprately since they were generated by the squint install.  Here's the output of my fcrontab -l:

Code:

15:19:45 listing root's fcrontab
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=
HOME=/

#
&nolog */1 * * * *      [ -x /bin/run-parts ] && run-parts --report /etc/cron.minutely
&nolog */5 * * * *      [ -x /bin/run-parts ] && run-parts --report /etc/cron.cyclic
01  * * * *     [ -x /bin/run-parts ] && run-parts --report /etc/cron.hourly
25  1 * * *     [ -x /bin/run-parts ] && run-parts --report /etc/cron.daily
47  2 * * 0     [ -x /bin/run-parts ] && run-parts --report /etc/cron.weekly
52  3 1 * *     [ -x /bin/run-parts ] && run-parts --report /etc/cron.monthly

#
@nolog 5        [ -x /bin/run-parts ] && run-parts --report /etc/anacron.cyclic
@ 1h    [ -x /bin/run-parts ] && run-parts --report /etc/anacron.hourly
@ 1d    [ -x /bin/run-parts ] && run-parts --report /etc/anacron.daily
@ 1w    [ -x /bin/run-parts ] && run-parts --report /etc/anacron.weekly
@ 1m    [ -x /bin/run-parts ] && run-parts --report /etc/anacron.monthly

# Summarize ip accounting info:
2 1 * * *       /usr/sbin/ipacsum -r -H `/bin/hostname` -t "the day 2 days ago" >/dev/null
3 1 * * 0       /usr/sbin/ipacsum -r -H `/bin/hostname` -t "the week 4 weeks ago" >/dev/null
4 1 1 2 *       /usr/sbin/ipacsum -r -H `/bin/hostname` -t "the year 2 years ago" >/dev/null
# squint squid reports
# Weekly, on Mondays
00 01 * * Monday  root /usr/local/bin/squint.cron.sh weekly
# Monthly, on the first day of the month
00 02 1 * *    root /usr/local/bin/squint.cron.sh monthly
# Daily at 3am
00 03 * * *    root /usr/local/bin/squint.cron.sh daily
57 23 * * *     [ -x /usr/lib/cron/logrotate.cron ] && /usr/lib/cron/logrotate.cron
00 15 * * * /bin/cat /var/log/squid/access.log_short >> /var/log/squid/squint/access.log_short
00 23 * * *     [ -x /usr/lib/cron/sargdaily.cron ] && /usr/lib/cron/sargdaily.cron

I'll post back tomorrow with hopefully positive results!

[gyp_the_cat]

Hi Entourage, wow   Looking forward to what you manage to get working

[nmatese]

This does appear to be working, I am also testing it this evening. 

[gyp_the_cat]

In that case nmatese I'm looking forward to both your findings

[entourage]

The only downside to this (so far) is that there isn't a live report.
You can create a once-off report to view current stats:

Code:

cat /var/log/squid/access.log_short | /usr/local/bin/squint.pl /home/httpd/html/instant 0 1270590423

Then browse to http://<ip>/instant

*Note: The 1270590423 is unix time.  I haven't experimented with other time formats, but you have to have some sort of date range in there (in my case 0 which is Jan 1, 1970 to 1270590423 which is April 6, 2010 @ 4:47pm.

[entourage]

Well...it appears that everything worked last night, except I forgot the change my cat cron job to 11:30pm.  So it didn't copy over my access.log_short.  I just needed one more reboot.  DANG

Oh well...try it again tonight!

How did things fair for you nmatese?

[nmatese]

Everything seems to have worked on my end, the best test will be this evening when a second days worth of data is added.  That is where we ran into trouble before, so I will report back again tomorrow AM.

Thanks,
Nick

[quickben]

This sounds exactly like what I need! Any chance of a step by step walk through on installing and configuring?

[entourage]

The basic installation of squint is relatively easy and included in the INSTALL file included with the gz file.
First, grab the squint.tar.gz file from the site:
http://www.ledge.co.za/software/squint/squint.tar.gz

If you don't already have it, I recommend getting WinSCP to transfer the squint.tar.gz file to a location on your Endian.  For me, I just copied it to the /root folder.
Here's a copy of the Installation text:

INSTALLATION

 # tar -zxvf squint.tar.gz
   I hope this part is easy enough

 # cp squint.pl squint.cron.sh /usr/local/bin
    Install the binaries in a nice place

 # squint.cron.sh init
   This creates a place for the reports in
   /usr/local/httpd/htdocs (or whereever the htdocs
   directory is found).  It installs a crontab entry to run
   the report daily, weekly and monthly.  You may wish to
   edit the crontab entry first -- the system has not been
   optimised for security, and runs (unnecessarily) as the
   user root, rather than the user squid.

   Note that if you run squint.cron.sh at all, it is likely to make
   crontab entries for you (!)

 # squint.cron.sh all
   Create current daily, weekly and monthly reports -- don't
   do this twice! -- the reports get rotated every time you
   run the task, so things can get a little messy.

 # squint.cron.sh regen
   Recreate all the reports for your initial system, and
   also for when the program manges to mess things up.  You
   need to run this to maintain weekly and monthly reports
   if your server happens to miss a cron job.

Once you've followed those initial instructions, find my post above about creating the squint folder and then what to modify in  the /usr/local/bin/squint.cron.sh file

All in all, it's not bad.  Let me know if I missed something.

[quickben]

Brilliant! Thanks for that and will post back if it goes pear shaped. I'm also looking at bandwidthd as a solution and should get to try it this weekend on a  VM's. Will start a new thread on that topic.

[nmatese]

Im happy to report that this is working as intended for me.  It is keeping track of multiple days of logs.  Thanks so much for your investigation entourage!

[entourage]

Excellent!  Glad to have helped!  Mine seems to be working well also!