|
Title: Squint Reports Post by: nmatese on Tuesday 23 February 2010, 02:16:46 am Has anyone been able to integrate Squint with Endian? I am having trouble with it, I have it installed and running every night but for some reason it only keeps 1 day of history reports. Is there anyone who knows how to keep it accumulating data more than 1 day period.
Thanks, Nick Title: Re: Squint Reports Post by: entourage on Saturday 27 March 2010, 02:32:24 am I just installed it also because I liked the depth of info much better than other log parsers, but I noticed that it was only getting 1 day.
Basically what I found was the the access.log is being reset/archived each night (as it's supposed to) That's why you have so many .gz files in your /var/log/squid folder. So unless there's a way to turn off archiving (which would probably break, or render useless, the native log viewer) or have it create a separate log file that isn't reset each evening. Any ideas anyone? Title: Re: Squint Reports Post by: entourage on Tuesday 06 April 2010, 01:40:20 am Ok, so I'm getting close. The command necessary to complete this is:
Code: cat /var/log/squid/access.log >> /var/log/squid/squint/access.log Then in your squint.cron.sh you can change the LOGDIR="/var/log/squid/squint" That way it's looking in the ../squint directory for the new access.log file which is compiling each night. What I haven't figured out is how to get this to run as a cron job. I'd like to have it run right before the logrotate which happens at 11:57pm each evening. The code appends the current access.log to the end of the new access.log and then squint reads that file. Hope that helps someone. Feel free to correct anything I've gotten wrong to this point. Title: Re: Squint Reports Post by: entourage on Wednesday 07 April 2010, 05:22:13 am Ok, bit of an update...
After much testing I figured out that squint is looking at the 'access.log_short' file instead of 'access.log' First I created a new folder called 'squint' in the /var/log/squid directory Next I modified my /usr/local/bin/squint.cron.sh file to reflect the new LOGDIR value: Code: LOGDIR="/var/log/squid/squint" Code: 30 23 * * * /bin/cat /var/log/squid/access.log_short >> /var/log/squid/squint/access.log_short What this will do is at 11:30pm it will append the current access.log_short to the end of the new access.log_short file in the new squint folder. Hopefully, and this has yet to be tested, the daily, weekly and monthly cron jobs will run approprately since they were generated by the squint install. Here's the output of my fcrontab -l: Code: 15:19:45 listing root's fcrontab Title: Re: Squint Reports Post by: gyp_the_cat on Wednesday 07 April 2010, 05:29:09 am Hi Entourage, wow :) Looking forward to what you manage to get working :)
Title: Re: Squint Reports Post by: nmatese on Wednesday 07 April 2010, 05:30:27 am This does appear to be working, I am also testing it this evening.
Title: Re: Squint Reports Post by: gyp_the_cat on Wednesday 07 April 2010, 05:32:40 am In that case nmatese I'm looking forward to both your findings :)
Title: Re: Squint Reports Post by: entourage on Wednesday 07 April 2010, 06:58:32 am The only downside to this (so far) is that there isn't a live report.
You can create a once-off report to view current stats: Code: cat /var/log/squid/access.log_short | /usr/local/bin/squint.pl /home/httpd/html/instant 0 1270590423 Then browse to http://<ip>/instant *Note: The 1270590423 is unix time. I haven't experimented with other time formats, but you have to have some sort of date range in there (in my case 0 which is Jan 1, 1970 to 1270590423 which is April 6, 2010 @ 4:47pm. Title: Re: Squint Reports Post by: entourage on Wednesday 07 April 2010, 11:01:03 pm Well...it appears that everything worked last night, except I forgot the change my cat cron job to 11:30pm. So it didn't copy over my access.log_short. I just needed one more reboot. DANG
Oh well...try it again tonight! How did things fair for you nmatese? Title: Re: Squint Reports Post by: nmatese on Thursday 08 April 2010, 04:27:43 am Everything seems to have worked on my end, the best test will be this evening when a second days worth of data is added. That is where we ran into trouble before, so I will report back again tomorrow AM.
Thanks, Nick Title: Re: Squint Reports Post by: quickben on Thursday 08 April 2010, 05:21:15 am This sounds exactly like what I need! Any chance of a step by step walk through on installing and configuring?
Title: Re: Squint Reports Post by: entourage on Thursday 08 April 2010, 06:20:30 am The basic installation of squint is relatively easy and included in the INSTALL file included with the gz file.
First, grab the squint.tar.gz file from the site: http://www.ledge.co.za/software/squint/squint.tar.gz (http://www.ledge.co.za/software/squint/squint.tar.gz) If you don't already have it, I recommend getting WinSCP to transfer the squint.tar.gz file to a location on your Endian. For me, I just copied it to the /root folder. Here's a copy of the Installation text: Quote INSTALLATION # tar -zxvf squint.tar.gz I hope this part is easy enough # cp squint.pl squint.cron.sh /usr/local/bin Install the binaries in a nice place # squint.cron.sh init This creates a place for the reports in /usr/local/httpd/htdocs (or whereever the htdocs directory is found). It installs a crontab entry to run the report daily, weekly and monthly. You may wish to edit the crontab entry first -- the system has not been optimised for security, and runs (unnecessarily) as the user root, rather than the user squid. Note that if you run squint.cron.sh at all, it is likely to make crontab entries for you (!) # squint.cron.sh all Create current daily, weekly and monthly reports -- don't do this twice! -- the reports get rotated every time you run the task, so things can get a little messy. # squint.cron.sh regen Recreate all the reports for your initial system, and also for when the program manges to mess things up. You need to run this to maintain weekly and monthly reports if your server happens to miss a cron job. Once you've followed those initial instructions, find my post above about creating the squint folder and then what to modify in the /usr/local/bin/squint.cron.sh file All in all, it's not bad. Let me know if I missed something. Title: Re: Squint Reports Post by: quickben on Thursday 08 April 2010, 08:55:00 am Brilliant! Thanks for that and will post back if it goes pear shaped. I'm also looking at bandwidthd as a solution and should get to try it this weekend on a VM's. Will start a new thread on that topic.
Title: Re: Squint Reports Post by: nmatese on Thursday 08 April 2010, 11:42:30 pm Im happy to report that this is working as intended for me. It is keeping track of multiple days of logs. Thanks so much for your investigation entourage!
Title: Re: Squint Reports Post by: entourage on Thursday 08 April 2010, 11:46:23 pm Excellent! Glad to have helped! Mine seems to be working well also!
Title: Re: Squint Reports Post by: entourage on Tuesday 13 April 2010, 01:40:32 am Ok, might have found a glitch...
Was hoping to see the new 'weekly' report show up which was scheduled to run Monday morning at 1:00am, however I think there's an error in fcrontab: Code: 00 01 * * Monday root /usr/local/bin/squint.cron.sh weekly Code: 00 01 * * 1 root /usr/local/bin/squint.cron.sh weekly Title: Re: Squint Reports Post by: entourage on Thursday 28 October 2010, 02:52:57 am So everything was going along nicely, until my root partition was filled up with a 10GB log file.
Yes, I knew that could happen eventually, but didn't know how fast... Anyway, what I would like to do is purge anything from the log that has a timestamp older than Aug 1st 2010 - Unix timestamp: 1280638801 Anyone have ANY ideas how to do that?? |