|
Title: Greater Blacklist Post by: finchwizard on Wednesday 26 April 2006, 12:43:20 pm I'm still working on it, but here's a screenshot of what I have so far.
Hopefully I'll release a script, or at least manual instructions for it. But the current Blacklist is very minimal and I personally need far more blacklists. So far mine is looking like this and seems to be working. (http://img137.imageshack.us/img137/646/block4ct.th.jpg) (http://img137.imageshack.us/my.php?image=block4ct.jpg) Will post back when I get it all working. Title: Re: Greater Blacklist Post by: euser4life on Friday 31 August 2007, 06:55:00 am This is probably widely known but for some like myself it is good information. You can easily update dansguardian blacklist within endian firewall with the newest list of sites / categories by downloading the list from:
<http://urlblacklist.com/?sec=download> Then (for windows user's use 7-zip to decompress into a folder called blacklists) and use a SFTP client (such as WinSCP) to upload the list to the /etc/dansguardian/blacklists directory. Once you refresh the page new categories will be there for you to choose from. I'm sure there are ways to automate (script) updates. The first download is free, they don't block additional downloads but go off of the honor system since they are providing a service. It works very well. Title: Re: Greater Blacklist Post by: Hinze57 on Wednesday 08 October 2008, 02:21:42 pm Can you post at least some raw instructions?
I can get the blacklists but am trying to figure out if I need to update the categories file. Thanks, K Title: Re: Greater Blacklist Post by: wharfratjoe on Monday 12 January 2009, 08:22:48 am Does this work on 2.2 RC3?
Title: Re: Greater Blacklist Post by: zman818 on Sunday 18 January 2009, 10:44:24 am I second that question. It seems that the categories file format is different between the current EFW and current blacklist files... ?
Title: Re: Greater Blacklist Post by: khhkom on Friday 20 February 2009, 01:15:30 am any update on this? even after just replacing the domains/urls of some categories dansguardian is stuck with 100% cpu :(
Title: Re: Greater Blacklist Post by: hinge on Monday 08 June 2009, 05:34:24 pm Itry this configuration to my content filter but my network cannot connect any website? they have any configuration i can do? i using 2.2 rc3.. thank you
Title: Re: Greater Blacklist Post by: davvidde on Wednesday 17 June 2009, 01:31:59 am I confirm: the big blacklist from urlblacklist.com do not works because the CPU got 100% for over two hours on a VM (1024MB) on HP proliant QuadCore 2.50GHz with VMWare ESXi 3.5 and at the end the dansguardian process do not start (terminated with segmentation fault). I try this on EFW 2.2rc3 and EFW2.2 final. I even substitute the /etc/dansguardian/blacklists/CATEGORIES provided by urlblacklist.com with the original provided by endian (the two formats are a bit different) but the problems seems to be the compilation of lists. I even notice the strange behaviour on my virtualbox VM installation: the compilation of the lists do not get over the eleventh. The lists processed are always 11 in total when the categories are 86. This appears on all my three VM on test.
May anyone post dansguardian detailed logs to compare? Title: Re: Greater Blacklist Post by: maurelio on Thursday 18 June 2009, 11:49:11 am Hi,
I had the same problem in version 2.1.2, but found that the way that files are being provided by the URLBlacklist is incompatible with the Dansguardian. These files are indexed alphabetically and Dansguardian to work with does not know this. Just to give you rearrange the files to a random order. I would like to know how to do this update in version 2.2 final, because the formats are different and the obstacles that exist by default are very old. []īs Title: Re: Greater Blacklist Post by: davvidde on Friday 19 June 2009, 04:58:18 pm I know of this "bad" implementation of Dansguardian which use a quicksort algorithm to sort an already sorted list but I don't understand how Dansguardian in Ubuntu 8.04 processes without problem, in less than a minute, the list provided by URLBLACKLIST.COM; so i decided to copy the already sorted from Ubuntu to Endian and Dansguardian restart without problem in few seconds. By the way I tested this in EFW2.2 final but this release suffers, in my testing, of authentication problem with LDAP that EFW2.2rc3 does not have. See post http://efwsupport.com/index.php?topic=673.0
Windows authentication in EFW 2.2 also do not work with group policies but I don'k know if this is by design. Title: Re: Greater Blacklist Post by: entourage on Thursday 09 July 2009, 05:27:56 am I ran into the same problem that you guys had with the URLBLACKLIST.COM killing my install. I was wondering if Shalla's list http://www.shallalist.de/ (http://www.shallalist.de/) would work any better. It would be nice if wget were installed, so we could run the scripts.
I'm going to manually download the list and move the folders over via WinSCP. Wish me luck! Title: Re: Greater Blacklist Post by: entourage on Thursday 09 July 2009, 06:49:32 am Ok, I've downloaded and unzipped the Shalla list.
I removed all of the current folders in /etc/dansguardian/blacklists and copied over the new unzipped folders. I then modified the categories file to reflect the new folders. I did it the quick and dirty way where they each have their own category: Code: Adv=adv I refreshed the Content Filter page and selected 10 or so categories to block. Once I hit 'Save' I would say it took approximately 2 minutes to refresh, but it clocked my selected categories. I noticed I missed a , so I selected about 5 more and hit 'Save' again. This time it was nearly instant. So as for manually up, this will work for me, but an automated script would be MUCH nicer! Hope that helps someone. Title: Re: Greater Blacklist Post by: Baggers on Saturday 01 August 2009, 11:24:55 pm Thanks, I've just automated this by firstly getting wget onto my box based on clues from here...
http://www.howtoforge.com/install-vmware-tools-on-endian-firewall-community-2.2-rc3-on-esxi-3.5 ....and then writing the following script and sticking it in /etc/cron.weekly.... #!/bin/sh wget -Y no http://www.shallalist.de/Downloads/shallalist.tar.gz -O /tmp/shallalist.tar.gz cd /tmp tar -xvzf shallalist.tar.gz cp -r BL/* /etc/dansguardian/blacklists/ rm -r /etc/dansguardian/blacklists/COPYRIGHT rm -r /etc/dansguardian/blacklists/global_usage date > /etc/dansguardian/blacklists/updated /etc/init.d/dansguardian restart exit 0 ... then I modified /home/httpd/cgi-bin/dansguardian.cgi so its displays when the blacklist has been updated in the proxy / content filter section by adding.... $output = `cat /etc/dansguardian/blacklists/updated`; $output = &cleanhtml($output,"y"); print "URL Blacklist Date : $output\n"; Hope someone finds this useful! :) Baggers Title: Re: Greater Blacklist Post by: hinge on Monday 03 August 2009, 06:05:35 pm hi! baggers.. this is work to EFW community 2.2? thank you..
Title: Re: Greater Blacklist Post by: Baggers on Tuesday 04 August 2009, 03:55:41 pm Yes. Thats the version i'm using.
Title: Re: Greater Blacklist Post by: hinge on Tuesday 04 August 2009, 06:15:06 pm hi! baggers :) can you give me a screenshoot what happen to your endian when you try this.. because i did not sacrifice may Endian. actually im beginner this kind of setting and i have scare to try and my endian is only one and i dont have any machine to try this... thank you for your understand.. and thank you for giving an idea.. thank you..
Title: Re: Greater Blacklist Post by: Baggers on Wednesday 05 August 2009, 04:03:05 am Hi Hinge,
Which bit are you stuck with and needing clarification on? As long as you make a copy of any files before they are changed you should be fine.... but if you really don't have a 'plan B' if your firewall goes 'up the spout' and would have no immediate access to the resources to fix it then you probably shouldn't try it! Do you know how to restore everything if it did go wrong? Title: Re: Greater Blacklist Post by: vondie on Saturday 15 August 2009, 11:12:13 pm just one question... if you enable the whole "A/D/U/L/T" filter on the "URL Blacklist" part of Content Filter... is it working? I've try it so many time to make it work but no luck.... i've even evaluate the size of the file of a/d/u/l/t file... it is super huge.... comparing to other url blacklist category or shalla's category....
Title: Re: Greater Blacklist Post by: jaebird on Sunday 16 August 2009, 03:45:10 am Here is a little contribution to this thread.
curl is installed by default with endian...so a quick mod of the script means you don't need wget: #!/bin/sh cd /tmp curl -C - -O http://www.shallalist.de/Downloads/shallalist.tar.gz tar -xvzf shallalist.tar.gz cp -r BL/* /etc/dansguardian/blacklists/ rm -r /etc/dansguardian/blacklists/COPYRIGHT rm -r /etc/dansguardian/blacklists/global_usage date > /etc/dansguardian/blacklists/updated /etc/init.d/dansguardian restart exit 0 My change to the cgi file is below, it puts the update date in the box title. In /home/httpd/cgi-bin/dansguardian.cgi find: openbox('100%', 'left', _('URL Blacklist: Filter pages known to have content of the following categories.'), "blacklist"); Replace with this: $output = `cat /etc/dansguardian/blacklists/updated`; $output = &cleanhtml($output,"y"); openbox('100%', 'left', _("URL Blacklist: Filter pages known to have content of the following categories. Update Date: $output"), "blacklist"); Notice the change from single to double quotes around the string. Title: Re: Greater Blacklist Post by: entourage on Monday 17 August 2009, 11:56:49 pm I just got around to adding this to my Endian 2.2 install and I can confirm it works GREAT!!!
Thanks for the effort! BTW, if you're a business, don't forget to sign the Shallalist agreement. It's free to use and they were very good about getting the info to me. Title: Re: Greater Blacklist Post by: Dougie on Tuesday 18 August 2009, 08:41:48 pm Thanks worked for me. Community 2.2.
Title: Re: Greater Blacklist Post by: whoiam55 on Tuesday 30 March 2010, 11:34:10 pm Today while reading up on urlblacklist.com I stuble upon this
Quote Q 10. What do you mean about needing unsorting with DansGuardian? The blacklist is provided as a gzipped tar file with directories containing text file lists. These lists are now provided in a sorted ordered list. DansGuardian, when reading in the files, will in memory sort the file. It needs to sort them to be able to quickly search them when doing the filtering. However DansGuardian uses the Quick Sort algorithm which is very quick to sort random lists but it will take a very LONG time to sort a list that is already sorted. DansGuardian will eventually start. If it does not then you might have a very old version with a bug. The input URL list being sorted should not affect any script or program and stop it actually working. Q 11. How do I unsort the blacklist? Unsorting a single file is very easy. Simply use the 'rl' or 'unsort' unix command on each of the text files. You can find information on these here: http://ch.tudelft.nl/~arthur/rl/ http://www.vanheusden.com/unsort/ To unsort all the files in the blacklist in one go you could: cd blacklistdir; find . -type f -exec rl '{}' -o '{}'.tmp \; -exec mv -f '{}'.tmp '{}' \; This will find all the files in blacklistdir and subdirectories and randomize the lines. It should only take a few seconds if that. If you are using unsort instead of rl you will have to edit the above. Pretty straight forward I guess. Title: Re: Greater Blacklist Post by: g.vecchi on Friday 10 June 2011, 02:11:46 am Hi all!
I'm new of EndianFW and I'm trying to create a better blacklist for Content Filtering. I'm running EFW Community 2.4.1 and I've modified jaebird script in this way: Code: #!/bin/sh url_to_blacklist = bigblacklist url <--- sorry for that but forum doen't allow me to post clickable link but dansguardian failed to start because it says that /etc/dansguardian/blacklists/violence/expressions doesn't exist So, I've modified /etc/dansguardian/profiles/1/bannedregexpurllist removing line about /etc/dansguardian/blacklists/violence/expressions and dansguardian restart to work: is that a good idea??? Thanks to all! Title: Re: Greater Blacklist Post by: portucale on Tuesday 04 October 2011, 10:19:43 pm thanks to g.vecchi and addicional info i've read in other foruns, my script looks like this:
(create the file update_blacklist.sh and save it in /etc/cron.weekly/) Code: #!/bin/sh IMPORTANT replace SITE in the line curl -C - -O SITE with a blacklist file like shallalist.tar.gz then in the file /etc/dansguardian/profiles/1/bannedregexpurllist remove the line with /etc/dansguardian/blacklists/violence/expressions save the file and in shell do (assuming the file you've create the file update_blacklist.sh in /etc/cron.weekly/ Code: cd /etc/cron.weekly/ this will execute the script go to endian->proxy->contentfilter->edit and in the tab Filter pages known to have content of the following categories. (URL Blacklist) select the categories you want to block after that you can restart dansguardian doing in shell Code: /etc/init.d/dansguardian restart all should all get's ok! |