Print Page - Blacklist Update Script

Title: Blacklist Update Script - Download
Post by: Gavin on Friday 13 November 2009, 02:43:29 am

Hey EFW Community,

I've written a python script to update the content filter blacklist automatically.

The script downloads the latest copy from shallalist, copies into the dansguardian blacklist directory and updates the categories accordingly.

To use it, scp the "blacklist_update.py" file to the "/tmp" directory on your EFW. Then, go to the "/tmp" directory on your EFW and run the command "./blacklist_updater.py".

I currently use this on Endian Firewall Community release 2.3.0.

Download Here:
http://geekforbrains.com/scripts/blacklist_updater.py (http://geekforbrains.com/scripts/blacklist_updater.py)

Gavin

Title: Re: Blacklist Update Script - Download
Post by: NinNin on Friday 13 November 2009, 11:39:45 pm

Thank you very much. :)

Title: Re: Blacklist Update Script - Download
Post by: Gavin on Saturday 14 November 2009, 03:00:06 am

My pleasure! ;D

One thing I forgot to mention is that this script will overwrite the current blacklist. If you want to keep the original, you may want to back it up first. You can do so by running the following command:

Code:

cp -ruv /etc/dansguardian/blacklists /etc/dansguardian/blacklists_orig

Title: Re: Blacklist Update Script - Download
Post by: sepp on Tuesday 17 November 2009, 08:32:39 pm

hi,
when I use your script to update the Endian, the pages wouldn't be filtered exactly. For example in the Category "s" there is a entry for "youtube.com" and "youtube.com" is filtered but "youtube.com/de" would't be filtered. Is there something wrong in the script ?

Is it possible that Endian only works with the "urls-file" and not with the "domains" file ?

Title: Re: Blacklist Update Script - Download
Post by: Gavin on Wednesday 18 November 2009, 02:54:50 am

Hi sepp,

I had the exact same problem you are. Its a bug in the restart file for dansguardian. The file is located at:

Code:

/usr/local/bin/restartdansguardian.py

Here is a cut-out of that file:

Code:

#exceptionsitefile = open("/etc/dansguardian/profiles/%s/exceptionsitelist" % number, "w")
exceptionsitefile = open("/etc/dansguardian/profiles/%s/exceptionsitelist" % number, "a")
exceptionsitefile.write(exceptionsitelist)
exceptionsitefile.close()

bannedsitelist = ""
if file_exists("%s/bannedsitelist" % path):
    bannedsitefile = open("%s/bannedsitelist" % path, "r")
    bannedsitelist = bannedsitefile.read()
    bannedsitefile.close()

#bannedsitefile = open("/etc/dansguardian/profiles/%s/bannedsitelist" % number, "w")
bannedsitefile = open("/etc/dansguardian/profiles/%s/bannedsitelist" % number, "a")
bannedsitefile.write(bannedsitelist)
bannedsitefile.close()

The lines I've commented out are the originals. Basically you're replacing the "w" with "a", which means append instead of write. The lines start around line 308.

Title: Re: Blacklist Update Script - Download
Post by: mrkroket on Wednesday 18 November 2009, 04:42:21 am

Two suggestions:

1- Put a link of that post on Customization forum, to group all custom mods/scripts together
2- A button on the HTTP Proxy page that exec this script. This way one can run the updater from the GUI, without much hassle

Title: Re: Blacklist Update Script - Download
Post by: nmatese on Friday 20 November 2009, 04:51:19 am

I really appreciate the update, this is great add-on.

Title: Re: Blacklist Update Script - Download
Post by: ragr91279 on Saturday 28 November 2009, 05:18:57 am

Hi, I have tried using your script, but I receive the following error

: bad interpreter: No such file or directory

in the / tmp folder I copied the script

What am I doing wrong?

Title: Re: Blacklist Update Script - Download
Post by: Gavin on Saturday 28 November 2009, 05:44:53 am

Sounds like its having an issue finding Python. What version of EFW are you running?

Try running the command:

Code:

whereis python

My output is:

Code:

python: /usr/bin/python2.4 /usr/bin/python /usr/lib/python2.4

Title: Re: Blacklist Update Script - Download
Post by: ragr91279 on Saturday 28 November 2009, 06:25:02 am

I have version 2.3, I will try what you say

Title: Re: Blacklist Update Script - Download
Post by: ragr91279 on Saturday 28 November 2009, 06:28:30 am

Quote from: Gavin on Saturday 28 November 2009, 05:44:53 am

Sounds like its having an issue finding Python. What version of EFW are you running?

Try running the command:

Code:

whereis python

My output is:

Code:

python: /usr/bin/python2.4 /usr/bin/python /usr/lib/python2.4

I ran the command you say and the output is equal to yours

python: /usr/bin/python2.4 /usr/bin/python /usr/lib/python2.4

Title: Re: Blacklist Update Script - Download
Post by: Gavin on Saturday 28 November 2009, 06:31:14 am

Hmm, how are you running the script? Is it executable?

Try running:

Code:

chmod u+x blacklist_updater.py

Then re-run the script:

Code:

./blacklist_updater.py

Does it work now?

Title: Re: Blacklist Update Script - Download
Post by: ragr91279 on Saturday 28 November 2009, 06:38:25 am

Quote from: Gavin on Saturday 28 November 2009, 06:31:14 am

Hmm, how are you running the script? Is it executable?

Try running:

Code:

chmod u+x blacklist_updater.py

Then re-run the script:

Code:

./blacklist_updater.py

Does it work now?

I'm afraid not, still shows the same output as before

root@Firewall:/tmp # ./blacklist_updater.py

: bad interpreter: No such file or directoryon

Title: Re: Blacklist Update Script - Download
Post by: ragr91279 on Saturday 28 November 2009, 06:44:52 am

if I run the command like this:

Code:

bash ./blacklist_updater.py

this is the output:

Code:

root@Firewall:/tmp # bash ./blacklist_updater.py
./blacklist_updater.py: line 2: import: command not found
./blacklist_updater.py: line 3: import: command not found
: command not found.py: line 4:
./blacklist_updater.py: line 5: print: command not found
./blacklist_updater.py: line 6: print: command not found
./blacklist_updater.py: line 7: print: command not found
: command not found.py: line 8:
./blacklist_updater.py: line 9: sourceURL: command not found
./blacklist_updater.py: line 10: sourceFile: command not found
./blacklist_updater.py: line 11: blacklistDir: command not found
./blacklist_updater.py: line 12: categoryFile: command not found
: command not found.py: line 13:
./blacklist_updater.py: line 17: print: command not found
./blacklist_updater.py: line 18: syntax error near unexpected token `('
'/blacklist_updater.py: line 18: `source = urllib2.urlopen(sourceURL)

Title: Re: Blacklist Update Script - Download
Post by: Gavin on Saturday 28 November 2009, 06:48:01 am

Remove "bash" from the command. Your executing a python script, not a bash script.

Code:

root@Firewall:/tmp ./blacklist_updater.py

Title: Re: Blacklist Update Script - Download
Post by: ragr91279 on Saturday 28 November 2009, 06:51:57 am

I get the same error

Code:

root@Firewall:/tmp # ./blacklist_updater.py
: bad interpreter: No such file or directoryon

Title: Re: Blacklist Update Script - Download
Post by: Gavin on Saturday 28 November 2009, 06:53:24 am

Weird...maybe try running it via python directly.

Code:

python blacklist_updater.py

Title: Re: Blacklist Update Script - Download
Post by: ragr91279 on Saturday 28 November 2009, 06:57:02 am

excellent, now if it works. I'll tell you if everything worked. thank you very much for your help

Code:

root@PDH-Firewall:/tmp # python blacklist_updater.py
Blacklist Updater for EFW 2.3
Written by Gavin Vickery <gdvickery@gmail.com>
==============================
Stage 1 of 5 | Downloading blacklist, please wait...
Stage 2 of 5 | Extracting files
Stage 3 of 5 | Updating blacklist
Stage 4 of 5 | Updating categories
Stage 5 of 5 | Cleaning up
==============================
All done! Enjoy ;)

Title: Re: Blacklist Update Script - Download
Post by: Gavin on Saturday 28 November 2009, 06:58:24 am

Glad to be of help ;D

Title: Re: Blacklist Update Script - Download
Post by: ragr91279 on Saturday 28 November 2009, 07:03:48 am

One last question, how often should I perform this update on my server

Title: Re: Blacklist Update Script - Download
Post by: Gavin on Saturday 28 November 2009, 07:11:46 am

Up to you, I usually do it every monday morning at our office. You could safely do it once a month though.

Title: Re: Blacklist Update Script - Download
Post by: ragr91279 on Saturday 28 November 2009, 07:13:52 am

Ok, again thanks for your help ;D ;D ;D

Title: Re: Blacklist Update Script - Download
Post by: randle1111 on Wednesday 06 January 2010, 03:01:30 am

Thank you very much for this script! I am using EFW 2.3 and had the same issues as ragr91279. Running the script as:

python ./blacklist_updater.py

worked!

Some research turned up why a python was needed in front of the script. I clicked on the original link and then copied the text into notepad (windows). The dos format puts cr and lf (^M) into the file which causes the program to not run correctly. I went back and right clicked on the link and selected save as, saved the file (did not open in notepad), transferred the file and then

./blacklist_updater.py

worked!!

I trying to run this script to run automatically. I tried to put the script in the /var/cron.minutely (will move it to cron.weekly after testing). It looks like it is working because /var/log/messages is not showing the script exiting error I was getting before (see below)

Job [ -x /bin/run-parts ] && run-parts --report /etc/cron.minutely terminated (exit status: 1)

Oh yea based on the following kb I dropped the .py from the name and put the permissions to 0700:

http://kb.endian.com/entry/43/

If I was dreaming I would ask for help putting the code below into your python script. The script adds a date updated to your file and then is read by the html to be displayed on the content filter page.

http://efwsupport.com/index.php?topic=4.15

I was able to do the following:

Quote

In /home/httpd/cgi-bin/dansguardian.cgi find:

openbox('100%', 'left', _('URL Blacklist: Filter pages known to have content of the following categories.'), "blacklist");

Replace with this:
$output = `cat /etc/dansguardian/blacklists/updated`;
$output = &cleanhtml($output,"y");
openbox('100%', 'left', _("URL Blacklist: Filter pages known to have content of the following categories. Update Date: $output"), "blacklist");

Ended up doing the following in dansguardian.cgi, which worked correctly. I moved the $output lines up a few more lines higher in the file than they recommended, as the output statement did would not work around the get_folding.

Code:


    get_category("phrase_$cat", _(ucfirst($catname)), $allchecked, \%subcategories, \%conf_phraselist);
    $count++;
$output = `cat  /etc/dansguardian/blacklists/updated`;
$output = &cleanhtml($output,"y");
}
printf <<EOF
    </tr></table></td></tr></table>
%s
%s
EOF
,
get_folding(),
get_folding("BLACKLIST", "start", _("Filter pages known to have content of the following categories. (URL Blacklist) Update Date: $output")),
;

I am not good with scripting and the problem I was having was I can not figure out how to add the following to the python script (don't know if you can mix python and sh script). Perhaps the sh script could be converted to python script:

code that needs to be converted:

Code:

#!/bin/sh
date > /etc/dansguardian/blacklists/updated
exit 0

So to be clear, I have been able to get the date updated to display correctly by modifying the dansguardian.cgi file. I can execute the

Code:

date > /etc/dansguardian/blacklists/updated

at the # prompt (using putty). And it displays the date properly in the EFW gui. But I can not get date >... code to work correctly in the python script so that the date is updated when the script runs successfully.

Please help! Thanks

p.s.
I am a newbie in regards to scripting and linux so if I am way off base please forgive me in advance.

Title: Re: Blacklist Update Script - Download
Post by: satish on Sunday 10 January 2010, 02:36:17 am

Alrite ever since i updated, i can no longer white list sites.
As a matter of fact i cant even visit this topic on the site, as the logs say its a porn site.
I tried changing that a/w as mentioned in 1 of the posts. but that hasnt helped.

Anyone care to give a heads up here? (Btw im using 2.3 community)

Title: Re: Blacklist Update Script - Download
Post by: jester95 on Wednesday 20 January 2010, 06:39:25 am

Quote from: Gavin on Friday 13 November 2009, 02:43:29 am

The URL for this script no longer exists. Anybody have a copy or an updated URL?

Title: Re: Blacklist Update Script - Download
Post by: Gavin on Friday 22 January 2010, 03:22:39 am

Hey, I've updated the link. It should work now.

;)

Title: Re: Blacklist Update Script - Download
Post by: etan15 on Monday 25 January 2010, 09:51:50 am

Hi,

I am new to Endian firewalls and was interested in this script but I am getting this error:

root@Endian:/tmp # python ./blacklist_updater.py
Blacklist Updater for EFW 2.3
Written by Gavin Vickery <gdvickery@gmail.com>
==============================
Stage 1 of 5 | Downloading blacklist, please wait...
Traceback (most recent call last):
File "./blacklist_updater.py", line 18, in ?
source = urllib2.urlopen(sourceURL)
File "/usr/lib/python2.4/urllib2.py", line 130, in urlopen
File "/usr/lib/python2.4/urllib2.py", line 358, in open
File "/usr/lib/python2.4/urllib2.py", line 376, in _open
File "/usr/lib/python2.4/urllib2.py", line 337, in _call_chain
File "/usr/lib/python2.4/urllib2.py", line 1021, in http_open
File "/usr/lib/python2.4/urllib2.py", line 996, in do_open
urllib2.URLError: <urlopen error (101, 'Network is unreachable')>

Any help is much appreciated!

Thanks in advance...

Title: Re: Blacklist Update Script - Download
Post by: Gavin on Tuesday 26 January 2010, 03:20:46 am

Looks like the firewall cant establish an external connection. Are you sure your network connections are configured correctly? Can you ping external servers?

Title: Re: Blacklist Update Script - Download
Post by: etan15 on Tuesday 26 January 2010, 07:54:40 am

I am a little confused... I have no problem getting out to the internet with the box in line so how would I know if the box was not able to get an external connection? Here is the the routing table config...

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
172.16.2.0 0.0.0.0 255.255.255.0 U 0 0 0 br0
172.16.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1

Do I have to setup a route manually?

Thanks,

Title: Re: Blacklist Update Script - Download
Post by: etan15 on Tuesday 26 January 2010, 07:58:56 am

Here are the listed interfaces...

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:10:f3:03:bd:4a brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:10:f3:03:bd:4b brd ff:ff:ff:ff:ff:ff
inet 172.16.1.2/24 brd 172.16.1.255 scope global eth1
4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:10:f3:03:bd:4c brd ff:ff:ff:ff:ff:ff
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
link/ether 00:10:f3:03:bd:4a brd ff:ff:ff:ff:ff:ff
inet 172.16.2.1/24 brd 172.16.2.255 scope global br0

Not very savvy with Linux yet so I am not sure if this is correct...

Title: Re: Blacklist Update Script - Download
Post by: bithead999 on Friday 14 May 2010, 05:21:46 pm

I used the script and it worked great!! My problem is that I put it in the /tmp directory and now it is gone. There must be a cron script that cleans up /tmp. Can someone please send me a copy or post a copy of the script!!??!!

Thanks you soooo much.

Bit

Title: Re: Blacklist Update Script - Download
Post by: rroginela on Thursday 19 August 2010, 03:06:54 am

Hi All,

Anybody have a working link for the script??

Any help is appreciated.

Thank you,
Rafal

Title: Re: Blacklist Update Script - Download
Post by: sasindu on Friday 03 September 2010, 03:43:41 pm

Quote from: Gavin on Friday 13 November 2009, 02:43:29 am

link is dead

EFW Support

Support => EFW SMTP, HTTP, SIP, FTP Proxy Support => Topic started by: Gavin on Friday 13 November 2009, 02:43:29 am