Bug 30184 - SquidGuard - integrate usage of external blacklists
SquidGuard - integrate usage of external blacklists
Status: CLOSED FIXED
Product: UCS@school
Classification: Unclassified
Component: Proxy services
UCS@school 3.1
Other Linux
: P5 enhancement (vote)
: UCS@school 4.0 R2
Assigned To: Sönke Schwardt-Krummrich
Florian Best
:
: 31747 (view as bug list)
Depends on:
Blocks: 36026 38358 38436 38561
  Show dependency treegraph
 
Reported: 2013-01-28 10:08 CET by Tim Petersen
Modified: 2015-05-19 11:00 CEST (History)
5 users (show)

See Also:
What kind of report is it?: ---
What type of bug is this?: ---
Who will be affected by this bug?: ---
How will those affected feel about the bug?: ---
User Pain:
Enterprise Customer affected?:
School Customer affected?:
ISV affected?:
Waiting Support:
Flags outvoted (downgraded) after PO Review:
Ticket number:
Bug group (optional):
Max CVSS v3 score:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tim Petersen univentionstaff 2013-01-28 10:08:33 CET
Asked via ticket #2013012521001021:

It would be nice having the possiblity to integrate external blacklists for SquidGuard via UCR.
See <http://wiki.ubuntuusers.de/squidGuard#Blacklist-installieren> for reference.
Comment 1 Ingo Steuwer univentionstaff 2013-05-30 17:14:05 CEST
also asked for in 2013052721001519

often requested is the "shalla blacklist": http://www.shallalist.de/
Comment 2 Erik Damrose univentionstaff 2013-06-14 15:14:18 CEST
*** Bug 31747 has been marked as a duplicate of this bug. ***
Comment 3 Jan Christoph Ebersbach univentionstaff 2013-07-16 16:13:38 CEST
An implementation was created at Ticket #2013062621001224.
Comment 4 Kevin Dominik Korte univentionstaff 2013-08-28 10:10:53 CEST
requested on Ticket#: 2013082021002249 again
Comment 5 Ulf Friedel 2014-07-17 15:01:02 CEST
This feature is requested again in association with UCS@school 3.2 R2:
http://forum.univention.de/viewtopic.php?t=3364&p=11859#p11859
Comment 6 Sönke Schwardt-Krummrich univentionstaff 2015-03-26 17:17:38 CET
The squidguard configuration should be able to import/include external filter lists.

By default in UCS@school the proxy checks are performed in the following order:
1) if defined, evaluate only the *whitelist* for the current computerroom (if the 
   computer is member of that room)
2) if defined, evaluate the filter blacklist/whitelist with the highest priority 
   that is assigned to one of the user's groups
3) allow the access, if 1) and 2) do not apply

Usecase 1) is only a whitelist and everything else is blocked → it makes no sense to add the external filter list here.

In case of 2) a filter is assigned to one of the user's groups. In case this filter is a white list, the external filter is also useless. In case of a blacklist, the external filter list should be optionally processed. It should be configurable via UCR if the external filter list has to be processed in this case (one global variable for all groups).

If a external list is configured, it should be always processed in case 3).

The external filter list is usually provided as raw text files. Split into 2 files - domains/ip addresses only and URLs only. squidguard should be able to include both. A UCR variable should define the text file names. A special update command should be implemented that converts the text file into a squidgard database file.

Please test with the mentioned shalla list and check if there are performance issues with 4 mio entries.
Comment 7 Sönke Schwardt-Krummrich univentionstaff 2015-04-24 14:21:12 CEST
External blacklists may now be placed below /var/lib/ucs-school-webproxy/ either directly or in a subdirectory. 2 new UCR variables have been introduced:
1) proxy/filter/global/blacklists/domains
2) proxy/filter/global/blacklists/urls

These variables should contain the filenames of either domainname blacklists or url blacklists. The filenames have to be relative to /var/lib/ucs-school-webproxy and have to separated by spaces, e.g.:

/var/lib/ucs-school-webproxy/extblacklist1/domains
/var/lib/ucs-school-webproxy/extblacklist1/urls
/var/lib/ucs-school-webproxy/bl2/list-domains
/var/lib/ucs-school-webproxy/bl2/list-urls
/var/lib/ucs-school-webproxy/bl3-dom
/var/lib/ucs-school-webproxy/bl3-urls

ucr set proxy/filter/global/blacklists/domains=\
            "extblacklist1/domains bl2/list-domains bl3-dom"
ucr set proxy/filter/global/blacklists/urls=\
            "extblacklist1/urls bl2/list-urls bl3-urls"

The UCR module ucs-school-webproxy.py concatenates all files of one type (domain/url) and call squidguard to convert it into a squidguard db file.

Please note: this is only done, if the UCR variables are set/have changed!

Depending on the size of the blacklist files, this may take several seconds!
In the test environment, 1.6 mio entries took about 12 seconds.

ucs-school-webproxy_11.0.3-2.81.201504241306_all.deb

xml changelog entry has been added
Comment 8 Florian Best univentionstaff 2015-04-27 16:58:16 CEST
This somehow does not work for me. I have no further internet rules set up. Tried with a teacher, student and Administrator.

# cd /var/lib/ucs-school-webproxy
# wget "$shallalist"
# tar xvzf shallalist.tar.gz
# ucr set proxy/filter/global/blacklists/domains="$(find BL -name domains | tr '\n' ' ')" \
          proxy/filter/global/blacklists/urls="$(find BL -name urls | tr '\n' ' ')"
# invoke-rc.d squid restart
# curl --proxy http://$USER:univention@10.200.27.10:3128/ "http://$(shuf -n 1 global-blacklist-domains)/" -i
# curl --proxy http://$USER:univention@10.200.27.10:3128/ "http://$(shuf -n 1 global-blacklist-urls)" -i

The blacklist database is successfully created and seems to be loaded:
2015-04-27 16:52:13 [27861] init domainlist /var/lib/ucs-school-webproxy//global-blacklist-domains
2015-04-27 16:52:13 [27861] INFO: loading dbfile /var/lib/ucs-school-webproxy//global-blacklist-domains.db
2015-04-27 16:52:13 [27861] init urllist /var/lib/ucs-school-webproxy//global-blacklist-urls
2015-04-27 16:52:13 [27861] INFO: loading dbfile /var/lib/ucs-school-webproxy//global-blacklist-urls.db
Comment 9 Sönke Schwardt-Krummrich univentionstaff 2015-04-29 11:13:53 CEST
(In reply to Florian Best from comment #8)
> This somehow does not work for me. I have no further internet rules set up.
> Tried with a teacher, student and Administrator.

I fixed a logic bug that has slipped in in a last minute change.
Package has been rebuilt.
Comment 10 Florian Best univentionstaff 2015-04-30 14:20:44 CEST
(In reply to Sönke Schwardt-Krummrich from comment #9)
> (In reply to Florian Best from comment #8)
> > This somehow does not work for me. I have no further internet rules set up.
> > Tried with a teacher, student and Administrator.
> 
> I fixed a logic bug that has slipped in in a last minute change.
> Package has been rebuilt.
OK, now it is fine!
I could not experience any performance problems.
The global blacklist overwrites every whitelist.

> The filenames have to be relative to /var/lib/ucs-school-webproxy
Well, technically this is not enforced.

Changelog: OK
Comment 11 Florian Best univentionstaff 2015-05-11 19:24:57 CEST
UCS@school 4.0 R2 v1 has been released:
http://docs.univention.de/release-notes-ucsschool-4.0R2v1-de.html

If this error occurs again, please use "Clone This Bug".