Squid and Dansguardian on Debian Etch

This short HOWTO will explain how to set up a proxy server that does content filtering with Dansguardian. The content is also virus checked.

This short howto is based on http://ubuntuforums.org/showthread.php?t=320733

You can also have a look at the Jimma dokuwiki for further finetuing http://www.ju.edu.et/icthelpdesk/doku.php?id=services:proxy

Installation

Add to your sources.list:

deb http://backports.org/debian etch-backports main
deb http://volatile.debian.org/debian-volatile etch/volatile main contrib non-free 
apt-get update; apt-get install debian-backports-keyring; apt-get update; apt-get upgrade

This is because we need a newer version for Squid to let ACLs and Dansgaurdian work together. See Debian Bug #408155. We also need the latest clamav.

apt-get install dansguardian clamav-freshclam clamav
apt-get install squid-common/etch-backports squid/etch-backports

Then check with

apt-cache policy squid

to see if it is the latest version.

Installing the squid3 package is not OK. That package doesn't contain the patch to let ACLS and Dansguardian work together.

Configuration

Squid

Adjust so it uses:

http_port 3128
hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
cache deny QUERY
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
access_log /var/log/squid/access.log squid
hosts_file /etc/hosts
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern .               0       20%     4320
acl snmppublic snmp_community public
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443          # https
acl SSL_ports port 563          # snews
acl SSL_ports port 873          # rsync
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl Safe_ports port 631         # cups
acl Safe_ports port 873         # rsync
acl Safe_ports port 901         # SWAT
acl purge method PURGE
acl CONNECT method CONNECT
follow_x_forwarded_for allow localhost
http_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
acl our_networks src 134.58.126.0/23
http_access allow our_networks
http_access allow localhost
http_access deny all
http_reply_access allow all
icp_access allow all
cache_effective_group proxy
visible_hostname proxy.mu.edu.et
snmp_port 3401
snmp_access allow snmppublic localhost
snmp_access deny all
snmp_incoming_address 0.0.0.0
snmp_outgoing_address 255.255.255.255
coredump_dir /var/spool/squid

Make note that the following settings are correct, visible_hostname, acl our_networks.

Check if the following directives are set (uncomment them!):

acl_uses_indirect_client on
delay_pool_uses_indirect_client on
log_uses_indirect_client on

Dansguardian

In /etc/dansguardian/dansguardian.conf:

  1. Comment the line that starts with 'COMMENT'
  2. Edit config file:
    forwardedfor = on
  3. Configure the following directives: emaildomain, postmaster, emailserver, downloaddir (/var/tmp/dgvirus), clamdsocket (/var/run/clamav/clamd.ctl)
  4. mkdir /var/tmp/dgvirus; chown dansguardian:dansguardian /var/tmp/dgvirus 
  5. Restart dansguardian /etc/init.d/dansguardian restart
  6. Install dglog
    apt-get install dglog

    , Surf to http://host/cgi-bin/dglog.pl

ClamAV, Freshclam

See mail server documentation for the installation of Clamav (only that is needed to be installed)

apt-get install clamav clamav-freshclam

Advanced configuration

SNMP

apt-get install snmp

In the right places: /etc/squid/squid.conf

acl snmppublic snmp_community public
snmp_port 3401
snmp_access allow snmppublic localhost
snmp_access deny all
snmp_incoming_address 0.0.0.0
snmp_outgoing_address 255.255.255.255

Testing:

snmpwalk -m /usr/share/snmp/mibs/SQUID.txt -v2c -c public localhost:3401 .1.3.6.1.4.1.3495.1

Setting up MRTG:

apt-get install mrtg-contrib mrtg libnet-snmp-perl
cd /tmp
# Upstream source with errors:
#wget http://chrismiles.info/unix/mrtg/mrtg-squid.cfg
#vi mrtg-squid.cfg
# Rudy's version with less mistakes 
wget http://users.ugent.be/~rgevaert/mrtg-squid.cfg
mv mrtg-squid.cfg /etc/mrtg-squid.cfg
mkdir /var/www/mrtgsquid
# Ignore the output of the following command
mrtg /etc/mrtg-squid.cfg
indexmaker --section title /etc/mrtg-squid.cfg > /var/www/mrtgsquid/index.html
echo '*/5 *   * * *   root    if [ -x /usr/bin/mrtg ] && [ -r /etc/mrtg-squid.cfg ]; then env LANG=C /usr/bin/mrtg /etc/mrtg-squid.cfg >> /var/log/mrtg/mrtg.log 2>&1; fi' >> /etc/cron.d/mrtg

Other monitoring

http://www.squid-cache.org/~wessels/squid-rrd/

Some tweaking has to be done to let it work under Debian:

  • Put the files on the proxy server in /var/www/squidgraphs.
  • In the poll script, remove 'T' from the first line. Edit so:
    my $rrdtool = '/usr/bin/rrdtool';
  • In the 1day.cgi file, make the path to rrdcgi correct:
    #!/usr/bin/rrdcgi
  • Enable expires module in apache.
    a2enmod expires
  • Edit apache config:
    <Directory "/var/www/squidgraphs">
    AllowOverride Indexes
    Options ExecCGI
    AddHandler cgi-script .cgi
    </Directory>
  • Put in /etc/crontab
    */5 * * * * /var/www/squidgraphs/poll.pl localhost
  • chown www-data:www-data /var/www/squidgraphs

Delay Pools

Another useful squid feature is delay pools. Conceptually, delay pools are bandwidth limitations - pools of bandwidth that drain out as people browse the Web, and fill up at a rate you specify - this can be thought of as a leaky bucket that is continually being filled. This is useful when you have limited bandwidth.

Links

Theory

There are 3 classes of delay pools:

  • class 1 is a single aggregate bucket
  • class 2 is an aggregate bucket with an individual bucket for each host in the class C
  • class 3 is an aggregate bucket, with a network bucket (for each class B) and an individual bucket for each host.

To configure the amount of delay pools, and specify which pool is which class, use the following format.

delay_pools 2      # 2 delay pools
delay_class 1 2    # pool 1 is a class 2 pool
delay_class 2 3    # pool 2 is a class 3 pool

To specify which pool a client falls into, create ACLs, e.g.

acl pool_1_acl src 10.10.0.0/255.255.0.0
acl pool_2_acl src 10.11.0.0/255.255.0.0

which specifies the ip ranges for each pool, and use the following:

delay_access 1 allow pool_1_acl
delay_access 1 deny all
delay_access 2 allow pool_2_acl
delay_access 2 deny all

Setting the parameters for each pool is done by:

delay_parameters pool aggregate network individual

where aggregate is the parameter for the aggregate bucket, network for the network bucket, and individual for the individual bucket. Aggregate is only useful for classes 1, 2 and 3, network for classes 2 and 3, and individual for class 3.

Each of these parameters is specified as restore / maximum

  • restore being the bytes per second restored to the bucket, in other words, if someone has exceeded his allowed amount of bytes/s he will be able to continue to download but at the restore speed, and
  • maximum being the amount of bytes that can be in the bucket at any time.

So, if you set the parameter to 8000/10000, the user will be able to surf at 10KByte/s (if the website is that fast enough) but as soon as he has reached the 10KByte/s speed, he will be reduced to 8KByte/s.

It is important to remember that they are in bytes per second, not bits. To specify that a parameter is unlimited, use a -1. If you wish to limit any parameter in bits per second, divide this amount by 8, and use the value for both the restore and the maximum. For example, to restrict the entire proxy to 64kbps, use:

delay_parameters 1 8000/8000

It is also possible to specify how full the bucket starts:

delay_initial_bucket_level 50

where the value is the percentage full.

Practical

Example 1: limit individual use for each connecting IP
  • Say we have an uplink of 4Mbit/s. This is 512KByte/s == 512000Bytes/s
  • Let us say that we limit use the http traffic of the proxy to 400KByte/s maxium, leaving enough room for other traffic (e.g. our university website and mail server need to be accessible from the outside).
  • And we limit the individual use (= one IP address) to 20KB/s initially with a refill of 10KByte/s.
  • Squid config:
    delay_pools 1
    delay_class 1 3
    acl our_networks src 10.140.0.0/16 10.141.0.0/16
    delay_access 1 allow our_networks
    delay_access 1 deny all
    delay_parameters 1 400000/400000 -1/-1 10000/20000
Example 2: limit individual use, usage per subnet and give extra bandwidth to a certain subnet
delay_pools 3
delay_class 1 3
delay_class 2 3
delay_class 3 3
delay_parameters 1 50000/50000 -1/-1 15000/20000
delay_parameters 2 200000/200000 -1/-1 40000/40000
delay_parameters 2 200000/200000 -1/-1 40000/40000
acl myFriends src 157.193.71.114-157.193.71.115/32
acl faculty1 src 157.193.71.0/24
acl faculty2 src 157.193.44.0/24
delay_access 1 allow myFriends
delay_access 1 deny all
delay_access 2 allow faculty1
delay_access 2 deny all
delay_access 3 allow faculty2
delay_access 3 deny all

You can see the effects in the picture below:

Example 3: example 2 + slowing down certain domains

Here we will set up 4 delay pools. 3 are the same as in example 2. But here we will also slow down certain domains.

delay_pools 4
delay_class 1 3
delay_class 2 3
delay_class 3 3
delay_class 4 3
acl slow_sites dstdomain .yahoo.com .hotmail.com
acl myFriends src 157.193.71.114-157.193.71.115/32
acl faculty1 src 157.193.71.0/24
acl faculty2 src 157.193.44.0/24
delay_access 1 allow slow_sites
delay_access 1 deny all
delay_access 2 allow myFriends
delay_access 2 deny all
delay_access 3 allow faculty1
delay_access 3 deny all
delay_access 4 allow faculty2
delay_access 4 deny all
delay_parameters 1 20000/20000 -1/-1 10000/10000
delay_parameters 2 50000/50000 -1/-1 15000/20000
delay_parameters 3 200000/200000 -1/-1 40000/40000
delay_parameters 4 200000/200000 -1/-1 40000/40000

Important remarks:

  • The delay_access rules are parsed in numerical order. This is why we put the slow sites in the first delay pool.
  • The whole university can reach the slow sites at a speed of 20KB/s. And each indivudual user can use 10KB/s to visit the slow sites.

ACLs

Fine tuning

Refresh pattern

Info:

  • Source: http://www.mail-archive.com/debian-user@lists.debian.org/msg107677.html \\
    refresh_pattern ^ftp:           1440    20%     10080
    refresh_pattern ^gopher:        1440    0%      1440
    refresh_pattern .               0       20%     4320
    refresh_pattern http://*.windowsupdate.microsoft.com/ 0 80% 20160
    reload-into-ims
    refresh_pattern http://office.microsoft.com/ 0 80% 20160 reload-into-ims 
    refresh_pattern http://windowsupdate.microsoft.com/ 0 80% 20160
    reload-into-ims
    refresh_pattern http://wxpsp2.microsoft.com/ 0 80% 20160 reload-into-ims 
    refresh_pattern http://xpsp1.microsoft.com/ 0 80% 20160 reload-into-ims 
    refresh_pattern http://w2ksp4.microsoft.com/ 0 80% 20160 reload-into-ims 
    refresh_pattern http://download.microsoft.com/ 0 80% 20160 reload-into-ims
    
    refresh_pattern http://download.macromedia.com/ 0 80% 20160
    reload-into-ims
    refresh_pattern ftp://ftp.nai.com/ 0 80% 20160 reload-into-ims
    refresh_pattern http://ftp.software.ibm.com/ 0 80% 20160 reload-into-ims
  • Source http://www.squid-cache.org/mail-archive/squid-users/200503/0648.html \\
    refresh_pattern windowsupdate.com/.*\.(cab|exe) 4320 100% 43200 reload-into-ims
    refresh_pattern download.microsoft.com/.*\.(cab|exe) 4320 100% 43200
    reload-into-ims
    refresh_pattern ^http://.*\.cnn\.com 360 50% 4320 override-lastmod
    refresh_pattern ^http://news\.bbc\.co\.uk 360 50% 4320 override-lastmod
    refresh_pattern microsoft 1080 150% 10080 override-lastmod
    refresh_pattern msn\.com 4320 150% 10080 override-lastmod
    refresh_pattern ^http://.*\.doubleclick\.net 10080 300% 40320 override-lastmod
    refresh_pattern ^http://.*FIDO 360 1000% 480
    refresh_pattern \.r[0-9][0-0]$ 10080 150% 40320
    refresh_pattern ^http://.*\.gif$ 1440 50% 20160
    refresh_pattern ^http://.*\.asis$ 1440 50% 20160
    refresh_pattern -i \.pdf$ 10080 90% 43200
    refresh_pattern -i \.art$ 10080 150% 43200
    refresh_pattern -i \.avi$ 10080 150% 40320
    refresh_pattern -i \.mov$ 10080 150% 40320
    refresh_pattern -i \.wav$ 10080 150% 40320
    refresh_pattern -i \.mp3$ 10080 150% 40320
    refresh_pattern -i \.qtm$ 10080 150% 40320
    refresh_pattern -i \.mid$ 10080 150% 40320
    refresh_pattern -i \.viv$ 10080 150% 40320
    refresh_pattern -i \.mpg$ 10080 150% 40320
    refresh_pattern -i \.jpg$ 10080 150% 40320 reload-into-ims
    refresh_pattern -i \.rar$ 10080 150% 40320
    refresh_pattern -i \.ram$ 10080 150% 40320
    refresh_pattern -i \.gif$ 10080 300% 40320 reload-into-ims
    refresh_pattern -i \.txt$ 1440 100% 20160 reload-into-ims override-lastmod
    refresh_pattern -i \.zip$ 2880 200% 40320
    refresh_pattern -i \.arj$ 2880 200% 40320
    refresh_pattern -i \.exe$ 2880 200% 40320
    refresh_pattern -i \.tgz$ 10080 200% 40320
    refresh_pattern -i \.gz$ 10080 200% 40320
    refresh_pattern -i \.tgz$ 10080 200% 40320
    refresh_pattern -i \.tar$ 10080 200% 40320
    refresh_pattern -i \.Z$ 10080 200% 40320
    
    refresh_pattern ^ftp:// 1440 50% 10080
    refresh_pattern ^gopher:// 1440 10% 1440
    refresh_pattern . 0 20% 4320
    negative_ttl 1 minutes
    positive_dns_ttl 15 hours
    negative_dns_ttl 1 minutes
    half_closed_clients off

Interesting information

vlir/projects/squid.txt · Last modified: 2009/12/03 05:09 by amanuel
 
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 3.0 Unported
Recent changes RSS feed Donate Powered by PHP Valid XHTML 1.0 Valid CSS Run by Debian Driven by DokuWiki