Piwik : Open source web analytics

Piwik is an open source web analytics. It’s free and great. If you are familier with google web analystics, this piwik is the same. You can have your own web analytics. It’s easy to install and use mysql database in background to store the data. Webbased installation makes the whole installation process very easy.

Piwik main site : http://piwik.org/

Download link : http://piwik.org/latest.zip

Demo : http://piwik.org/demo/

Delete Postfix Queue based on domain name or email pattern

I found this particluar perl file very effective. This scripts give me the option to delete mail from the queue based on domain name or email id pattern. I have found this script in www.cyberciti.biz

#!/usr/bin/perl

$REGEXP = shift || die "no email-adress given (regexp-style, e.g. bl.*\@yahoo.com)!";

@data = qx</usr/sbin/postqueue -p>;
for (@data) {
  if (/^(\w+)(\*|\!)?\s/) {
     $queue_id = $1;
  }
  if($queue_id) {
    if (/$REGEXP/i) {
      $Q{$queue_id} = 1;
      $queue_id = "";
    }
  }
}

#open(POSTSUPER,"|cat") || die "couldn't open postsuper" ;
open(POSTSUPER,"|postsuper -d -") || die "couldn't open postsuper" ;

foreach (keys %Q) {
  print POSTSUPER "$_\n";
};
close(POSTSUPER);

Save this scripts and change the chmod to 755 (make the scripts executable). Now you can run the following command

./script.pl domain.com which will delete the domain.com mail from the mailq.
./script.pl xyz which will delete all queued messages that contain the word "xyz" in the e-mail address.

Restore MySQL data from crashed server

This is good to know where the MySQL data are stored. People usually backup there MySQL database when the server is up using mysqldump syntex. But what happen when server crash or server is show kernel panic? It’s easy. Boot the Server using any live distro (Knoppix, Ubuntu or Linux/CentOS in rescue mode). Mount your HDD. MySQL data’s are stored in /var/lib/mysql. So after mouting your HDD the location will be /your_mount_point/var/lib/mysql. Copy the files in your new server. Check the ownership. It should be mysql:mysql. Start the mysqld service. You are done.

Moving to MAC

I got a new MacBook. It is stunning with following configuration :

Processor : 2 GHz Intel Core 2 Duo
Memory : 2 GB
No of Processor : 1
HD : 150 GB
Graphics : NVIDIA GeForce 9400M
Display : 13.3 inch
Networking : WiFi, Ethernet, Bluetooth

You can get the details from http://store.apple.com/us/browse/home/shop_mac/family/macbook

BTCL Block YouTube??!!##@!

Can’t access YouTube. Seems BTCL filter it out . Here is traceroute result

[root@mrtg ~]# traceroute www.youtube.com

traceroute to www.youtube.com (208.65.153.238), 30 hops max, 40 byte packets

1  202.4.96.1 (202.4.96.1)  1.151 ms  1.768 ms  2.034 ms
2  core-router.dhakacom.com (202.4.100.5)  2.564 ms  2.618 ms  2.673 ms
3  203.112.207.253 (203.112.207.253)  3.814 ms  3.873 ms  3.933 ms
4  * * *
5  * * *
6  * * *
7  * * *

I think this is the first time BTCL filtering out some thing Three cheers to BTCL

Pop Before SMTP & Postfix

Though POP Before SMTP is not a secured way, but it is an easiest way to configure mail server to allow roaming user to use the mailserver  as SMTP gateway. Before configuring POP Before SMTP make sure that your Mail Server & POP3/IMAP sever working properly. Following Perl Module are required for the POP Before SMTP :

Time:: HiRes
File:: Tail
Date:: Parse
Net:: Netmask
BerkeleyDB

Download the POP Before SMTP perl files from http://popbsmtp.sourceforge.net/. Untar the files & copy them to the following locaiton

% cp pop-before-smtp.init /etc/rc.d/init.d/pop-before-smtp
% cp pop-before-smtp /usr/sbin/
% cp pop-before-smtp-conf.pl /etc

Edit the pop-before-smtp-conf.pl file to customize it for your system.

$ logto = ‘/ var / log / pop-before-smtp’;
$ dbfile = ‘/ etc / postfix / pop-before-smtp’;
$ grace = 10 * 60; # the time to log 10 minutes and $ file_tail ( ‘name’) = ‘/ var / log / dovecot’;

Uncomment the following lines which you will get after “# For Dovecot POP3/IMAP when it does its own logging.”

$pat = ‘^dovecot: [LOGTIME] Info: (?:imap|pop3)-login: ‘ .
‘Login: .+? rip=[:f]*(\d+\.\d+\.\d+\.\d+),’;
$out_pat = ‘^dovecot: [LOGTIME] Info: (?:imap|pop3)-login: ‘ .
‘Disconnected.*? rip=[:f]*(\d+\.\d+\.\d+\.\d+),’;

Next comment the following two lines :

= cut #====================== Postfix BerkeleyDB ======================= START =
= cut #====================== Postfix BerkeleyDB ======================= == END =

Now edit /etc/dovecot.conf file :

protocols = imap imaps pop3 pop3s
imap_listen = *
pop3_listen = *
log_path = / var / log / dovecot
info_log_path = / var / log / dovecot
log_timestamp = “% b% d% H:% M:% S”

Finally make the following changes to the /etc/postfix/main.cf file

smtpd_recipient_restrictions = permit_mynetworks,reject_non_fqdn_recipient, check_client_access hash:/etc/postfix/pop-before-smtp, reject_unauth_destination

% touch /etc/postfix/pop-before-smtp
% postmap /etc/postfix/pop-before-smtp
% /etc/init.d/pop-before-smtp start
% /etc/init.d/postfix reload

Now configure you outlook/eudora. Do POP before sending any mail. You will get the logged IP in /var/log/pop-before-smtp file.

If you’re having trouble getting pop-before-smtp to work; you will get more in http://popbsmtp.sourceforge.net/debug.shtml

Empty Mail Trash Folder

User always forget to empty there trash folder. User delete mail from there inbox; but don’t delete those mail from Trash folder. At the end, it eats up a lot of your server disk space. Empty this Trash folder is big headache for the administrator. Shell script can easily rescue the administrator. First create a list for you user. You can create it easily from awk commad. Save it in a file named user. Now create a new bassh file named trash_empty.sh and chown it to 755. Add the following syntex to the trash_empty.sh file

#!/bin/sh
for i in `more user`
do
echo $i
cat /dev/null > $i/mail/Trash
done

[user Trash folder in /home/username/mail/Trash. It may varies depending on your mail server configuration. ]

Now run the file trash_empty.sh WOW all your users Trash folder will be empty

Please take your own responsibility to empty users trash folder

Windows Blue Screen

The “Great Blue Screen” of Windows in Zia International Airport Terminal.The “Great Blue Screen” of Windows in Zia International Airport Terminal.The “Great Blue Screen” of Windows in Zia International Airport Terminal.

Transparent Squid Proxy Server with WCCP support

For Squid Proxy with WCCP support we need to recompile the kernel. We need to enable firewall & IP forward option. We use FreeBSD 7.0 as server.

[root@proxy ~]# cd /usr/src/sys/i386/conf
[root@proxy ~]# cp GENERIC to DCOM
[root@proxy ~]# vi DCOM

cpu I486_CPU
cpu I586_CPU
cpu I686_CPU
#ident GENERIC Commented on 29/06/08
ident DCOM #Change the ident name

# To statically compile in device wiring instead of /boot/device.hints#hints “GENERIC.hints”
# Default places to look for devices.
makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols
options IPFIREWALL
options IPFIREWALL_DEFAULT_TO_ACCEPT
options IPFIREWALL_VERBOSE
options IPFIREWALL_FORWARD

[root@proxy ~]# /usr/sbin/config DCOM
[root@proxy ~]# cd ../compile/DCOM
[root@proxy ~]# make depend
[root@proxy ~]# make
[root@proxy ~]# make install

Reboot the server.

Download the latest stable version of Squid.

[root@proxy ~]# wget http://www.squid-cache.org/Versions/v3/3.0/squid-3.0.STABLE10.tar.gz
[root@proxy ~]# tar –zxvf squid-3.0.STABLE10.tar.gz
[root@proxy ~]# cd squid-3.0.STABLE10
[root@proxy ~]# ./configure –prefix=/usr/local –enable-async-io –enable-icmp –enable-useragent-log –enable-snmp –enable-cache-digests –enable-follow-x-forwarded-for –enable-storeio=”aufs” –enable-removal-policies=”heap,lru” –with-maxfd=16384 –enable-poll –disable-ident-lookups –enable-truncate –exec-prefix=/usr –bindir=/usr/local/sbin –sbindir=/usr/local/sbin –datadir=/usr/local/etc/squid –libexecdir=/usr/local/libexec/squid –enable-external-acl-helpers=ip_user session unix_group wbinfo_group –enable-default-err-language=English –mandir=/usr/local/man –enable-ipfw-transparent ‘i386-portbld-freebsd7.0′ ‘build_alias=i386-portbld-freebsd7.0′ ‘host_alias=i386-portbld-freebsd7.0′ ‘target_alias=i386-portbld-freebsd7.0′ ‘CC=cc’ ‘CFLAGS= -DNUMTHREADS=150 -O2 -fno-strict-aliasing -pipe ‘ ‘LDFLAGS=’ ‘CPPFLAGS=’
[root@proxy ~]# make
[root@proxy ~]# make install

Next we need to do some fine tune to our firewall and kernel parameters.

[root@proxy ~]# echo if_gre_load=”YES” > /boot/loader.conf

Add the following lines to the /etc/rc.local

/sbin/ifconfig gre0 plumb
/sbin/ifconfig gre0 link2
/sbin/ifconfig gre0 tunnel 172.16.1.2 172.16.1.1
/sbin/ifconfig gre0 inet 192.192.192.1 192.192.192.2
/sbin/sysctl -w net.inet.icmp.icmplim=50
/sbin/sysctl -w net.inet.tcp.msl=3000
/sbin/sysctl -w kern. =65536
/sbin/sysctl -w kern.maxfiles=32768
/sbin/sysctl -w kern.ipc.somaxconn=8192
/sbin/sysctl -w net.inet.tcp.sendspace=65535
/sbin/sysctl -w net.inet.tcp.recvspace=65535
/sbin/sysctl -w net.inet.udp.recvspace=65535
/sbin/sysctl -w net.local.stream.recvspace=65535
/sbin/sysctl -w net.local.stream.sendspace=65535
/sbin/sysctl -w kern.ipc.nmbclusters=32768/sbin/sysctl -w kern.ipc.maxsockets=32768

We need to take care of gre0 inet line. Here first IP is the Proxy Server IP and the second IP is the Cisco Router External IP. 192.192.192.1 and 192.192.192.2 are the fake IP used to establish the GRE tunnel.

[root@proxy ~]# echo net.inet.ip.forwarding=1 > /etc/sysctl.conf

Add the following lines to /etc/rc.conf

inetd_enable=”NO”
keymap=”us.iso”
linux_enable=”YES”
router_enable=”YES”
sendmail_enable=”YES”
sshd_enable=”YES”
squid_enable=”YES”
firewall_enable=”YES”
firewall_type=”open”
gateway_enable=”YES”
natd_enable=”YES”

Lastly, add the following lines to /etc/rc.firewall

#!/bin/sh
ipfw -f flush
ipfw add fwd 127.0.0.1,8080 tcp from any to any 80 recv gre0

Reboot the server.

In this point we are done with the installing the proxy and fine tune the firewall parameter. Next we configure the squid proxy server.

Usually squid installed in /usr/local/etc/squid.conf. Backup the original squid.conf file and use the following configuration.

#—–start of squid.conf file——-
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8
acl DCOM src 172.16.0.0/16 # your ip block which are allowed to use the proxy.

acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT

http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny to_localhost
http_access allow localhost
http_access allow DCOM
http_access deny all
icp_access deny all
htcp_access deny all

http_port 172.16.1.2:8080 transparent
http_port 127.0.0.1:8080 transparent

hierarchy_stoplist cgi-bin ?

cache_mem 100 MB
maximum_object_size_in_memory 50 KB
cache_replacement_policy heap LFUDA
cache_dir ufs /cache 18000 16 256

maximum_object_size 25600 KB
cache_swap_low 90
cache_swap_high 95

access_log /usr/local/var/logs/access.log squid
cache_log /usr/local/var/logs/cache.log
cache_store_log /usr/local/var/logs/store.log

logfile_rotate 10
pid_filename /usr/local/squid/squid.pid
buffered_logs on

refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern (cgi-bin|\?) 0 0% 0
refresh_pattern . 0 20% 4320

quick_abort_min 0 KB
quick_abort_max 0 KB

half_closed_clients off
cache_effective_user squid

wccp2_router 172.16.1.1 # Router IP
wccp2_forwarding_method 1
wccp2_return_method 1
wccp2_service standard 0

icp_port 3130log_icp_queries off
dns_nameservers 4.2.2.2
hosts_file /etc/hosts

memory_pools off
client_db off
coredump_dir /cache

#—-End of squid.conf file——-

As we are storing the cache in /cache we need to create the folder and change the permission.

[root@proxy ~]# mkdir /cache
[root@proxy ~]# chown squid:squid /cache
[root@proxy ~]# /usr/local/sbin/squid –k
[root@proxy ~]# /usr/local/sbin/squid

At this point our squid is installed. Next we configure the router to forward the HTTP traffic to the proxy server.

core-router# config t
core-router(config)# ip wccp web-cache redirect-list 110
core-router(config)# access-list 110 permit ip any any
core-router(config)# interface FastEthernet1/0
core-router(config-if)# ip wccp web-cache redirect out

Troubleshoot

To check whether we are receiving the http request from the router through GRE tunnel we can use the following syntax.

[root@proxy ~]# tcpdump -nn -i re1 | grep gre-proto
[root@proxy ~]# tcpdump -nn -i gre0 | more

Check the log from tail –f /usr/local/var/log/access.log

Youtube Hijack Saga

On February 24th, 2008, the Youtube routing has been hacked . Ya, that’s true. As Pakistan Government notify PTA (Pakistan Telecommunication Authority) to block Youtube access from Pakistan. And they announce Youtube IP Block. Defcon 16th conference describe the steps very nicely :

1. You Tube announces 5 prefixes : -A /19, /20, /22 and two /24s. The /22 is 208.65.152.0/22

2. Pakistan’s government decides to block You Tube.

3. Pakistan Telecom internally nails up a more specific route (208.65.153.0/24) out of You Tube’s /22 to nul0 (the routers discard interface)

4. Somehow redists from static –> bgp, then to PCCW

5. Upstream provider sends routes to everyone else..

6. Most of the net now goes to Pakistan for You Tube, gets nothing!

7. You Tube responds by announcing both the /24 and two more specific /25s, with partial sucess

8. PCCW turns off Pakistan Telecom peering two hours later

9. 3 to 5 minutes afterward, global bgp table is clean again.

Heheheh..that’s awesome. The details are in RIPE websites http://www.ripe.net/news/study-youtube-hijacking.html

So if your a transit ISP, please be careful. Please don’t be lazy to apply appropriate prefix list and as path filter.