Starting mysqld and httpd on demand with xinetd

My father gave me an awesome present besides life :)
He gave me a Raspberry Pi B model, its basically a low power consumption, low spec computer board, at a very low price intended for tinkerers, and electronic education.

I am using it to host my own PBX which is based of this project: Asterisk for raspberry
I am also using it for other purposes such as monitoring my home network as an active IDS, a dns server, a proxy, a vpn server, basically I want to use it for all the things you always wanted to have on a always-on server inside your house.

The problem I encountered was that the Pi model B comes with only 512 mb of RAM, and the Freepbx software uses Apache PHP5 and Mysql which consumes a good amount of RAM.
First step was easy, I just installed lighttpd and stopped apache. Second step I started looking into changing Freepbx to a SQLite engine, to get rid of MySQL, but Freepbx support for it was not 100%.

So after looking for a while I found a very nice solution for all my troubles, using xinetd and netcat (awesome tools) I start services on demand, when a connection comes in I start the required service and leave a background process to check on if the service is still required, if the port sees no further activity the service gets shutdown. So I can now have all the services that I want on the resource limited device.

I struggled to find how to do this. So I am sharing how I did it. I am also basing myself of this great post which does something similar but with another purpose in mind. Automatic Tunnels with xinetd and netcat.

Tools required:

  • xinetd
  • netcat
  • service httpd, mysqld
  • bash proxy script

This is my startup script I created here /etc/xinetd.init.d/lighttpd

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
#!/bin/bash
# inspired on this article
# https://www.linuxnet.ch/automatic-tunnels-with-xinetd-and-netcat/
# script is called from xinetd
# script starts and stops service using /etc/init.d scripts
 
SERVICE=/etc/init.d/lighttpd  # script used to start/stop service
PORT=81 # port where end service should listen
PID_FILE=/var/run/lighttpd.pid # pid file generated by init script
REAPER_PID_FILE="/var/run/lighttpd_reaper.pid" # pid file generated by this script
REAPER_SLEEP=180 # The reaper sleeps in seconds and checks for idle conns
LOG=/var/log/syslog # where to log messages
 
# this function checks if we already have reaper
check_reaper(){
        if  [ -s $REAPER_PID_FILE ]
        then
                reaper_pid=`cat $REAPER_PID_FILE 2>/dev/null`
                ps -p $reaper_pid &> /dev/null
                if [ $? -ne 0 ]
                then
                        start_reaper &    # If we dont have a reaper we start one
                        echo $! > $REAPER_PID_FILE
                fi
        else
                start_reaper &
                echo $! > $REAPER_PID_FILE
        fi
 
}
 
# this function starts a reaper, which is a background process that will kill the end service if its inactive
start_reaper(){
        while [ -f $PID_FILE ]
        do
                sleep $REAPER_SLEEP                                      # We wait
                touched=`stat --printf %W $REAPER_PID_FILE 2>/dev/null`  # We check when the reaper PID was last touched
                now=`date +%s 2>/dev/null`
                let idle=$now-$touched
                if [ $idle -gt $REAPER_SLEEP ]              # If reaper pid has not been touched in more than a sleep cycle we stop the service
                then
                        echo `date`" REAPER STOPPING SERVICE AFTER BEING $idle" >> $LOG   
                        $SERVICE stop >> $LOG               # This is the stop service instruction
                        rm $REAPER_PID_FILE 
                        exit 0
                fi
        done
}
 
# This is where we start our service
start_service(){
        sleep 1                    # Added a delay to trouble shoot as browsers kickstart several connections, we need to allow the PID file to be created this can be improved.
        if [ -s $PID_FILE ]        # We check if the PID file for the end service exist to avoid calling the start script when the service has already been started
        then
                return
        else
                echo `date`" STARTING $SERVICE" >> $LOG
                $SERVICE start &>> $LOG                  #this is the start service instruction
                return
        fi
}
 
# We probe and wait for the service to come on line
wait_for_service(){
        nc -w30 -z 127.0.0.1 $PORT &>/dev/null          # probe end port with a timeout of 30 seconds
        if [[ $? -ne 0 ]]
        then
                echo `date`" XINET SERVICE START ON $PORT TIMED OUT" >> $LOG
        fi
}
 
# This is were all the magic happens netcat passes traffic back and forth
transmit(){
        nc -w30 127.0.0.1 $PORT 2>/dev/null  # netcat is awesome, timeout flag of 30 seconds can be adjusted
}
 
# this is the main program that is called every time
main()
{
        nc -z 127.0.0.1 $PORT &>/dev/null  # We probe the end service
        if [[ $? -ne 0 ]]                  # If its not responding
        then
                start_service              # We start service
                wait_for_service           # We wait for service to became online
        fi
        check_reaper                       # We always check we have a reaper
        touch $REAPER_PID_FILE             # We log activity by touching the reaper PID file
        transmit                           # We transmit data 
        exit 0                             
}
 
main

This is the configuration for xinet.d

1
2
3
4
5
6
7
8
9
10
service http
{
        disable         = no
        socket_type     = stream
        protocol        = tcp
        port              = 80
        server = /etc/xinetd.init.d/lighttpd
        user            = root
        wait            = no
}

For mysqld I did something similar but I had to add a wait time as mysql takes longer to boot
mysql startup script I placed it in /etc/xinetd.init.d/mysqld
Another thing I encountered was that mysql clients tend to default to use the socket file, so I had to reconfigure so it would listen on the eth0 address, as a precaution I added a rule to only allow the same IP to access it, I can probably improve this too but I got it working like this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
#!/bin/bash
# inspired on this article
# https://www.linuxnet.ch/automatic-tunnels-with-xinetd-and-netcat/
 
SERVICE=/etc/init.d/mysql
PORT=3307
PID_FILE=/var/run/mysqld/mysqld.pid
REAPER_PID_FILE=/var/run/mysqld/mysqld_reaper.pid
REAPER_SLEEP=300
LOG=/var/log/syslog
IP='192.168.1.50'
 
check_reaper(){
        if  [ -s $REAPER_PID_FILE ]
        then
                reaper_pid=`cat $REAPER_PID_FILE 2>/dev/null`
                ps -p $reaper_pid &> /dev/null
                if [ $? -ne 0 ]
                then
                        start_reaper &
                        echo $! > $REAPER_PID_FILE
                fi
        else
                start_reaper &
                echo $! > $REAPER_PID_FILE
        fi
 
}
 
start_reaper(){
        while [ -f $PID_FILE ]
        do
                sleep $REAPER_SLEEP
                touched=`stat --printf %W $REAPER_PID_FILE 2>/dev/null`
                now=`date +%s 2>/dev/null`
                let idle=$now-$touched
                if [ $idle -gt $REAPER_SLEEP ]
                then
                        echo `date`" REAPER STOPPING SERVICE AFTER BEING $idle" >> $LOG
                        $SERVICE stop >> $LOG
                        rm $REAPER_PID_FILE
                        exit 0
                fi
        done
}
 
start_service(){
        if [ -s $PID_FILE ]
        then
                return
        else
                echo `date`" STARTING $SERVICE" >> $LOG
                $SERVICE start &>> $LOG
                return
        fi
}
 
wait_for_service(){
        sleep 10
        nc -w30 -z $IP $PORT &>/dev/null
        if [[ $? -ne 0 ]]
        then
                echo `date`" XINET SERVICE START ON $PORT TIMED OUT" >> $LOG
        fi
}
 
transmit(){
        nc -w30 $IP $PORT 2>/dev/null  # netcat is awesome
 
}
 
main()
{
        nc -z $IP $PORT &>/dev/null
        if [[ $? -ne 0 ]]
        then
                start_service &
                wait_for_service
        fi
        check_reaper &>/dev/null
        touch $REAPER_PID_FILE &>/dev/null
        transmit
        exit 0
}
 
main

And its xinet config file looks like this:
/etc/xinetd.d/mysqld

1
2
3
4
5
6
7
8
9
10
11
12
service mysql
{
        disable         = no
        type            = UNLISTED
        socket_type     = stream
        port              = 3306
        server = /etc/xinetd.init.d/mysqld
        user            = root
        wait            = no
        bind            = 192.168.1.50
        only_from       = 192.168.1.50
}

 

 

Telmex troncal SIP con Asterisk11

Contrate una línea Telmex y me instalaron un gateway Thomson TG712, la línea telefónica que sale del teléfono ya no se conecta a la pared, se conecta a un puerto FXS en el módem.
Al parecer ya no instalan teléfonos analógicos, y de hecho Telmex empezó a ofrecer paquetes de solo Internet.

Lo primero que hice fue obtener los datos de la cuenta voip, esto lo hice en la pagina de configuración de voz del módem.

https://192.168.1.254/cgi/b/_voip_/cfg/?be=0&l0=4&l1=2

Le di editar y obtuve el Usuario, Contraseña y SIP URI (En el caso de la contraseña se puede obtener del codigo fuente de la pagina, o con un plugin para mostrar passwords en el navegador)

555XXXXXXX@lpi.telmex.com   y una contraseña numérica 12345678901234

Buscando en internet lpi.telmex.com encontré este proxy 201.116.100.193

Probé los datos con un cliente zoipper y en un instante se conecto pude hacer y recibir llamadas de mi línea Telmex.

Configure una troncal en mi asterisk de la siguiente manera:

[555XXXXXXX@lpi.telmex.com]
type=peer
insecure=invite,register
host=lpi.telmex.com
outboundproxy=201.116.100.193
user=555XXXXXXX@lpi.telmex.com
secret=12345678901234
authdomain=lpi.telmex.com
fullcontact=555XXXXXXX@lpi.telmex.com
fromuser=555XXXXXXX
fromdomain=lpi.telmex.com
context=from-trunk-sip-lpi.telmex.com

[lpi.telmex.com]
type=peer
host=lpi.telmex.com
outboundproxy=201.116.100.193
user=4496880517@lpi.telmex.com
secret=122076218223
authdomain=lpi.telmex.com
fromuser=4496880517
fromdomain=lpi.telmex.com
fullcontact=4496880517@lpi.telmex.com
context=from-trunk-sip-lpi.telmex.com

register=4496880517:122076218223@lpi.telmex.com/4496880517

No tengo mucha experiencia en conmutadores y trate de configurarlo utilizando las instrucciones pero por más que intente no se conectaba.

Capture y compare los paquetes de el zoipper con lo que mandaba el asterisk encontré 3 diferencias en la parte de Authorization:

 

  1. El campo de username= zoiper y thomson mandan username=”555XXXXXXX@lpi.telmex.com”, Asterisk manda username=”555XXXXXXX” (sin el dominio, manda únicamente el numero
  2. El campo de uri= zoiper y thomson mandan uri=”sip:lpi.telmex.com”
  3. El campo de cnonce= zoiper y thomson mandan un hash cnonce=”fbe4225b8d4a9aXX8f320d12ad6aX72X”, Asterisk manda solo unos cuantos caracteres cnonce=”478ce5d6″

Autenticación digest como lo envía asterisk

12:17:48.691772 IP (tos 0×60, ttl 64, id 1157, offset 0, flags [none], proto UDP (17), length 678)
127.0.0.1.sip > lpi.telmex.com.sip: SIP, length: 650
REGISTER sip:lpi.telmex.com SIP/2.0
Via: SIP/2.0/UDP 127.0.0.1:5060;branch=z9hG4bK5c8ebb6c
Max-Forwards: 70
From: ;tag=as7b5837f5
To:
Call-ID: 36d16558757888ef679d138e42f56190@127.0.0.1
CSeq: 103 REGISTER
User-Agent: FPBX-2.11.0beta2(11.1.0)
Authorization: Digest username=”555XXXXXXX”, realm=”ims.telmex.com”, algorithm=MD5, uri=”sip:sip:ttcnserver@ims.telmex.com”, nonce=”27902ec82235618e07f9a134c9599455″, response=”d43dee94422d68b246c1433748ed07ce”, qop=auth, cnonce=”478ce5d6″, nc=00000001
Expires: 120
Contact:
Content-Length: 0

La parte que no cuadra es esta:

Authorization: Digest username=”555XXXXXXX”, realm=”ims.telmex.com”, algorithm=MD5, uri=”sip:sip:ttcnserver@ims.telmex.com”, nonce=”27902ec82235618e07f9a134c9599455″, response=”d43dee94422d68b246c1433748ed07ce”, qop=auth, cnonce=”478ce5d6″, nc=00000001

Revisando el modulo chan_sip.c note que asterisk no calculaba los hashes para cnonce y el usuario le quitaba la parte despues del @.

Asi que modifique el modulo, la única función que se necesita modificar es la de build_reply_digest Justo en la parte que tiene el comentario /* Calculate SIP digest response */

Ahí justo despues del comentario agregue el siguiente bloque de código para cuando la uri contenga Telmex cumpla con los requisitos del servidor:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
  if(telmex){
                char cnonce_hash[256];
                ast_md5_hash(cnonce_hash, cnonce);
                snprintf(a1, sizeof(a1), "%s@lpi.telmex.com:%s:%s", username, p->realm, secret);
                snprintf(a2, sizeof(a2), "%s:sip:lpi.telmex.com", sip_methods[method].text);
                if (!ast_strlen_zero(md5secret))
                        ast_copy_string(a1_hash, md5secret, sizeof(a1_hash));
                else
                        ast_md5_hash(a1_hash, a1);
                ast_md5_hash(a2_hash, a2);
 
                p->noncecount++;
                if (!ast_strlen_zero(p->qop))
                        snprintf(resp, sizeof(resp), "%s:%s:%08x:%s:%s:%s", a1_hash, p->nonce, p->noncecount, cnonce_hash, "auth", a2_hash);
                else
                        snprintf(resp, sizeof(resp), "%s:%s:%s", a1_hash, p->nonce, a2_hash);
                ast_md5_hash(resp_hash, resp);
 
                /* only include the opaque string if it's set */
                if (!ast_strlen_zero(p->opaque)) {
                        snprintf(opaque, sizeof(opaque), ", opaque="%s"", p->opaque);
                }
 
                /* XXX We hard code our qop to "auth" for now.  XXX */
                if (!ast_strlen_zero(p->qop))
                        snprintf(digest, digest_len, "Digest username="%s@lpi.telmex.com", realm="%s", algorithm=MD5, uri="sip:lpi.telmex.com", nonce="%s", response="%s"%s, qop=auth, cnonce="%s", nc=%08x", username, p->realm, p->nonce, resp_hash, opaque, cnonce_hash, p->noncecount);
                else
                                        snprintf(digest, digest_len, "Digest username="%s@lpi.telmex.com", realm="%s", algorithm=MD5, uri="sip:lpi.telmex.com", nonce="%s", response="%s"%s", username, p->realm, p->nonce, resp_hash, opaque);
 
                append_history(p, "AuthResp", "Auth response sent for %s in realm %s - nc %d", username, p->realm, p->noncecount);
 
                if (credentials) {
                        ao2_t_ref(credentials, -1, "Unref auth for digest");
                }
                return 0;
        }

Python vimrc

A nice tip if you work with python under vi, use this vimrc config file.

http://svn.python.org/projects/python/trunk/Misc/Vim/vimrc

you can just copy it to your home directory
wget http://svn.python.org/projects/python/trunk/Misc/Vim/vimrc ~/python_vimrc

you can start vi with vi -u ~/python_vimrc

Or make it your default .vimrc you will end with nice syntax highlighting

Nagios Acknowledge for the Masses

I made this simple perl script to help with the acknowledging of multiple alerts.

When running in a large environment, and during a large maintenance alerts can flood the user and even with the use aid of servicegroups and hostgroups the alerts can overwhelm the user.

The script lists any problem unacknowledged or without unscheduled downtime.
Similar to what this link does:

/cgi-bin/status.cgi?host=all&type=detail&servicestatustypes=29&hoststatustypes=15&serviceprops=10

To setup the script, make sure you edit the paths to your nagios status.dat, and the command FIFO file.
Script should be able to write to the FIFO file.

To use the script, run without arguments, in interactive mode.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
#!/usr/bin/perl
 
#####################################################################################################
#   This script provides help to acknowledge multiple services during a large maintenance
#   Sometimes host groups and service groups do not suffice
#      Script requires the setup of the location of the status.dat and the FIFO file
#      Script should be able to write to the FIFO file
#	Command is run interactively
#      Santiago Velasco - sanxiago.com
#########################################################################################
 
	my $command_file = "/usr/local/nagios/var/rw/nagios.cmd";
	my $status_file = "/usr/local/nagios/var/status.dat";
	my $time = time();
	my %state = (1 ,'WARNING', 2,'CRITICAL', 3,'UNKNOWN');
	my $user = $ARGV[0];
	my $msg = $ARGV[1];
	my $search_string = $ARGV[3];
 
print STDERR "nnACKNOWLEDGE AND SCHEDULE DOWNTIME FOR MULTIPLE SERVICESnn";
 
while(!defined($user) or $user =~ /;|[|]/  or length($user)<=1){
	print STDERR "Type in yout USER that acknowledges:n";
	$user = <>;
	$user =~ s/n//;
}
while (!defined($msg) or $msg =~ /;|[|]/ or length($msg)<=1 ){
	print STDERR "Type in the MESSAGE that will be used for all acknowledges:n";
	$msg = <>;
	$msg =~ s/n//;
}	
print STDERR "Type in a string that matches the service_description of the services you want to ack.n Leave it blank to list all alerts):n"; $search_string = <>; $search_string =~ s/n//; if(length($search_string)<=1){
	$search_string='.*';
}
 
if (-r $status_file){	
	open (STATUS, $status_file);
}
else {
	print STDERR "FAILED TO READ NAGIOS STATUS FILEn";
	exit 1;
}
while(<STATUS>){
	if($_ =~ /service {/){
	$is_service = 1;
	}
	if($_ =~ /}/ and $service_description=~/$search_string/){
	$is_service =0;
		if(defined($current_state) and $current_state and $acknowledged==0 and $scheduled_downtime==0 ){
		# Command Format:
		# [time] ACKNOWLEDGE_SVC_PROBLEM;<host_name>;<service_description>;<sticky>;<notify>;<persistent>;<author>;<comment>
		# [time] SCHEDULE_SVC_DOWNTIME;<host_name>;<service_desription><start_time>;<end_time>;<fixed>;<trigger_id>;<duration>;<author>;<comment>
 		undef($ack_true);
		print STDERR "n---------------------------------------------------------n";
		print STDERR "Acknowledge $service_description @ $host_name $current_state".$state{$current_state}."?n$plugin_outputn[y/n/s] (s followed by the number of minutes of scheduled downtime) (Enter to skip)n";
		$ack_true=<>;
		# if acknowledge yes
		if($ack_true=~/^y/){
			if (!(-w $command_file)){ print STDERR "FAILED TO OPEN FIFO FILE"; exit 1; }
			open (CMD, '>>'.$command_file);
			print CMD "[$time] ACKNOWLEDGE_SVC_PROBLEM;$host_name;$service_description;1;0;1;$user;$msgn";
			close (CMD);
		# if schedule downtime 
		}elsif($ack_true=~/^s(.*)/){
			my $duration = $1;
			if($duration=~/[^d]*([0-9]+).*/){
				#expect duration in minutes convert to seconds
				$duration=int($1)*60;
			}else{
				$duration=3600;
			}
                        my $end_time = $time + $duration;
 
                        if (!(-w $command_file)){ print STDERR "FAILED TO OPEN FIFO FILE"; exit 1; }
                        open (CMD, '>>'.$command_file);
			print CMD "[$time] SCHEDULE_SVC_DOWNTIME;$host_name;$service_description;$time;$end_time;1;0;$duration;$user;$msgn";
                        close (CMD);
		}
		}
	undef($current_state);
	undef($host_name);
	}
	if($is_service){
		if($_=~/host_name=(.*)/){
		$host_name=$1;
		}
		if($_=~/service_description=(.*)/){
		$service_description=$1;
		}
                if($_=~/current_state=([0-9]*)/){
                $current_state=$1;
                }
                if($_=~/problem_has_been_acknowledged=([0-9]*)/){
                $acknowledged=$1;
                }
                if($_=~/plugin_output=(.*)/){
                $plugin_output=$1;
                }
		if($_=~/scheduled_downtime_depth=([0-9]*)/){
		$scheduled_downtime=$1;
		}
	}
}
close(STATUS);

Python object dump

When working with objects and arrays there are times you need to debug a certain object or array and list all its contents.
Most languages provide a functionality to do so, in the case of Perl you have Data::Dumper and in PHP you have print_r

I was looking for something like that and found:

Recipe

I truly recommend it.
It can be easily added as an extra module.

Bank decreases security in attempt to increase password strength

I was asked to set my phone password by my bank, following these rules:
1. Password must have 7 digits
2. No digits can repeat in a password
3. Consecutive digits are not allowed

Some security expert thought the best way to protect the “stupid” users from choosing easy passwords.
Was to enforce rules 2 and 3.
Lets keep in mind that without the rules 2 and 3 we had 9’999’999 possible passwords.

Rule 2 means you must pick 7 numbers out of the 9 digits without repeating any digit.
Using simple math we have
nPr = n! / (n-r)!
Were n is 9 as there are 9 digits in a phone, as r is 7 as that is the digits we must pick out.
We have: 9! / 2 = 181,440
As a result we have only 181,440 Valid passwords, this rule alone reduces the hackers guessing effort in a 98% Nicely Done!

*Rule 3, sequences of numbers are not allowed, this is the cherry on top.
NCm – ( N – m + 1 )Cm
We have: 6435 – 84 = 6351
Thats 6351 passwords we are unable to use.
181,440 – 6,351 = 175,089

It reduces even further the possibilities, this alone is not a bad rule, but since someone reduced the set most users will not be able to choose a password they can relate, so they are confined in this set, so my guess is most users ended up choosing a password based on the phone layout, as the rules above are too restrictive.

Look at the common phone digits layout:
1 2 3
4 5 6
7 8 9
    0

This are my guesses on the most common passwords:
1-4-7 2-5-8-0
3-6-9 2-5-8-0
2-5-8-0 1-4-7
2-5-8-0 3-6-9

* http://www.albaiges.com/matematicas/combinatoria/combinacionesordenadas.htm

Python Logging Module

I started using Python’s logging module in my Python scripts.

Below is an example on how I created a python logger object.

First I created this constructor method to ease the logger object creation.

The create_logger function receives the following parameters:

  • logName
  • This is the only parameter that is mandatory and does not have a default value.

  • logLevel=’debug’
  • This is the logging level in lower case

  • logFile=’test.log’
  • This is the log path/file were log will be written

  • stream=0
  • 0 no STDOUT, 1 print to STDOUT

Hardcoded you will see the max size of the log before it rotates. Log rotation is done using logging.handlers.RotatingFileHandler if you do not want to use log rotation you would probably need to use logging.handlers.FileHandler instead.

#my_log.py
import logging
import logging.handlers
 
def create_logger(logName, logLevel='debug', logFile='test.log', stream=0):
	LOG_FILENAME=logFile
	# Files will rotate at 200MB
	MAX_BYTES = 209715200
	# Files versions to keep
	BACKUP_COUNT = 10
	LEVELS = {'debug': logging.DEBUG,
        	  'info': logging.INFO,
	          'warning': logging.WARNING,
	          'error': logging.ERROR,
	          'critical': logging.CRITICAL}
        logLevel = LEVELS.get(logLevel, logging.NOTSET)
 
	# Set up a specific logger with our desired output level
	my_logger = logging.getLogger(logName)
	my_logger.setLevel(logLevel)
        if(logFile != 0  and logFile != ''):
		# Add the log message handler to the logger
		handler = logging.handlers.RotatingFileHandler(LOG_FILENAME, maxBytes=MAX_BYTES, backupCount=BACKUP_COUNT)
	        # create formatter
	        formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
	        # add formatter to ch
	        handler.setFormatter(formatter)
		my_logger.addHandler(handler)
	# If required we print out stream, or if no logFile was specified
	if(stream != 0 or logFile == 0 or logFile == ''):
	        handler = logging.StreamHandler()
        	my_logger.addHandler(handler)
	return my_logger

This is a simple example on how the logging module is used:

#test_log.py
import sys
import my_log.py
# We create the logger, using the script name sys.arv[0] and that is the logName, can be an arbitrary string.
# Named arguments are optional, and all have default values if left undeclared
my_logger=cacti_util.create_logger(sys.argv[0],stream=1,logFile='testing.log', logLevel='debug')
 
# We log the program start
my_logger.info(sys.argv[0] + ' started')
 
i=0
try:
	while(1<2):
		i = i + 1
		my_logger.warn('i = %d' % i) # this is an example warning in the log
except:
    my_logger.exception(sys.exc_info()[0]) # logger.exception is used to print full details on python exceptions
    my_logger.info(sys.argv[0] + ' died')

This is an example of what would be written in the log.


2010-04-30 16:47:33,200 - ping.py - INFO - ping.py started
2010-04-30 16:47:33,201 - ping.py - WARNING - i = 1
2010-04-30 16:47:33,201 - ping.py - WARNING - i = 2
2010-04-30 16:47:33,201 - ping.py - WARNING - i = 3
# ... AFTER SEVERAL LINES LATER I PRESS CTRL + C TO BREAK THE LOOP
2010-04-30 16:47:33,254 - ping.py - WARNING - i = 407
2010-04-30 16:47:33,254 - ping.py - ERROR -
Traceback (most recent call last):
File "ping.py", line 14, in
my_logger.warn('i = %d' % i) # this is an example warning in the log
File "/usr/lib/python2.6/logging/__init__.py", line 1060, in warning
self._log(WARNING, msg, args, **kwargs)
File "/usr/lib/python2.6/logging/__init__.py", line 1165, in _log
self.handle(record)
File "/usr/lib/python2.6/logging/__init__.py", line 1175, in handle
self.callHandlers(record)
File "/usr/lib/python2.6/logging/__init__.py", line 1212, in callHandlers
hdlr.handle(record)
File "/usr/lib/python2.6/logging/__init__.py", line 673, in handle
self.emit(record)
File "/usr/lib/python2.6/logging/handlers.py", line 73, in emit
logging.FileHandler.emit(self, record)
File "/usr/lib/python2.6/logging/__init__.py", line 852, in emit
StreamHandler.emit(self, record)
File "/usr/lib/python2.6/logging/__init__.py", line 792, in emit
self.flush()
File "/usr/lib/python2.6/logging/__init__.py", line 754, in flush
self.stream.flush()
KeyboardInterrupt
2010-04-30 16:47:33,271 - ping.py - INFO - ping.py died

The logging module offers other features such as email logging.
Here is a nice simple example.

Mysql Innodb monitoring with nagios

I am building a full innodb nrpe plugin.

We ran into a problem, the status gets trunctated and we are unable to plot all the correct values.
A colleague pointed me to this link.
http://www.mysqlperformanceblog.com/2008/10/31/full-innodb-status/

Below is a small script to get the full innodb status file using the hack discussed in the link above.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
mysql_port=3306;
 
#Get PID of mysql_d running on $mysql_port
mysql_pid=`pgrep -f "mysqld.*--port=$mysql_port "`;
 
#If no PID was found, no mysqld instance is running on $mysql_port
if  `test -z $mysql_pid`
then
        echo "ERROR: No Mysqld instance running on port $mysql_port"
        exit 2;
fi
# We look for the full INNODB status file as described in:
# http://www.mysqlperformanceblog.com/2008/10/31/full-innodb-status/
for i in `ls -l /proc/$mysql_pid/fd | grep deleted | grep tmp | awk '{print $9}'`;
        do
        if [ ${#innodb_status_file} == 0 ]
        then
        innodb_status_file=`grep -l "INNODB MONITOR OUTPUT" /proc/$mysql_pid/fd/$i` ;
        else
        break;
        fi
done;
# If we did not find the file
if [ ${#innodb_status_file} == 0 ]
then
        echo "ERROR: unable to find INNODB MONITOR file"
        exit 2;
fi
 
cat $innodb_status_file;

Today I found this project, I will test this out.
http://www.xaprb.com/blog/2006/07/02/innotop-mysql-innodb-monitor/

Check IIS servers that require user authenticaction

We needed to monitor a couple of IIS servers that required user authentication.
We currently use nagios and cacti to monitor our servers.

I cooked this simple script, that provides a method to check a IIS webserver page that require NTLM Authentication.
Horse work is done entirely by curl, I tested 7.12.1 with libcurl/7.12.1

To test if your current curl binary does the trick call curl like this

curl -u $user:$pass --ntlm  --stderr /dev/null $uri  -i

Examine the results, you should first see a page with a 401 unauthorized response, then you should see the authorization being sent over to the server, If the user and pass are correct and curl ntlm worked then you should see the end page with status code 200 OK or 302 Page Moved if its a redirect.

The script receives a URL as a parameter, logins to the IIS server using the curl binary, then it parses the output of the command and after it sees the authentication was sent, captures the response code.

Timeout pass and user values are hard-coded in the below example, the script currently only has handlers for some response codes, but a switch was used to add more in an easy way.
Response code is found with regexp /HTTP/1.1 ([0-9]{3}) .*/
if your server returns a different status code you might need to change that.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
#!/usr/bin/perl
#       02/Feb/10                       nagios@sanxiago.com
#  check_http page for IIS servers with ntlm authentication
#
# this check receives a URL as a parameter, logins to the IIS server
# using the curl binary, then it parses the output of the command
# and captures the response code. Timeout pass and user values are currently hardcoded
# script currently only has handlers for some response codes, but a switch was used to 
# add more in an easy way. Response code is found with regexp /HTTP/1.1 ([0-9]{3}) .*/
 
use Switch;
use Time::HiRes;
use Getopt::Long;
 
sub print_usage (){
print "Usage: $0 --uri="http://somepage" --user=DJohn --pass=p4ssw0rdn" ;
}
 
GetOptions( "U|uri=s" => $uri, "u|user=s" => $user,"p|pass=s"=> $pass);
 
if(!defined($uri) and !defined $user and !defined $pass){
print_usage();
}
 
$timeout=30;            # Timeout in seconds
 
$start = Time::HiRes::time();
run_command("curl -u $user:$pass --ntlm  --stderr /dev/null $uri  -i ");
$time = sprintf("%.2f",Time::HiRes::time()-$start);
 
switch ($http_code){
case 200 {print $time."s OK"; exit(0);}
case 302 {print $time."s PAGE MOVED"; exit(1);}
case 404 {print $time."s PAGE NOT FOUND"; exit(2);}
case 500 {print $time."s SERVER ERROR"; exit(2);}
case 401 {print $time."s UNAUTHORIZED"; exit(2);}
else     {print $time."s ERROR $output"; exit(-1);}
}
 
sub run_command {
$command=shift;
$pid = open(PIPE, "$command  |") or die $!;
eval {
       $output="";
       local $SIG{ALRM} = sub { die "TIMEDOUT" };
       alarm($timeout);
        while (<PIPE>) {
                if($_=~/HTTP/1.1 ([0-9]{3}) .*/ && $authentication_sent){
                        $http_code=$1;
                }
                if($_=~/WWW-Authenticate/){
                        $authentication_sent=1;
                }
                $output=$output.$_;
        }
        close(PIPE);
};
if ($@) {
    die $@ unless $@ =~ /TIMEDOUT/;
    print "TIMEOUT";
    kill 9, $pid;
    $? ||= 9;
    exit(2);
}
}

As you will see this script can easily be edited to serve as a data input in cacti or other monitoring app.