Config Fun

Few days ago I had that nice experience setting up a server on (already installed not by me) Fedora 7. Among other things, this server should be a gateway for LAN running DHCPd, DNS, iptables and all those other services for service the LAN. One thing was just driving me crazy: I had static IP for LAN ethernet, but each time I was restarting either network service or dhcp, that ethernet was getting an IP from local dhcp (that sux). I checked almost everything to make sure I specified that the ethernet should use statically assigned IP and still had no luck.

After fighting for some time I found our that NetworkManager was running and watching all those network changes. Each time I was restarting network or dhcp, NetworkManager was ignoring all the bloody settings and just trying to find best solution for me screwing everything up

Just read a nice article about PC based opensource firewall which was mostly based on Smoothwall, M0n0wall and pfSense. Being a user of pfSense for almost a year now and I totally agree with the author about pfSense being one of the best alternatives to go with. From my side I can only add that I have 5 pfSense based firewalls in different places and I didn’t had any problems with them yet.

If you are interested in the topic and/or looking for PC based firewalls, check the article to get some really true points.

I’ve been working with RRDTool graphs a lot for the past few days and noted a couple of nice tips for myself which I want to share here:

Graphing Real Numbers as Integers

Sometimes your database contains number in real format (ex: 4.23) and you want to see only integer values on the graphs. For example the database keeps track of number of users sessions and computes AVERAGE, but since having 2.123 sessions number is not that nice (for me at least in this case), it is possible to round it up to integer and have 2 (or 3 if the original real number was more or equal to 2.5). For this, in the graph definition, you need to create a variable to store integer as follows (hate this postfix notation type):

CDEF:var_int=var_real,1,%,0.5,GE,var_real,CEIL,var_real,FLOOR,IF

In normal notation it would look like <code>var_int = (var_real % 1 >= 0.5) ? round_up(var_real) : round_down(var_real)</code> and would round the real var_real number to the upper or lower corresponding integer according to the value after the delimiter. So after such definition you can use var_int for graphing any shapes.

Graphing the Unknowns

It is a good practice to change the background of the graph parts where all variables are unknown. In order to graph such areas I first check if the all variables are defined and if not, I create two other variables, one with value from 0 to -unlimited and second from 0 to +unlimited and display them as AREAs of the same colors. So here we go:

CDEF:u=var1,var2,+,UN,INF,UNKN,IF creates a variable to indicate 0 to +unlimited if the sum of all actual variables is unknown

CDEF:u2=var1,var2,+,UN,NEGINF,UNKN,IF creates a variable for 0 to -unlimited under the same conditions as u.

AREA:u#FFFFB9
AREA:u2#FFFFB9

displays the AREAs with a color I like.

Examples

Let me just post couple of examples of graph definitions here:

<RRD::GRAPH firewall-Net:Traf-LAN.png -w 845 -s <RRD::CV period> -c BACK#FFFFFF -c SHADEA#FFFFFF -c SHADEB#FFFFFF -c FRAME#0000 00 -W 'IT Department' -v "Bits" --title="firewall - LAN Traffic"
DEF:up_rrd=firewall-Net\:Traf-LAN.rrd:up:AVERAGE
DEF:down_rrd=firewall-Net\:Traf-LAN.rrd:down:AVERAGE
CDEF:up=up_rrd,8,*
CDEF:down=down_rrd,8,*
CDEF:down_show=down,-1,*
VDEF:down_min=down,MINIMUM
VDEF:down_ave=down,AVERAGE
VDEF:down_max=down,MAXIMUM
VDEF:up_min=up,MINIMUM
VDEF:up_ave=up,AVERAGE
VDEF:up_max=up,MAXIMUM
COMMENT:' '
COMMENT:' Max'
COMMENT:' Ave'
COMMENT:' Min\n'
COMMENT:' \n'
AREA:down_show#00FF00:'Incomming '
GPRINT:down_max:'%8.2lf%Sb'
GPRINT:down_ave:'%8.2lf%Sb'
GPRINT:down_min:'%8.2lf%Sb\n'
AREA:up#0000FF:'Outgoing '
GPRINT:up_max:'%8.2lf%Sb'
GPRINT:up_ave:'%8.2lf%Sb'
GPRINT:up_min:'%8.2lf%Sb\n'
CDEF:u=up,down,+,UN,INF,UNKN,IF
CDEF:u2=up,down,+,UN,NEGINF,UNKN,IF
AREA:u#FFFFB9
AREA:u2#FFFFB9
>

The above definition is for traffic monitoring

<RRD::GRAPH firewall-Net:Conn.png -l 0 -w 380 -s <RRD::CV period> -c BACK#FFFFFF -c SHADEA#FFFFFF -c SHADEB#FFFFFF -c FRAME#000000 -W 'IT Department' -v "Connections" --title="firewall - Established TCP Connections"
DEF:total=firewall-Net\:Conn.rrd:total:AVERAGE
CDEF:total_show=total,1,%,0.5,GE,total,CEIL,total,FLOOR,IF
VDEF:total_min=total,MINIMUM
VDEF:total_ave=total,AVERAGE
VDEF:total_max=total,MAXIMUM
COMMENT:' '
COMMENT:' Max'
COMMENT:' Ave'
COMMENT:' Min\n'
COMMENT:' \n'
AREA:total_show#0000FF:'Connections '
GPRINT:total_max:'%10.0lf'
GPRINT:total_ave:'%10.0lf'
GPRINT:total_min:'%10.0lf\n'
CDEF:u=total,UN,INF,UNKN,IF
CDEF:u2=total,UN,NEGINF,UNKN,IF
AREA:u#FFFFB9
AREA:u2#FFFFB9
>

And this one is showing the number of established TCP connections.

Note that both graphs use the <RRD::CV period> as a definition of the beginning of the graph period. I use this when viewing the graph by supplying the ?period=-1h or similar to display the graphs for different time periods.

I have worked with GNU screen WM for a while now and I have heard a lot of people also like it. Personally, I find the tool to be very handy when you need to run something on a remote terminal without being connected to it all the time. The idea is that I login to the remote machine using SSH, then start screen (screen -a), then start the app I need (for ex top) and then just detach the screen using <Ctrl>-A,D shortcut and disconnect from the remote server, while my top command continues running there. In while, when I need to check how thing are going with my app, I just connect back through SSH and reattach the screen by using screen -r  command.

Though top is a too trivial example of a command one may run, I use screen for real-time parsing of Nagios perf-data which is retrieved by my script (will publish it here one day) from named pipe (to which nagios writes all performance data) and then creates RRDs.

Here it is - I managed to get it all together and set up SMS notification from Nagios alerts via twitter. The job to be done is very easy:

First create an account on twitter to be used along with nagios. You probably also want to indicate that the account should be private via twitter settings in order for the whole internet not to see what is going on with you monitoring :). So go twitter and get an account.

Second, here is a piece of code for a script which I use as a notification handler:


#/usr/bin/perl -w
use strict;
use Net::Twitter;
my $msg = shift;
my $twit = new Net::Twitter(username => "twitter_user",password => "twitter_pass");
if (defined($msg) && $msg !~ /^\s*$/) {
$twit->update($msg);
}


Don’t forget to replace twitter_user and twitter_pass with the details of twitter account you have created in above stage.

Now modify the nagios notification commands in the misc. commands config file (/etc/nagios/misccommands.cfg in my case) to utilize the erlier created twitter.pl script. I have the following definitions:


define command {
command_name notify-by-epager
command_line /path/to/twitter.pl "$NOTIFICATIONTYPE$ - $HOSTNAME$-$SERVICEDESC$ - $SERVICESTATE$ - $SERVICEOUTPUT$"
}

and the same for host-notify-by-epager.

Finally adjust your nagios contact definition to use the above commands for notifications and don’t forget to enable all the notifications you need.

The above part will post all alerts to nagios twitter account. In order to be able to receive them via SMS or GTalk, add nagios twitter account as a friend to your personal twitter account (don’t forget to grant access to become a fried from nagios twitter account if you made it private) and follow up