Nagios: Sending SMS notifications through gsmsmsd

Critical, that’s a point you would not ever want to reach. Monitoring is all, or quite all, on recovering a situation before it becomes a problem.

When it becomes a problem, you must move quickly and solve the issue.

So,  when you have a problem, we do not need a genius, we need flat, plain procedures which help you to undestand that

  1. There’s a problem
  2. How fast you need to move
  3. How to solve it

To help you to understand the severity of a problem, there’s a common practice: what is really critical to you business must be notified with a less used, probably more expensive channel. So, using sms adresses these needs. Reading and dealing with a SMS text message is far more unconfortable than working with emails, but reaches you in a more pervasive way than other channes. Sending emails is virtually free of costs, SMS costs. So, you use sms just to read important messages, your daddy who want’s to know where you are, the one you love who reminds  you about buyng milk coming back home, your dear Nagios alerting you on a CRITICAL, really critical problem on you IT infrastructure.

On the SMS server side

So, let’s start with the sms server part. On the server you must be sure that gsmsend is always on, watching the right directory and with its hands (doh, does gsmsend has hands?) on the terminal connected to the sms modem (I used a Fastrack Supreme 10, serial, for this post).

If you issue a command like

you should see something like:

How to reach this goal? Simple, let’s have a look to the root user crontab:

And here’s what we can see:

Using the following chart, it’s easy to undestand what we found in the crontab.

“Execute /usr/local/sbin/check_ gsmsmsd every minute”.

As you can undestand from its name, it is in charge of check if gsmsmsd is running and relaunch it if necessary.

And here’s its content:

Nothing special: the script checks wether the pid of gsmsmsd proces exists or not. If it exists, exit nicely logging that the daemon is running, else it logs that the daemon is not working, remove any contents from the sms spool directory and launches gsmsmsd as user gsmsend with the device on ttyS0 and the sms spool directory in /var/sms.  Not the “&”, the process is being detached from the terminal.

Do we have the sms spool directory? Bet not, in this case, let’s create it:

Now, let’s be sure that the user the gmssend daemon belongs to is able to read and write on that directory:

We have no more to do on the sms server side. Time to go on Nagios server.

On the Nagios server side

Here is the actual Nagios notification plugin. Use it as the notification plugin for you sms host and service contact.

Let’s give a look at the plugin:

The first lines,  start with the shabang and few command substitutions that will auto configure the script finding the right paths to the system utilities we are going to use.
The following line will create a (pseudo) random string. On the sms server we will have to create a file containing the actual sms message and number, and each file must have a unique file name to avoid “collisions”.

/dev/urandom is character special device which gives access to the random numer generator of the kernel. With the string we wrote, we have a command substitution a bit more complex than usual:

  1. We extract some random byte from urandom with cat  /dev/urandom;
  2. Pass the result (|) to tr wich will  delete from the random string everything but letters and numbers, so we won’t have any strange chars. To be true, tr translates all the chars from /dev/urandom to the chars defined in the charset we provide after “-dc”;
  3. On the third step we pass the output of the previouse operation (|) to the fold utility which formats everything in a 32 chars width column;
  4. Finally we take only the first line of the column an this is our random string.
Passwordless ssh connection

Well, what the plugin does is to put a file on the sms server. Easy, that’s it, but we want to do it in a secure way and here is where ssh becomes handy: you will setup everything we need to let our plugin connect to the sms-server using ssh and without any user/password prompt.

How do you reach this goal?

First, keep in mind wich user will execute this script: it’s “nagios” user.

So you have to setup a way to let nagios user to connect to the sms-server as users gmssend, so it will be able to write in the spool.

To do the magic you must create an rsa private key and it’s  public counterpart and the transfer the public key on the sms server. Then you reference the public key in the

of the gsmsend user and so, when you will connect the server will encrypt all the trafic with you public key of user nagios on the Nagios server and only this user will be able to decrypt it with the matching private key.

Now it’s time to create a new pair of keys for the user nagios, so let’s “su” to this user:

As user nagios, you can finally issue the keygen command:

Time to create a safe directory to store the keys in:

And now let’s move the keys in:

Let’s restrict the permissions on the key files:

Let’s check the permissions on files and directories:

Once you have the pair of key you can upload the public one to the sms-server using the ssh-copy-id utility with the following syntax:

The username is the name of the user you will connect as to the sms-server, for this example is the gsmsend, since we want to write in its sms spool. The host is the sms-server and so:

Now , try loggin onto the remote host using:

and check that only the key(s) you wanted were added.

Time to see if you can login without being prompted for a password:

Well, WOW, you are in and no password prompt seen in the meanwhile!

We can make things easier writing down some lines of configuration so that the ssh connection to the sms-server will act as we will.

Configure a ssh connection to a host for a user is a straightforward task: go to the user’s home directory, and check for the presence of a

directory, with the proper access rights: we already created it, so you should have it and with the proper access rights. If you don’t see it, look back in this post and you’ll find how to create.

Move inside the .ssh directory:

There you’ll find the private and public keys we already created. Ignore them, now it’s time to edit a new file:

Here you are going to write the following lines:

Let’s see what do they mean, line by line:

As said in the manual

The keyword host limits the scope of the following declarations. Take the word following Host as a label for a bunch of instructions and give it a proper name.

That is for the “kind” of address we use to connect to the remote host.  Any, inet to use ipv4 only or inet6 for ipv6.

Let’s say that after 10 tries we give up. One try per second. The default is 1.

Do you want to forward to the remote host the connection to the autentication agent ? No, trust me, you do not want.

We do not need, we do not use, better to switch it off. We do not want to have the X11 connections  automatically redirected over the secure channel and DISPLAY set.

We do not want remote X11 clients to fiddle with data owned by trusted X11 clients.

We allow remote hosts to connect to local forwarded ports. Can be useful.

We do not need to try rhosts based authentication with public key authentication.

Is just an alias to be used instead of the real host name when looking up or saving the host key in the host key database files. Useful when you have multiple servers running on the same host.

It should specify the real host name we want to log into. Well we use it to specify an alias for the host we want to connect to.  Write what makes it easy for you to remember the host you want to connect to.

Easy to guess, here we point to the file containing the identity we will use to autenticate to the remote host.

We are using an identity file to authenticate so let’s make sure we do not fiddle with passwords.

Here we define wich remote port to connect to. The port 22 is the standard for the ssh service but if you want to make more difficult a brute force attack from automated penetration tools, change the port on the remote host and write here the new value.

Really? You would use the old protocol version 1? Not at all!

It sets the number of max retry sending server alive messages and not receiving an answer from the remote host. Let’s take it as a timeout counter. If the host doesn’t answer to the server alive messages for 3 times, the ssh session will be disconnected.

This keyword sets the interval in seconds after wich if no data is received from the remote server ssh will send a server alive message through the ssh encrypted channel.

TCP keepalive are not sent through a secure channel and so they are spoofable. Better not to use it.

This is the user on the remote host we are logging as.

The dirty thingy string

At the end of the script you’ll find the string that actual do the dirty job.

  1. First it connects to the remote hosts;
  2. Then, on the remote host, it echoes the mobile number;
  3. A new line char follows;
  4. And on the second line there will be the notification text;
  5. Finally, the standard output to  a file in the spool directory, using as a file name  a random string calculated at the beginning of the script .

That’s it. We won’t look at the rest of the code, it’s quite simple and should give us no problems.

Just have a look to a handy variable definition:

When the variable is defined with a

means that if you do not pass any value, the variable defaults to

call it a fallback value.

Here you find the actual script:

Setting up Nagios to send SMS

Now we have all the bits we require to send some notifications using SMS, what we need now is to glue all together in Nagios.

We need to:

  1. Copy the code above and save on a file;
  2. Upload the file on the Nagios plugin directory on the Nagios server;
  3. Make the file executable by the nagios user;
  4. Create a Nagios command to call the script from within Nagios;
  5. Create a contact which will use this command as a notification command;
  6. Assign the command.

Let’s say we call the new script

Copy it on the Nagios plugin directory server, make nagios user its owner

and make it executable by Nagios, restricting a bit the rights on it:

Now we create two different notify command, one for hosts notifications and another for services notifications:

Notice that

points to the Nagios plugins directory and is defined in the file

All the other variables are passed to the script by Nagios. If you want more informations on Nagios macros, follow this link

http://nagios.sourceforge.net/docs/3_0/macros.html

Now it’s time to create a contact which will use the new command

Just put after

the mobile number to send SMS notifications to.

Let’s attach the SMS notification to a service

Now, let’s do the same for the host:

Keep in mind that these are just examples: modify your Nagios objects accordingly.

Final step, check that the modified configuration is not broken

If it’s all ok, reload Nagios and try to get a CRITICAL notification on a fake service or host witch uses oncall-sms as contact. Don’t you have it? Create a fake just for testing and if you do not see any errors and everything is working fine, attach the new contact to your most valuable host and services.

Nagios – Check BigIP F5 memory

This is the last plugin I coded for F5 BigIP. As usual, I just found this old Nagios plugin of mine, sitting in a directory, taking dust.

As usual, I’m not a programmer, so I just do quick and dirty tricks to get what I need, so here it is the plugin in all it’s bash glory.

The plugin is commented, although in Italian, but google can translate the comments.

I wrote this plugins some years ago, it works using SNMP as protocol and with the standard (at those times) OID for BigIP. If they’ve not changed, it should work with no efforts. It’s based on v3 of SNMP protocol and the auth and priv are “hardcoded”, so you may want to change them.

Have fun.

 

 

 

Nagios – Check BigIP F5 load

As for the previous F5 plugin, I just found this old Nagios plugin of mine, sitting in a directory, taking dust.

As usual, I’m not a programmer, so I just do quick and dirty tricks to get what I need, so here it is the plugin in all it’s bash glory.

The plugin is commented, although in Italian, but google can translate the comments.

I wrote this plugins some years ago, it works using SNMP as protocol and with the standard (at those times) OID for BigIP. If they’ve not changed, it should work with no efforts. It’s based on v3 of SNMP protocol and the auth and priv are “hardcoded”, so you may want to change them.

Have fun.