Monitoring Puppet with Munin

Just recently we had some hickups in our puppet infrastructure.

The question we were facing afterwards was, if all the nodes are regularily getting their updates. The specific problem is, that the infrastructure of the puppet and svn system is maintained by another department and so we don't have access to their exact monitoring data.

So the idea was to write a munin plugin, that will alarm me, when one or many hosts are late for their updates.

This would be trivial. I wanted a special solution. I wanted the plugin itself, not only configured by the puppet, but I wanted it to be completely generated by puppet.

There is a puppet-module Concat which allows to build Textfiles from Fragments.

So My idea was, to generate a Plugin completely from fragments that are generated on all the puppetnodes themselfes. And Thats what I did: Put the following receipt into the manifest for the host, where the plugin should be shown:

    concat { "/usr/local/share/munin/plugins/puppetlastrun":
        owner => 'munin',
        group => 'munin',
        mode => '755',
        notify => Service["munin-node"],
        }
    concat::fragment {"/usr/local/share/munin/plugins/puppetlastrun.header1":
        target     => "/usr/local/share/munin/plugins/puppetlastrun",
        order => 04,
        content =>
"#!/bin/bash
#
# Generated by Puppet:
# $Revision
#
#
if [ \"\$1\" = \"config\" ]; 
    then
        echo 'graph_title Puppetrun-Ages'
        echo 'graph_args --base 1000 -l 0 '
        echo 'graph_category Updates'
        echo 'graph_info This graph shows age of last puppetrun'
        echo 'runtime.label last runtimes'
"
        }

    concat::fragment {"/usr/local/share/munin/plugins/puppetlastrun.header2":
        target     => "/usr/local/share/munin/plugins/puppetlastrun",
        order => 06,
        content =>
'               exit 0
    fi
'
        }

    concat::fragment {"/usr/local/share/munin/plugins/puppetlastrun.work8":
        target     => "/usr/local/share/munin/plugins/puppetlastrun",
        order => 28,
        content =>
'
exit 0
'
        }

    Concat::Fragment <<| (tag == 'muninplugin_puppetruntimes') |>>

    file {"/usr/share/munin/plugins/puppetlastrun":
        ensure => link,
        target => "/usr/local/share/munin/plugins/puppetlastrun",
        require => File["/usr/local/share/munin/plugins/puppetlastrun"]
        }

This will generate a skeleton of the munin-plugin on the munin-node, thats monitoring. You can spot the Concat::Fragment <<||>> - which gathers external resources. These will be fillled by the nodes themselfes. Now we need to fill the plugin with data, that every node has to supply.

Put the following receipt into each puppetnodes manifest:

@@concat::fragment {"/usr/local/share/munin/plugins/puppetlastrun.header.$hostname":
    target   => "/usr/local/share/munin/plugins/puppetlastrun",
    tag => [ "${hostname}","muninplugin_puppetruntimes","$environment"],
    order => 05,
    content =>
"
            echo \"runtime_$hostname.label runtime_$hostname\"
            echo \"runtime_$hostname.draw AREASTACK\"
            echo \"runtime_$hostname.info $hostname \"
            echo \"runtime_$hostname.warning 28800\"  # 8 hours
            echo \"runtime_$hostname.critical 86400\"  # 24 hours
"
    }
@@concat::fragment {"/usr/local/share/munin/plugins/puppetlastrun.work.$hostname":
    target   => "/usr/local/share/munin/plugins/puppetlastrun",
    tag => [ "${hostname}","muninplugin_puppetruntimes","$environment"],
    order => 25,
    content =>inline_template(
"           echo runtime_$hostname.value \$(( \$(date +%s) - <%= Time.now.to_i %> ))  # Last Puppet run was <%= Time.now %>
"),
    }

This fills the munin-plugin once with the config-options for this host, and also generates the stuff neccessary to return the value to munin.

The interesting part is the line:

content =>inline_template("echo runtime_$hostname.value \$(( \$(date +%s) - <%= Time.now.to_i %> ))  # Last Puppet run was <%= Time.now %>")
  • First we're using inline_template() - this allows us to get the current timestamp, and embedd it into the returned string. Thats the part with <%= Time.now.to_i %>. This gives us the current timestamp in seconds since 1.1.1970.
  • this Timestamp gets embedded in an bash-command:
    • echo $(( )) allows us to do some small calculations on cmdline
    • $(date +%s) returns the time when called in the plugin