PICA manual

Miguel Armas del Río¹
Esteban Manchado Velázquez²

Abstract: This manual tries to show the basic PICA usage, as well as describe its inner workings (just the necessary to make PICA do what it can do at all). It was initially based in an article from SysAdmin. This version applies to the PICA 0.4.x series.

1 Introduction

At the ULPGC's Network Division we administer several servers that run the critical network services such as DNS, DHCP, Network monitoring, etc. Because these services are critical, we run a number of scripts on every server that check the sanity of these services and try to fix basic error situations.

We needed a way to distribute all these scripts and the important services' configuration files from a centralized location with little differences to adapt them to each host. We also needed a way to register any change on the configuration files, to be able to detect when a particular error was introduced (and who did it ;-)). We also wanted to centralize all network incident notifications and alarm management.

To meet all these needs, we developed PICA. With PICA we have a central repository of configuration files and alarm scripts. This repository is managed using CVS, so we can recover old versions, see changelogs, and let various admins to work concurrently. Actually, every sysadmin has a local copy of the working tree, and CVS does all the dirty work.

In this scenario PICA is used to distribute the configuration files and alarm scripts to the various servers. PICA uses SSH to establish secure connections to the remote servers, which is very convenient, since we where already using SSH with RSA authentication to access all remote servers.

The alarm scripts send incident notifications and service status reports to our central NetSaint server using asynchronous checks (see netsaint documentation). If a critical error is detected, an alarm is also sent via e-mail and as a SMS message to the sysadmin mobile phone.

2 Concepts

PICA relies on several important concepts:

Objects: Objects are ``thingies'' we want installed in our servers. Of course, one thingie can be different depending on the server it is to be installed in. For this purpose, objects are represented with files which can contain special directives, interpreted by the Perl PreProcessor (one of the PICA components). There are two types of thingies (also called ``distribution files'', or ``distributed files''): files and alarms.
Files: Regular files to be installed in a given path of the remote machine.
Alarms: Alarms are executable files, to be installed in special directories which PICA takes care of. Alarms are executed from time to time, and they are supposed to produce some output if there is something to notify. Furthermore, alarms can have dependencies, which are files that have to be (re)installed if the alarm depending on them is (re)installed.
Hosts: Your (theoretically remote) machines. You are supposed to have your configuration files in one host, to copy them in a bunch of machines. These machines are the ``hosts''.
Groups: PICA recognized two types of groups: host groups and object groups. Groups are used to organize and easily manage large collections of objects. More on groups later.
Attributes: Attributes are object properties. There are mandatory and optional attributes, depending on the type of object (file or alarm).
Variables: You can use ``variables'' that can be referred to in the preprocessor and the files to be distributed. Sometimes we use the adjective ``local'' to refer to object variables, and ``group'' to refer to those of...you get the idea.
Definitions: They are variables defined as command line arguments (more on that later).

2.1 Groups

If one refers to a group, the action affects all the objects in that group. You can think of alarm dependencies as implicit groups, only that the group is an object per se. There is one important difference between host groups and object groups: one host can belong to an arbitrary number of groups, but an object belongs to one group or to none at all.

2.2 Attributes

Attributes are properties associated with objects (both files and alarms). They are used to define local and remote paths to the objects, permissions to be set on installing, and so on. They are not to be confused with local variables (rather different concept). The difference is that attributes are for pica to know about them and behave accordingly, while local variables are used-defined variables to substitute into the installed objects, or to make conditionals or other references in Perl code.

2.3 Definitions

Definitions are very useful for changing little things (or big, for that matter) at command line. They are defined with the +D flag. One can define (the existence of) a variable, or assign it some value. The syntax is +D myvariable for the former or +D myvariable=myvalue for the latter.

3 Configuration files

There are currently three configuration files, parsed in that order: pica.conf, hosts.conf and objects.conf. All of them are placed in the /etc/pica directory. The first is a configuration file to control behaviour of the pica main script; the second, a description of the hosts PICA will work on; the last, a description of the objects to process.

3.1 Basic syntax

All the configuration files share a common syntax, similar to that of the DNS config files. Before being read, they are parsed by the Perl PreProcessor. Generally speaking, there are properties and constants, assigned to those properties. The properties must match the following regex:

(\w|/|\.|-)+

That is, letters, slashes, dots and hyphens, in whatever order you want. Note that there are some reserved words, that vary between configuration files.

On the other hand, constants come in three flavours:

Plain: The same as properties.
Single quoted constants: Whatever you want to put between single quotes, including (but not limited to ;-)) single quotes, as \'.
Double quoted constants: Whatever you want to put between double quotes, including double quotes, as \".

Please note that the difference between single and double quotes is purely aesthetic. Just a last note: everything is case-sensitive in every configuration file!

3.2 The Perl PreProcessor

This PICA component is the one that reads your files first, to transform them into what PICA will really read. In fact, is a Perl sub (well, two), not an independent program or CPAN module or something. The idea is being able to change (in a general and efficient way) the configuration files, depending on variables (yes, local variables) and some other checks. All of that, using arbitrary Perl expressions, that can not only return boolean values for conditions, but also generate part of the configuration or distribution files.

As the preprocessor is used on every PICA configuration and distribution file, the results become terribly dynamic. Imagine generating a configuration file for your servers, based on the name of the host, the name of the final file to be installed, the time of the day or some other external file or condition! You have all the power of Perl in your hands!

The directives recognized by PPP are:

#include For including external files. You can use both angle brackets and double quotes. In the first case, the file is read from the directory where the file was read; in the former, the file will be read from $picainclude, usually something like /var/lib/pica/include.
#if/#elsif/#else/#fi For conditionals, from parenthesized arbitrary Perl conditions
#perl/#lrep For executing completely arbitrary Perl code. Anything returned by this code will be substituted for the original code
<# ... #> Inline #perl/#lrep environment (for one-liners)

To support your nifty conditions, there are sub definitions (actually, only two, ingroup and members) and some variables ($picahost and $picaobject, only defined for objects.conf; more on that later). Subs are defined in user.pm, a special module to put all of that stuff, placed in the PICA directory, usually /var/lib/pica. The user is supposed to modify that file if he wants to add a new function. The current list of defined subs is:

ingroup: Takes one argument and returns true if the current processing host (that is, picahost) belongs to the given group.
members: Takes one argument, and returns the list of members of the given group, or "" if there aren't any.

To add a function, you have to write it in user.pm and add it to the EXPORT and EXPORT_OK variables. Moreover, if you want to access a new variable, you'll have to make a glob to see it and be able to modify it's value. You can copy and paste the current subs and globs.

3.3 `pica.conf`

This configuration file defines some paths needed by PICA. It's a defaults environment with some definitions. There is one special definition, protecteddirs, with a list of directories never to be deleted, under any circumstances (this list is checked every time PICA is about to delete some directory recursively). Here's a pica.conf sample file:

defaults {
   picaroot  = /var/lib/pica;               # config files
   picatmp   = /var/lib/pica/tmp;
   picasrc   = /var/lib/pica/src;           # distribution files
   picainclude = /usr/lib/pica/include;

   sshpath   = '/usr/local/bin/ssh '; # ssh binary
   diffpath  = '/usr/bin/diff'; # diff binary for -f command
   tarpath   = '/bin/tar';      # tar binary
   rsyncpath = '/usr/local/bin/rsync';

   protecteddirs {
      /,
      /bin,
      /usr/bin,
      /lib,
      /usr/lib,
      /var/lib/pica
   }
}

I hope all of this is self-explanatory.

3.4 `hosts.conf`

It is used to configure the hosts that should be administered using PICA. We can organize the hosts in different, possibly overlapped, groups that can be later used in conditions, so we could install a given file only in hosts belonging to a given group. We can also define variables for a given host and use them later. We also can specify different values for an object variable depending on the group we are working on by calling the ingroup sub.

Note that every host should be ``declared'' for PICA to know about it. It's a way to try to prevent you from shooting yourself in the foot.

A simple example of the hosts.conf file could be the following:

defaults {
   vars {
      ## ---- User defined variables ----
      docdir  = '/home/httpsd/html/red';
   }
}

##
## Hosts Definitions
##
host machine3;
host machine2 {
    vars {
       var = 'this variable sucks';
    }
}
host machine1;
host machine4;
host machine5;
host machine6;
host machine7;
host machine8 {
    fqdn = machine8.otherdomain.net;
}

###########################
# Host Group Definitions  #
###########################

##
## Group Definitions
##

## Servers
hostgroup servers {
    members { machine2, machine1, machine4, machine7, machine5, machine6 }
}

## RedHat Linux Servers
hostgroup redhat {
    members { 
        machine2, machine1, machine4, machine5, machine6
    }
}

## Debian Linux Servers
hostgroup debian {
    members { 
        machine8
    }
}

#####################
# Mail servers
#####################
## No relay
hostgroup mailmaster {
    members { machine2, machine1, machine4 }
}
## Mailing lists servers
hostgroup listservers {
    members { machine4 }
}
## Main domain server
hostgroup ulpgcmaster {
    members { machine4 }
}

################
## DNS Servers
################
hostgroup dnsservers {
    members { machine2, machine1, machine4, machine5, machine6 }
}
hostgroup dnsmaster {
    members { machine5, machine6 }
}

################
## NTP Service
################
## Stratum-2
hostgroup stratum2 {
    members { machine2, machine1, machine4, machine5, machine6, machine7 }
}

##
## GSM server
##
hostgroup sms {
    members { machine5 }
}

################
## Misc services
################
hostgroup imapservers {
    members { machine4 }
}
hostgroup popservers {
    members { machine4 }
}
hostgroup simapservers {
    members { machine4 }
}
hostgroup tftpservers {
    members { machine4 }
}
hostgroup ftpservers {
    members { machine4 }
}
#######################
## Some networks
#######################
hostgroup net1 {
    members { machine2, machine4, machine7 }
}
hostgroup net2 {
    members { machine1 }
}
hostgroup net3 {
    members { machine5, machine6 }
}
#########################################
## Documentation hosts
#########################################
hostgroup doc {
    members { machine5, machine6 }
}

Whoa! What a giant file. Well, perhaps you have noticed under all this mess that hosts (not hostgroups) can have attributes, such as ``fqdn''. This way we can specify host information such as the name PICA will use to create the secure connection. The list of possible host attributes is:

fqdn: As we already said, it sets the name PICA will use to connect to the host. If we don't specify it, PICA will use the host identifier (yes, the word after the reserved word ``host''). This attribute is useful if you have hosts in different domains and don't want to use the FQDN for the host definition. We can also define which user we want to connect as for a particular machine, with the ``sshuser'' attribute.
method: Is the method used to copy files to the host. Its possible values are ``ssh'' and ``tar''. It will be discused later, in page ??.
shellcmd: Is an attribute that specifies the complete command used to connect to the remote host. It uses the variables $sshargs and $cmd, which will be interpolated by the ssh arguments and the command to be executed, respectively. It defaults to ``$sshpath \$sshargs picasshuser@$picafqdn $cmd''.
sshuser: The user we will connect to the machine with. It's usually root, but for some machines we would want to change the default.
defpathdir: Default path to install, for the objects with relative ``source'' attribute values and no ``path'' attribute.

One very important thing to note here is that there are some special values in the hosts.conf defaults environment. These variables point to important directories in the remote machines³. We would like them to change in a per-host basis, but, by now, they are ``global''. Of course, chances are this will change sometime in the future.

The list of special values is:

picaalarms: Directory where the alarms reside. In fact, directories with the name of the different priorities are created, and the alarms really reside there.
picabin: Directory where misc binaries will be put, as scheduler.
picalib: Directory for collections of things.
picaobj: Directory for objects used by alarms, mostly.

Just one last note: it is possible to specify hostgroups in the ``members'' attribute of a hostgroup, so you can build hostgroup hierarchies. It's very useful if you have a group like ``DNS servers'' and subgroups like ``Windows DNS servers'' and ``Linux DNS servers'' (i.e.: perhaps you want to treat all DNS servers at once for some things, and separately for other things).

3.5 `objects.conf`

This one defines all the objects that will be distributed using PICA, and depends on the current host. In general, one can include or generate dynamically parts of it, depending on evaluations of previously defined variables. The variables available at this time for the PPP will be:

Variables defined in the hosts.conf ``defaults'' environment
Variables defined in the objects.conf ``defaults'' environment
Variables defined in any of the groups which the current host belongs to
Variables defined in the current host

The latter, of course, have preference in case of multiple definition. We also can use the internal variable $picahost to get the name of the current host. Because of that, the possibilities explode here: we can write entire Perl scripts (or in other languages, as far as we call them from Perl) to dynamically generate out configuration files on the fly, depending on defined variables, the day of the week, or whathever condition we can imagine. And here resides the power of PICA.

The mandatory and optional attributes for the two types of objects are:

Mandatory attributes
source
where to read the file source from. If this is a relative path, $picasrc is prepended.
priority
Determines the priority of the alarm. It also determines where the alarm will be installed in the remote host. ($picaalarms/$priority)
source
where to read the alarm code from. If this is a relative path, $picasrc is prepended.
Additional attributes
uid
uid to set in the installed file. Default: 0
gid
gid to set in the installed file. Default: 0
verbatim
If set to 1, the object will be installed without preprocessing it. It is useful to install binary files, and files that could include preprocessor directives that we don't want to parse. Default: 0
perms
file permissions to set in the installed file. Default: 644
path
where to install the file in the remote host. If it isn't specified, use the same as ``source'' attribute. This attribute is only used in file objects.
uses
in the case of alarms, we can specify a uses clause with the alarm dependencies. These dependencies are regular files, that only will recognize the alarm as its group, not any group the alarm may belong to (most importantly, in the case of variables and default attributes).

For any of these attributes we can define default values in the ``defaults'' environment (see example).

A simple objects.conf file could be:

## Default values
defaults {
      uid = 0;
      gid = 0;
      perms = 644;
      verbatim = 0;
      vars {
         globalvar1 = 'value1';
      }
}

file motd {
    path = '/etc/motd';
    source = 'Services/Info/motd';
}

##################################
# Configuration for Service: NTP #
##################################
group NTP {
    file ntp.conf {
       path = '/etc/ntp.conf';
       source = 'Services/NTP/ntp_conf.cfg';
    }
    file step-tickers {
       path = '/etc/ntp/step-tickers';
       source = 'Services/NTP/step-tickers.cfg';
    }
    ##
    ## Documentation for this Service
    ##
#if (ingroup('doc'))
    file README.ntp {
       path = '<#$docdir#>/Services/NTP/README';
       source = '<#$picasrc#>/Services/NTP/README';
    }
#fi
}

##################################
# Configuration for Service: DNS #
##################################
#if (ingroup('dnsservers'))
group DNS {
    file named.conf {
         path = '/etc/named.conf';
         source = 'Services/DNS/named_conf.cfg';
    }
    ##
    ## Documentation for this Service
    ##
#  if (ingroup('doc'))
    file README.dns {
       path = '<#$docdir#>/Services/DNS/README';
       source = 'Services/DNS/README';
    }
#  fi
#fi
}

## DNS check
alarm DNSChkUrgent {
    priority = 'Urgent';
    source = 'alarms/DNSChk';
    perms = '755';
    uses {
        file dnschk.conf {
            source = 'alarms/conf/dnschk.conf';
            path = '<#picaobj#>/conf/dnschk.conf';
        }
    }
}

We also have a defaults section that will be seen from any object, and we can also define variables that will only be seen by the object they are defined in.

Anytime PICA operates with an object, it will build a namespace containing all the variables seen by that object in the host it is working on. Of course, the most specific definition has precedence. That is, if we define a variable in an object, and that object belongs to a group where the same variable is also defined, the value used is the one given in the object definition. In addition to these variables, we always have available $picahost and $picaobject, which hold the current host and object we are processing.

As a side note, since pica 0.3.6, files that happen to be directories are copied recursively, but treated verbatim (that is, no preprocessing is done on any file).

4 Command line syntax

Once we have built our hosts and objects definitions, we can start using PICA to do something useful. Basically, PICA always does ``something'' to a list of objects on each of the given hosts. As the the help option ``-h'' states, PICA has the following command line syntax:

(PICA) Perl Installation and Configuration Agent 
Version: 0.3.4 
Usage: ../pica -[ixtflh] [-n] [-v] [--with-picaconf newpica.conf]
          [--with-hostsconf newhosts.conf] [--with-objectsconf newobjects.conf]
          +D defines +|-F objects +|-H hosts
       -i : Install objects
       -x : Execute object/command
       -t : Delete object
       -f : Diff object
       -l : List objects
       -h : Shows this help
       -n : Debug. Do not install/delete things, just testing 
       -v : Be verbose

       --with-picaconf    : Changes pica.conf path
       --with-hostsconf   : Changes hosts.conf path
       --with-objectsconf : Changes objects.conf path

       +D   : Build defines list
       +|-F : Build object list
       +|-H : Build hosts list

The command line has three different mandatory parts:

The command: this is the first group of options, and determines what we want to do.
The object arithmetic: Determines what objects we want to operate on. The object list is built using +F/-F to add/delete objects
The hosts arithmetic: Determines what hosts we want to operate on. The hosts list is built using +H/-H to add/delete hosts

Moreover, there is an environment variable, called PICAARGS, that will be pasted at the end of the command line (useful for having some parameters fixed at the end of every PICA call).

4.1 Object and host arithmetic

The object and host list on which we want to operate is built with +F/-F and +H/-H. With +H/-H we add/delete hosts or groups to the hosts list in the same order they are entered. For example, the expression:

+H dnsservers solaris -H deimos

will result in the host list ``fobos mercurio sar'' since PICA will add the members in groups dnsservers and solaris and delete host deimos⁴. Note that you can specify hosts not defined in hosts.conf. Those hosts will belong to no group, and connections will be made to that name. Moreover, only ``common'' objects will be seen. But it's a start.

One more feature you will want to know of is the ``automatic exclusion''. Before doing the actual host count, an implicit -H sysdown is added at the end of the list. This allows the sysadmin to automatically delete some hosts that can be down at a given time.

The object list is built the same way using +F/-F, but the object arithmetic is evaluated for every host, since the objects available for every host can be different because we can use conditionals in the objects file. For example, in the objects file above, all documentation objects will only be seen by members of the group ``doc''.

Since we always need at least one host and object, options +F and +H are mandatory. If after doing the host/object arithmetic PICA gets an empty list (either objects or groups) it gives an error message and aborts execution.

4.1.1 Implicit groups

Implicit groups are groups defined automatically by PICA to ease host and group arithmetic. Currently, there are only two implicit groups defined:

all: Obviously, it refers to all of the hosts or objects (depending on where we use it).
alarms: Used in objects, it refers to all of the objects with type alarm, and all of the files uses'd by them.

Moreover, if sysdown is not defined, PICA will define it automatically (to a empty list, of course).

4.1.2 Definitions

We can also use +D/-D to build an optional list of definitions that can be used to preprocess files and as variables in object definitions, as if they where defined in the ``defaults'' environment of the hosts.conf file. Actually there is a little but very important difference, since they are defined before reading the hosts.conf file, these are the only variables that can be used to conditionally preprocess this file.

4.2 PICA commands and options

Right now PICA supports five different internal commands, but more could be added in the near future since we are still adding new features.

With each of these commands we can use the -v option for verbosity, and -n for debug (as with make, it does nothing, simply print information and "simulate" it's working). Option -n gives a lot of output and doesn't really do anything, just prints what it would do.

All these commands are executed using SSH connections to the remote hosts. Right now PICA doesn't have any access control system, and will probably never have one. We like SSH and it gives us everything we need to access remote servers securely, so we used it for PICA. You know, as they say: KISS⁵.

Just one suggestion, if you are willing to use PICA, you better configure properly the RSA authentication, or be ready to type a lot of passwords... One nice trick is to distribute the SSH's RSA authentication files using PICA as explained in one of the real life examples described later.

Install (-i)

This command will install the given objects in each of the given hosts. The install command first generates the objects it will install in a local dir ($picaroot/tmp/$picahost), and then install them in the remote host.

PICA supports three different methods for remote file installation:

ssh: This is the default method. It install each file using an SSH connection. This is the default method because it only needs SSH, but it's painfully slow because it makes a different connection for each file.
tar: This method uses tar over a SSH connection to transfer all files using the same SSH connection. Right now this is the recommended method.
rsync: This method is still under development because we have found some problems with the way rsync handles directory permissions and symlinks. It will be the recommended method when we fix this problems because it's the fastest. Right now use it at your own risk, what currently means: DON'T USE IT.

To change the installation method, set the attribute ``method'' per machine, in hosts.conf (you can set it globally or individually, distinct for each machine). To use methods tar and rsync, you must also set the binary path to this utils with the attributes ``tar'' and ``rsync''.

The command:

pica -iv +F NTP -F step-tickers +H all -H deimos

Will install all objects in the group NTP except step-tickers in all servers except deimos.

Execute (-x)

This command will execute the given list of commands in each of the given hosts. If PICA finds an object with the given name, it will read its path attribute and use it for remote execution. This way we don't have to remember the location of the script. If it doesn't find an object with that name, PICA assumes it's a Unix command and tries to run it.

For example, the command

pica -xv +F DNSChkUrgent +H servers

will execute /var/lib/pica/alarms/Urgent/DNSChkUrgent in all servers, since the object DNSChkUrgent exists and PICA can build the remote path using the object's attributes.

On the other hand, the command

pica -xv +F "ndc reload" +H dnsservers

will run the command ``ndc reload'' in each member of the dnsservers group. Since PICA can't find an object named ndc, it assumes it's a Unix command and tries to execute.

List (-l)

This command lists the given objects. It basically executes ``ls -l'' for every object's path. It can be used to see if an object is installed, and if the uid/gid and file permissions for that object are correct.

Delete (-t)

This command deletes the given objects in each of the given hosts.

Diff (-f)

The diff command finds differences between the object that should be installed in a host, and the one really installed. It basically generates the object for that host, and makes a ``diff -u'' between this object and the one installed in the remote host.

It's very useful to see if a host has the latest version of an object.

5 PICA Framework for Integrated Alarms (PIFIA)

The design of the PIFIA floats around the following concepts:

Flexibility
Simplicity

Alarms are a special type of object: we will normally want to execute it and it can depend on some files (regular files, not other alarms). So, it needs one more attribute (priority), has an additional optional attribute, and a special additional environment to support dependencies. But, as the rest of the objects, it's installed in the appropriate hosts, so we will always have the possibility of execute any alarm by hand, even if the ``central'' host is down or unreachable.

The basic idea is that there is a program, the scheduler, that is executed by cron (a pifia.cron file is in the PIFIA distribution). Each time it's run, it looks for alarm scripts to be executed...and executes them. Alarms have a priority, that defines in which directory will they reside. This lets the scheduler treat at different time intervals (defined in pifia.cron, of course) alarms of different priorities. By default, the defined priorities are:

Emergency: Every 10 minutes.
Urgent: Every 2 hours.
Warning: Once a day.

5.1 Directories and caller scripts

By default, alarms are searched in $picasrc/alarms, but you can specify an absolute path to override this. They get installed in $picaalarms/priority in the remote machine. Because one can pass arbitrary (yes, we love that word) parameters to the alarm, it's not called directly, but through a script, called ``alarmname-picacaller''. This also lets us use scripts not designed originally for PICA. The only thing one needs is to define the calling convention (it's one of the alarm optional attributes). In its definition we can take advantage of the many features of the preprocessor, as variable substitution or Perl on-the-fly code generation. It describes the parameters the alarm will be called with. For example, we could define an alarm like this:

alarm DNSChkUrgent {
   priority = 'Urgent';
   source = 'DNSChk';
   perms = '755';
   calling-convention = '-w 1 -c $threshold';

   vars {
      threshold = 8;
   }
}

Supposing $picaalarms points to /var/lib/pica/alarms, DNSChkUrgent would get installed in

-rwxr-xr-x zoso zoso /var/lib/pica/alarms/Urgent/DNSChkUrgent

(note that we haven't specified any uid or gid) and the contents of DNSChkUrgent-picacaller would be this:

#!/bin/sh
/var/lib/pica/alarms/Urgent/DNSChkUrgent -w 1 -c 8 "$@"

Cool, uh? The last "$@" lets you add more parameters if you want to call it by hand.

5.2 The scheduler

We have a ``scheduler'' script that is run periodically from crond. The cron file is looks more or less like this (I probably won't bother changing this every time I change that file):

## 
## PIFIA crontab entry
## From this crontab entry we will run the PIFIA scheduler periodically for
## each priority
## 

# Emergency (every 10 minutes)
*/10 * * * * root <#$picabin#>/scheduler Emergency

# Urgent (every 2 hours)
15 */2 * * * root <#$picabin#>/scheduler Urgent

# Warning (once a day)
20 1 * * * root <#$picabin#>/scheduler Warning

Cron runs the scheduler for every priority, and the scheduler runs every alarm in a given priority, using the -picacaller file, gets every alarm output and generates a report that is sent via e-mail (or whatever command you specify). Right now each alarm has to manage it's own variable persistency. All of this will be solved as the time goes by, by the PIFIA lib.

5.3 The PIFIA Perl package

By now you should have realized that we like Perl and we have used it throughout PICA. Well, for better integration with the program, the alarms are also written in Perl, of course, and the facilities are a bunch of Perl subs in a package. As you have probably guessed, the package is pifia, and is distributed with PICA (no recursive dependency here, no alarms are used to install a file).

The most important feature of the alarms package is the environment perseval (persistent eval, it's not that we like Arturic legends and we can't write the name of the knight of Galles). It looks like this:

perseval {
   if ($lastnotify - time() > $timebetweennots &&
         some_other_condition()) {
      print "Heck, some other condition, and I haven't notified you since ",
            "$lastnotify\n";
   }

   @down = check_hosts();
   foreach my $host (@down) {
      print "New host down: $host" unless grep { $_ eq $host }
                                               @downhosts;
   }

   $attrs{'something'} = calculate_something();
} preserve '$lastnotify', '@downhosts', '%attrs';

As you've probably noticed here, the perseval environment lets you write code with persistent variables. Those are specified in the preserve clause, after the program. Run the program, check whatever you want, and the variables you specify will have the same value the had the last time you ran the program. Really useful. All of this is implemented with the MLDBM package, so you need it installed in your system for this this to work.

6 PICA inner workings

PICA is built up from several components. There is a central executable, pica, a Perl preprocessor, and an alarm manager and Perl library (PIFIA). The central executable reads the supplied command line arguments and does the job; the Perl preprocessor is used for every configuration and non-verbatim distribution file before processing (obviously); and the alarm manager is used to setup and install alarms, and comes with a handy Perl library.

6.1 Variables and namespaces

As we already said when describing the configuration files, the PICA central executable begins by preprocessing and reading pica.conf. After that, hosts.conf is parsed. Here, the program finds out about the machines, their attributes, variables and such. Then, and once for each host, PICA preprocess and reads objects.conf. Obviously, as the file is preprocessed once for each machine, the result can be different, because the preprocessor namespace is, in general, distinct. That way, we can define different distribution files, or in different ways, for each machine. Thus, we can:

Make the host definition depend on command line definitions,
Make the objects definition depend on command line and host variables and
Make the contents of the objects depend on almost everything

Understanding this is very important to understand PICA behaviour. Once you get the hang of it, you will know how to make PICA do what you want.

6.2 PIFIA

PIFIA is a collection of files and conventions to get your servers look at themselves and act if they see something wrong (trying to fix it, telling you, or both). To make this easy to handle, there is a file to be included in your objects.conf file (with #include <pifia.conf>, yes). A pifia.conf file looks like this (yes, usual disclaimers about content updating apply):

###############################################
# PIFIA (PICA Framework for Integrated Alarms #
###############################################
group pifia {
   # Scheduler executable
   file scheduler {
      path = '<#$picabin#>/scheduler';
      source = 'alarms/scheduler';
      perms = '755';
      vars {
         # Where to send mail notifications
  notifymail = 'kuko@ulpgc.es,zoso@ulpgc.es';
  # How to send mail notifications
         mailcmd = '/usr/sbin/sendmail $notifymail';
         # Where to send pager (or sms) notifications
  notifypager = 'kukom@airtel.net';
  # How to send pager notifications
         pagercmd = '/usr/sbin/sendmail $notifypager';
         #pagercmd = 'cat';
      }
   }
   # Cron file
   file pifia.cron {
      path = '/etc/cron.d/pifia.cron';
      source = 'alarms/pifia.cron';
      perms = '644';
   }
   # PIFIA lib (Perl package)
   file pifia.pm {
      path = '/usr/local/lib/site_perl/pifia.pm';
      source = 'alarms/pifia.pm';
      perms = '644';
   }
   # README file (forces the creation of the persitence files dir, so leave it
   # here)
   file README {
      path = '<#$picaalarms#>/persistence/README';
      source = 'alarms/README';
      perms = '644';
   }
}

7 Real life examples

Here are some real life examples of PICA distribution files explained. With PICA comes a tiny (not-so-real-life) example which will make you understand how the Perl PreProcessor can be used. The file is called pica-rules.

7.1 Two files from the same source

Let's begin with an example for distributing two files from the same source. At the Network Division we run the University's primary nameservers. But we also give support to people that run secondary servers. In one of our primary servers, we also wanted to publish the configuration file needed to run a secondary server. We wanted to use the same source to keep both files consistent. This apparently simple task has created a lot of problems with other tools we have used. With PICA the solution is simple:

In the case of the masters, we have to install one version for production (the master server file) and one for documentation (the slave DNS server file). Thus, we defined, in hosts.conf, the following:

hostgroup dnsservers {
    members { fobos, deimos, mercurio, ulpnet, ulpnet2 }
}
hostgroup dnsmaster {
    members { ulpnet, ulpnet2 }
}
hostgroup doc {
    members { ulpnet, ulpnet2 }
}

This way we can differentiate between documentation, DNS master and DNS slave servers. Thus, in objects.conf, we can define the relevant files as:

#if (ingroup('dnsservers'))
group DNS {
    file named.conf {
         path = '/etc/named.conf';
         source = "DNS/named_conf.cfg";
    }

    ## Documentation for this Service
#  if (ingroup('doc'))
    # ...
    file named.conf.sample {
       path = '<#$docdir#>/Servicios/DNS/named.conf.sample';
       source = 'DNS/named_conf.cfg';
    }
#  fi
}
#fi

The only thing left to do is defining the named.conf file contents. The trick is checking the name of the object: if it ends with ``.sample'', it's a slave sample file; it not, then it's a master file.

# ...
zone "ulpgc.es" {
#if ((ingroup('dnsmaster')) && ($picaobject !~ /\.sample$/))
   type master;
   file "mydb.db";
   also-notify {
      # ...
   };
#else
   type slave;
   file "mydb.db.bak";
   masters {
      # ...
   };
#fi
};
# ...

This shows the power of using arbitrary perl code in PICA conditions.

7.2 Installing RSA authentication files within PICA

When you start administering a new server with PICA, one of the first things you should do is configuring SSH's RSA authentication, to be able to access that server without typing any password.

This task can be simplified by distributing the needed files using PICA. We use SSHv2, so we will assume this version of SSH. First of all, every sysadmin needs to have their private/public key pair. Let's say we are two sysadmins and our public keys are in SSHv2 format in the files sysadm1.pub and sysadm2.pub. We will add the following entries to the objects.conf file:

# SSH RSA authentication files
group RSAAuth {
    # SSHv2 authorization file
    file ssh_auth {
         path = '/root/.ssh2/authorization';
         source = "SSH/authorization.cfg";
    }
    file sysadm1.pub {
         path = '/root/.ssh2/sysadm1.pub';
         source = "SSH/sysadm1.pub";
    }
    file sysadm2.pub {
         path = '/root/.ssh2/sysadm2.pub';
         source = "SSH/sysadm2.pub";
    }
}

Different versions of SSH (SSHv2 or SSHv1) can be used in different hosts and use conditionals in the previous entries. This is left as an exercise to the challenged student ;-).

We could even generate the authorization file on-the-fly with the needed Key entries with the following code snippet:

#perl
my @return;
# Get key files reading group members and skipping 'ssh_auth'
my @keys=grep(/\.pub$/,members('SSHAuth'));
foreach my $key (@keys) {
  push @return,"Key $key\n";
}
# Return the array (will be printed)
@return;
#lrep

This code will generate one ``Key file.pub'' entry for each public key file we define in the group, thus allowing access to the server with that key. This is really outside the scope of this article, but is a good example of what can be done with the #perl/#lrep environment.

With this configuration, after adding the new host to the hosts.conf file you could run the command:

pica -iv +F SSHAuth +H new_server

You will then have to type the server's password only this time, because after installing this files both sysadmins will be able to access the server without typing any password (assuming they are running ssh-agent).

7.3 Execution of perl code within configuration files

Here at Network Division, we have a very handy script to restart critical services via Web, just in case SSH is down on a given server. Of course, it has a secret password to allow access only to authorized administrators.

Before PICA, we had to manually create the password's MD5 hash using the crypt perl function, and include the hash in the source file. Now, we can resort to something as elegant as this:

$passwd='<# crypt('secret-key-that-you-wont-ever-figure-out','salt') #>';

This way, just before sending the data, the command is executed and the encrypted key is put automagically in the distributed file. This saves time and work, and is a perfect example of the inmense power of PICA (and Perl, of course).

1: <kuko@ulpgc.es>
2: <zoso@demiurgo.org>
3: you might ask why didn't we put them in pica.conf, as the rest of the directories. Well, the reason is, the directories in pica.conf are local directories.
4: All of this, of course, suppossing that dnsservers is a group containing ``fobos mercurio deimos'', solaris contains ``sar'' and deimos is a host, not a group.
5: Keep It Simple, Stupid

This document was translated from L^AT_EX by H^EV^EA.