Perl for System Administration: Managing multi-platform environments with Perl

Perl for System Administration: Managing multi-platform environments with Perl

by David N. Blank-Edelman

Some people plan to become administrators. The rest of us are thrust into it: we are webmasters, hobbyists, or just the default "technical people" on staff who are expected to keep things running. After some stumbling around repeating the same steps over and over again (and occasionally paying the price when we forget one), we realize that we must automate


Some people plan to become administrators. The rest of us are thrust into it: we are webmasters, hobbyists, or just the default "technical people" on staff who are expected to keep things running. After some stumbling around repeating the same steps over and over again (and occasionally paying the price when we forget one), we realize that we must automate these tasks, or suffer endless frustration. Thus enters Perl.The Perl programming language is ideal for writing quick yet powerful scripts that automate many administrative tasks. It's modular, it's powerful, and it's perfect for managing systems and services on many platforms.Perl for System Administration is designed for all levels of administrators—from hobbyists to card-carrying SAGE members—sysadmins on multi-platform sites. Written for several different platforms (Unix, Windows NT, and Mac OS), it's a guide to the pockets of administration where Perl can be most useful for sites large and small, including:

  • Filesystem management
  • User administration with a dash of XML
  • DNS and other network name services
  • Database administration using DBI and ODBC
  • Directory services and frameworks like LDAP and ADSI
  • Using email for system administration
  • Working with log files of all kinds
Each chapter concentrates on a single administrative area, discusses the possible pitfalls, and then shows how Perl comes to the rescue. Along the way we encounter interesting Perl features and tricks, with many extended examples and complete programs. The scripts included in the book can simply be used as written or with minimal adaptation. But it's likely that readers will also get a taste of what Perl can do, and start extending those scripts for tasks that we haven't dreamed of.Perl for System Adminstration doesn't attempt to teach the Perl language, but it is an excellent introduction to the power and flexibility of Perl, and it whets the appetite to learn more. It's for anyone who needs to use Perl for system administration and needs to hit the ground running.

Editorial Reviews

Blank-Edelman (computer science, Northeastern University) has served as system/network administrator for several multiplatform environments in the Boston area. This readable, self-help type book about Perl language and its uses is directed at non-specialists who know some Perl and work in system administration. Five tutorials, in RCS, LDAP, XML, SQL, and SNMP, are included. The paper binding is designed to open flat. Annotation c. Book News, Inc., Portland, OR (
Internet Book Watch
Perl For System Administration is designed for all levels of administrators, from novice single-box Linux users to seasoned SAGE members. While covering several different platforms including Unix, Windows NT, and Mac OS, Perl For System Administration also covers such core issues as filesystem management, user administration, directory services, database administration, log files, security, and network monitoring. Superbly written and presented, Perl For System Administration will enable the beginner to launch confidently into Perl for administrative tasks, and has much to offer even the more experienced manager.
—Internet Book Watch

Product Details

O'Reilly Media, Incorporated
Publication date:
O'Reilly PERL Series
Product dimensions:
7.06(w) x 9.22(h) x 0.90(d)

Read an Excerpt

Chapter 9: Log Files

If this weren't a book on systems administration, an entire chapter on log files would seem very peculiar indeed. But systems administrators have a very special relationship with log files. Like Doctor Doolittle, the peculiar character from children's literature who could talk to the animals, systems administrators are expected to be able communicate with a large menagerie of software and hardware. Much of this communication takes place through log files, so we become log file linguists. Perl was born to deal with log files, so it can be a big help in this process.

It is impossible to touch on all of the different kinds processing and analysis one can do with logs. Entire books have been devoted to just statistical analysis of this sort of data. However this chapter should give you some general approaches to the topic and Perl tools to whet your appetite for more.

Text Logs

Logs come in different flavors, and so we need several approaches for dealing with them. The most common type of log file is one composed entirely of lines of text. Popular server packages like Apache (web), INN (Usenet news), and Sendmail (e-mail) spew log text in voluminous quantities. Most logs on UNIX machines look similar because they are created by a centralized logging facility known as syslog. For our purposes, we can treat files created by syslog like any other text file.

Here's a very simple Perl program to scan for the word `error' in a text-based log file:

open(LOG,"logfile") or die "Unable to open logfile:$!\n"; while(<LOG>){        print if /\berror\b/i; } close(LOG);

Perl-savvy readers are probably itching to turn it into a one-liner. For those folks:

perl -ne 'print if /\berror\b/i' logfile

Binary Log Files

Sometimes you don't get off that easy writing programs to deal with log files. Instead of nice, easily parseable text lines, some logging mechanisms produce nasty, gnarly binary files with proprietary formats that can't be parsed with a single line of Perl. Luckily, Perl isn't afraid of these miscreants. Let's look at a few approaches we can take when dealing with these files. We're going to look at two different examples of binary logs: UNIX's wtmp file and NT/2000's event logs.

Back in Chapter 3, User Accounts, we touched briefly on the notion of logging in and logging out of a UNIX host. Login and logout activity is tracked in a file called wtmp on most UNIX variants. It is common to check this file whenever there is a question about a user's connection habits (e.g. from what hosts does this person usually log in from?).

On NT/2000, the event logs play a more generalized role. They are used as a central clearinghouse for logging practically all activity that takes place on these machines including login/logout activity, OS messages, security events and etc. Their role is most closely analogous to the UNIX syslog service we mentioned earlier.

Using unpack( )

Perl has a function called unpack( ) especially designed to parse binary and structured data. Let's take a look at how we might use it to deal with the wtmp files. The format of wtmp differs from UNIX variant to UNIX variant. For this specific example we'll look at the wtmp files found on SunOS 4.1.4 and Digital UNIX 4.0 because they are pretty simple. Here's a plain text translation of the first three records in a SunOS 4.1.4 wtmp file:

0000000    ~  \0  \0  \0  \0  \0  \0  \0   r   e   b   o   o   t  \0  \0 0000020   \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 0000040    ,   /   ;   4   c   o   n   s   o   l   e  \0   r   o   o   t 0000060   \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 0000100   \0  \0  \0  \0   ,   /   ; 203   c   o   n   s   o   l   e  \0 0000120   \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0 0000140   \0  \0  \0  \0  \0  \0  \0  \0   ,   /   < 230  Insert dump here

Unless you are already familiar with the structure of this file, that "ASCII dump" (as it is called) of the data looks mostly like line noise or some other kind of semi-random garbage. So how do we become acquainted with this file's structure?

The easiest way to understand the format of this file is to look at the source code for programs that read and write to it. If you are not literate in the C language, this may seem like a daunting task. Luckily we don't actually have to understand or even look at most of the source code, we can just examine the portion that defines the file format.

All of the operating system programs which read and write to the wtmp file get their file definition from a single, short C include file which likely to be found at /usr/include/utmp.h. The part of the file we need to look at begins with a definition of the C data structure that will be used to hold the information. If you search for "struct utmp {" you'll find the portion we need. The next lines after "struct utmp {" define each of the fields in this structure. These lines should be each be commented using the "/* text */" C comment convention.

Just to give you an idea of how different two versions of wtmp can be, compare the relevant excerpts from utmp.h on these two operating systems:

SunOS 4.1.4:

struct utmp {         char    ut_line[8];   /* tty name */         char    ut_name[8];   /* user id */         char    ut_host[16];  /* host name, if remote */         long    ut_time;     /* time on */ };

Digital UNIX 4.0:

struct utmp {         char    ut_user[32];  /* User login name */         char    ut_id[14];    /* /etc/inittab id- IDENT_LEN in init */         char    ut_line[32];  /* device name (console, lnxx) */         short   ut_type;      /* type of entry */         pid_t   ut_pid;       /* process id */         struct exit_status {             short e_termination; /* Process termination status */             short e_exit;        /* Process exit status */         } ut_exit;            /* The exit status of a process                                * marked as DEAD_PROCESS.                                */         time_t  ut_time;      /* time entry was made */         char    ut_host[64];  /* host name same as MAXHOSTNAMELEN */ };

These files provide all of the clues we need to compose the necessary unpack( ) statement. unpack( ) takes a data format template as its first argument. It uses this template to determine how to disassemble the (usually) binary data it receives in its second argument. unpack( ) will take apart the data as instructed, returning a list where each element of the list corresponds to an element of your template.

Let's construct our template piece-by-piece based on the C structure from the SunOS utmp.h include file above. There are many possible template letters we can use. I've translated the ones we'll use below, but you should check the pack( ) section of the perlfunc manual page for more information. Constructing these templates is not always straightforward because C compilers occasionally pad values out to satisfy alignment constraints. The command pstruct that ships with Perl can often help deal with quirks likes these...

Meet the Author

David N. Blank-Edelman is the Director of Technology at the Northeastern University College of Computer and Information Science. He has spent the last 25 years as a system/network administrator in large multi- platform environments, including Brandeis University, Cambridge Technology Group, and the MIT Media Laboratory. He was also the program chair of the LISA 2005 conference and one of the LISA 2006 Invited Talks co-chairs.

Customer Reviews

Average Review:

Write a Review

and post it to your social network


Most Helpful Customer Reviews

See all customer reviews >