Package Management for System Administration

The maintenance and deployment of Unix system configuration presents issues of complexity that can scale much faster than the number of systems to be managed. It is common practice for the tasks of standardizing of Unix servers to be performed using custom scripts or expensive and complex products like BladeLogic.

•	Localization
•	Site-specific configuration files (/etc/hosts, nsswitch.conf, etc.)
•	3rd Party software and/or associated OS dependencies
•	Custom security settings, permissions, and disclaimers

An alternative is to use the same tools used to actually provision the system to provision the configuration as well.

•	Versioning: The ability to quickly determine a server's deployed state, upgrade, or even roll-back changes.
•	Consistency: Text formatting, file location, and other system-to-system variations are eliminated.
•	Integration: Using the existing Package Management methods allows for increased flexibility without buying expensive tools, installing non-native agents, or replacing system supplied binaries.
•	Efficiency: Fast Provisioning, using OS native methods.
•	Validation: The ability to detect changes in a system using the package validation process. Additionally, package validation code is deployed along with the package it will validate in a much more logical, "object-oriented" manner.
•	Security: No changes to existing security requirements. (Access already in place for deployment, can also be used for management)

This paper targets several specific examples that are designed to cover the needs of system administrators for the purpose of configuration management. It is not intended as a all-encompassing paper on packaging, nor is it intended beyond the scope of the system administration crowd. This limited scope was a self imposed requirement that facilitated getting this paper published in this lifetime. Both formats have an extensive feature set that would each be too much to write on without this restricted view. The goal is to write a paper that both provides a unique and meaningful discussion while providing a simplified subset for the target audience.

The examples that follow were created for installp format on AIX and RPM format on Red Hat Linux. As each major step of the example is described, there will be an opportunity to jump to (and back from) package format specific content. The package specific content begins with the following links here in the introduction.

The first example moves straight to the heart of configuration management by addressing issues associated with managing config files via the package manager. This example will deal with two sets of issues. The first is the obvious mechanics of deploying config files in packages. The second is the less obvious issues when using a package manager in this manner.

A more realistic configuration would be more location specific and potentially handle more configuration options. This example will forgo the complexity of these options to focus on the basic concepts that will enable a package manager as a configuration distribution and validation mechanism. With the foundation provided here, the reader should have sufficient tools to expand as necessary.

The package manager will need to install a configuration file as part of its file manifest as a non-configuration file. Normally, packages have a special option for configuration files that acknowledges that they may be changed and need not be checked with the same rigidity as a binary. Here that must be ignored so that the package manager will acknowledge all changes to these files.

Because the package manager will be used as an enforcement tool for configuration file versioning, it must be able to validate files that it has placed on the system. This means that the files it places must be tracked in the package manifest (typically via a checksum). If the file has changed, then the package validation should fail.

Next, the package must contend with issues stemming from the overwriting of files belonging to other packages. Any problems associated with dual ownership must be acknowledged and handled.

Also, the package must not serve as a security hole for the system. Any configuration files written must maintain the proper ownership and permissions. It would be unfortunate if important configuration files were written to the system that could be modified by regular users.

Finally, package manager operations must never leave the system in an unstable state. If a configuration package is removed it must restore a working copy of the configuration file. Typically it should never be an issue to exist momentarily without configuration files during a package upgrade, but should a package be permanently removed from the system, it should not leave the system in an undefined state.

Each packaging tool requires that content and package config files be staged in a specific directory structure. The examples here were all automated via script that pulled the content from project specific directories. This staging process carries with it some of the baggage of the project / paper that it was written for. The desire is to relate the layout and all layout options without getting too hung up on the specific method used for this paper.

Ideally, content will live in a project directory that the package build process can run from rather than relying upon complex scripting to pull in each component to set up a build environment each time a package is created. But regardless of the method used, the goal is to make the content setup as manageable and simple as possible.

Each of the tool-specific break-out sections will relate the layout required for the build, and some subtle variation options on that layout. From this, the intent is for the reader to create a method that works best for them.

The package manifest is a package tool specific meta-info repository for the package that will be created. While the syntax and some basic design varies significantly between the two package managers, the underlying concepts do not. Each package manifest must encapsulate the following information:

•	Package information such as name, version, (short) description, license, etc...
•	All requisites for this package (other content this package requires to run)
•	A list of all installable files in the package
•	Active / scripting content that will run during various stages of the package life-cycle (before install, after install, during upgrade, on removal, etc...)

Each of the packages maintains features beyond this subset. The reduced list here is what will be required for the projects in this paper. The first example will utilize all the above parts except for the requisite package section.

In each example the goal is to create a complete and well-structured layout of the content and manifest. With a proper focus on the first steps, the result is that the build process will tend to be a non-event.

The build process will need two pieces of information; the location of the staged content and the location of the manifest file. From this, it will create the package that can be distributed to other systems. Each package toolset is slightly different, but follows the same concepts.

The focus of this paper is managing configurations via a package manager. This example will appear somewhat tangential in that it will look more like a traditional package - yet targeted more to the content of administrators.* While it will be a very simple example it includes some additional requirements placed by the ACME corporation on IP and liability concerns associated with the ACME ToolKit.

[* Each packaging tool leverages a rich set of capabilities associated with the building and precise deployment of software. The example here will contend more with a simplistic set of files and focus in on a fundamental core set of package features required for these tasks.]

This package will need to deploy a ToolKit consisting of several binaries and associated configuration files. The goal of this example is to define the most basic distribution method for some simple system tools (scripts) written by internal IT staff.

Content for this package is a set of three binaries and two (default) config files for those binaries. The binaries will be installed to /usr/bin but the config files will be installed to a special sub-directory within /etc. This is done to present the issues associated with directory creation in a package. (Most people would probably be inclined to simply put a set of files this small directly into /etc.)

The inclusion of man pages in a ToolKit of this nature is highly recommended. They are not included here only as a means to simplify the contents of the package.

Additionally, the choice was made to install directly to /usr/bin. It may be more appropriate to install to a ToolKit specific bin directory and adjust user paths or generate links* to access these utilities. There are a number of different options for this type of installation - here the most straightforward method is addressed.

[* As mentioned, this example does not create links. But if links were required, they would be added as expected (based upon the underlying tools used to capture the files in the package). Meaning; each package type is aware of, and reads hard and soft links the same as files - each will properly package the links and not the referenced files.]

Like a more traditional package, this manifest will be more about the installable file content than other potential package tool features such as active / scripting content. The resulting package manifest should look more like a minimalist configuration for software deployment.

The installp version will leverage a license capability that does not have an equivalent in the RPM format. The original goal was to make this license acceptance a requirement in both tool-specific examples. Because the RPM version would require a more interactive install the "requirement" has been reduced to just the installp example.

Once again, a properly formatted manifest and staged set of files should produce a clean package build. This section will introduce a few methods to pull the package back apart and retrieve individual files as a means to debug build problems.

Rather than maintain a monolithic package, it is much easier to break a distribution up into multiple components that can be maintained separately in a "topdown" manner - possibly even by different people. This allows for:

•	Simplified individual package design
•	Exclusion of some packages on one-off systems
•	Delegation of projects to other groups where efforts are compartmentalized into individual packages
•	The ability to have multiple location specific versions of some packages
•	The ability to add and debug additional packaging content without testing all the other components
•	The ability to group sub-packages for different personalities of systems that are built

When deploying a group of packages it is much easier to define a "personality" package that makes all relevant packages a dependency. This way the package management (install) tool can resolve all dependencies when the user picks a single personality package to install.

The obvious design issue is for this example is to have one package install result in the installation of a group of packages. The less obvious challenge is to test the resolution of package dependencies in the install environment.

The point of the "group" package is to develop a package that has dependencies on all packages that make up its "personality"* and to place a flag file in the root directory that will be leveraged by some of the ACME admin tools.

[* This example is a bit bland to show "personality". The point is that the package could be defined for "all production servers in the west coast data center" or "all development systems" and have a relevant naming convention. This is a simplistic example that these concepts could be built on.]

The example here will not go (deep) into dependency resolution and repository creation but instead, focus only on package creation and minimal installation debugging. Once again there are numerous, and complete sources on the Internet for repository creation / management.

As there is virtually no content for the group package (only a flag file that admins can use for quick inspection of the system) the staging is quite trivial. It is effectively an absolute minimum of files to stage for the build. The real staging of content is all done post-build in the package repository. For this reason the real challenge in debugging the package is external to the build process, and more in the repository building and deployment processes.

The real complexity of the build process for this type of package is all in the dependencies section of the manifest. All other parts of the manifest will be relatively simple with a focus on getting proper package names and version synchronization.

installp formatted package files will allow for grouping of related filesets*. While this could be leveraged in a grouping like this, it will not be covered until example four.

Because the package tools will be unable to determine at build time if a requisite package is legitimate, the build will most likely succeed if there is an error in the package name or version. A package of this type can really only be validated within the package repository / deployment infrastructure. The tool-specific sections will have some limited examples for testing the group package on a target system.

Some configuration changes are not pushed as simple files but as commands to be run or changes to complex files. Some examples are running a command, adding a user, modifying a file, or changing a system option. In each case we need to account for capabilities that are clearly in the packaging tools, but somewhat out of the norm of simply dropping a file on the system and removing it when the package is de-installed.

Each package version will be different in this example. installp will make ODM changes and deal with managing those changes through each version of the package installation. rpm will make a service setting change.

Each section will detail the specific challenges, but the most basic challenge is confining the change to the package. This means that the package should make the change, and change it back on removal. Furthermore, multiple applications of the file should not cause multiple (repeated & redundant) changes to the system.

That said, some packages should not completely reverse the changes when removed. For example, a package that creates a user / service account should not necessarily delete the home directory for the account when it is removed. This will vary by package content type and exceptions may need to be made for some types of content.

Virtually all the content in this example will be non-"file" content, but will be active scripting. For this reason, the staging will look different than the other package examples. The "traditional" file content will be minimalist like the group example, and the active content will be more involved like the first example.

A primary concern of a package of this nature is not just the inclusion of scripting content, but inclusion at the correct stage of the install process and balancing the do of the install with the undo of the package removal. The orchestration of this complexity is (typically) handled through specifications in the manifest file.

As with the grouping / dependencies in the previous example, the real test of this package is in the installation. The build process will not be able to properly validate the active / script content of the package.

Writing "robust" and "defensive" scripts will be important. Scripts written for this environment should not make assumptions about files, services, or whatever it will be modifying. This assumption begins with not assuming that the item even exists. (Packages should not take the environment for granted - they should have dependencies and test for specifics before they act. This will serve to reduce the impact of errors in deployment.) Additionally, scripts should capture stdout, stderr, and return values (such as in an "if" clause) and handle adverse results appropriately. While it may be ideal to develop and test a script as a regular user on a system, the environment provided to the user may be very different than what is provided during the install process.

This project encompassed multiple systems, and multiple system types at multiple locations. The reflexive impulse was to write a script to stage the build directory so that a pristine source could be documented and built from. The problem with this method is the complex and unnecessary nature of the staging process. I have tried to make note of different staging options in the paper, while not dwelling on the imperfect mode I used specific to this paper. The point to read from this is that a permanent stage location for each project may be more appropriate - or just that my one method is not the sole answer to this problem.