6.3 Different Types of Installation Management

As discussed in the introduction, setup and management of clusters can come down to a personal style choice. However, there are technical and practical reasons to choose one type of installation method over another. We will describe the tradeoffs of each type of installation. Popular open-source management systems include: NPACI Rocks, OSCAR, SCore, Scyld, and XCAT. The Linux space is enormous and cluster toolkit designers have to make some decisions on what types of generalizations are the most important. In particular, authors have to determine if and how they will scale across distributions and how they will support different kinds of hardware. None of these toolkits covers the complete space of distributions ? hardware.

The Linux distribution space is a moving target and includes (as of this writing) a number of players: RedHat, SuSE, Mandrake, Debian, United Linux, and more. Each distribution has it's own style, file layout, package format, package definitions, and support for a variety of hardware. Small differences such as using shadow passwords (or not), using xinetd or inetd, and SysV-compatible startup differentiate distributions. For cluster software service designers (e.g., PBS, MPICH, Sun Grid Engine, Ganglia), differences in packaging definitions can cause more headaches, especially in resolving dependencies. Redhat packages, for example, are put together differently than similarly named SuSE packages. Debian uses dpkg and apt-get. For the cluster toolkit designer, supporting all these variations is an impossibly large task. As a would-be cluster builder, you have to choose the distribution and the toolkit. Do pick a toolkit as a starting point for your cluster and become involved in its improvement. Don't roll your own installation and management from the ground up, as this just becomes a waste of your time in retreading old ground. You have more important things to do with your time—using your new cluster.

Each of the commercial Linux distributions must do hardware detection to automatically install the right device drivers on the widest variety of hardware possible. Distributions that don't detect the most common hardware (and support the quickly changing network device world), simply are discarded by the user community in a classic case of "survival of the fittest." This last point is of significant importance to cluster builders because of the desire to use the best, fastest, newest, and cheapest. No judgment is made here as to which distribution is superior—rather it is the fact that these differences exist that is important.

The hardware space gets more complicated each day. Beowulf clusters used to be only IDE disks. Today, IDE (EIDE, UltraATA, Serial ATA), SCSI, Integrated Drive Arrays, and Storage Area Network (SAN) adapters are de rigeur for cluster builders. High-speed interconnects including Scali, Myrinet, Quadrics, and Infiniband need to be supported by cluster toolkits. Motherboards, specialized chipsets, variants of x86, Itanium, and Opteron add more to the hardware mix. Again, none of the toolkits covers all of this hardware space. You may asking yourself, "What's the problem? Why don't the cluster toolkits support all of these?". It comes down to practical issues of time, money and resource. Each of the cluster toolkits tests a release on as much hardware as possible—that collection of hardware is simply too small to cover most cases that users see in the field. The toolkit designers and implementers must decide on "which disributions" and "what hardware" is supported. Another way to ask the question is

Should the toolkit scale across Linux distributions? or Should the toolkit scale across hardware?

Scaling across both, the obvious answer, is simply not something that the toolkit designers are able to practically accomplish.

The tradeoffs are simple—if you choose to scale across (support) multiple distributions, then one is practically forced to make some generalizations to fit all the various distributions. The generalization uniformly selected by toolkits that scale across distributions is to take over the base installation and hardware detection piece from the vendors and make these core pieces of the cluster toolkit. These approaches don't leverage the installer supplied by the distribution. Instead, they build their own customized installation programs that can handle a number of different distributions. The advantage is that users have more choice of specific distributions. The drawback here is that the hardware space is large, and it is quite a job to manage the hardware detection and customized kernel modules for lots of hardware.

If you choose a single distribution, then you can leverage the installation and hardware detection of the commericial vendor and worry only about extensions to specific pieces of hardware, such as Myrinet. The clear drawback is that if the distribution does a poor job of this, then the toolkit suffers the same fate.

What we see in the cluster toolkit space is a dichotomy of approaches—disk imaging and description methods. Disk imaging was described in the first edition of this book as the practical way to clone a system onto your cluster nodes. Commercial and open-source tools image-based programs are popular and include: Norton Ghost, PowerQuest Drive Image, SystemImager, Chiba City Imager, and Power-Cockpit. Complete clustering toolkits that use imaging include OSCAR, Chiba City, and CLIC. Description-based installers, on the other hand, use text files to provide a list of packages and instructions needed to completely configure a node. Text-based installers are distribution-specific: Redhat Kickstart, SuSE YaST, and Debian Fully Automatic Installation (FAI). Most of have their genesis (or, at least, inspiration) from Sun's Jumpstart installer. The text-based installation captures disk partitioning, package listing, and software configuration. The advantage here is that a reasonably general text description can work on many different variants of hardware because the distribution's installer is handling all the low-level details of hardware detection. One popular description-based cluster toolkit is NPACI Rocks. But other examples exist like IBMs partially-open source XCAT, and the European Data Grid's LCFG.