1.4 Building and Using a Cluster

1.4.1 Choosing a Cluster

When choosing the components for a cluster, or selecting from a prebuilt cluster, you must focus on the applications that will be run on the cluster. The following list covers some of the issues to consider.

1. Understanding the needs of your application. Some of this has been covered above; you will find more on understanding the performance of applications in Part II.

2. Decide the number and type of nodes. Based on the application needs, select a node type (e.g., uni-processor or SMP), processor type, and memory system. Chapter 2 covers node hardware. As described above, raw CPU clock rate is not always a good guide to performance, so make sure that you have a good understanding of the applications. Other issues to consider when choosing the processor type include whether you will run prebuilt applications that require a particular type of processor, whether you need 64-bit or 32-bit addressing, or whether your codes are integer or floating-point intensive.

3. Decide on the network. Determine whether your applications require low latency and/or high bandwidth in the network. If not, for example, running a throughput cluster with embarassingly parallel applications, then simple fast Ethernet with low-cost switches may be adequate. Otherwise, you may need to invest in a high-performance cluster network. These network choices are covered in more detail in Chapter 4. Note that the cost of a fast Ethernet network is very low while a high performance network can double the cost of a cluster.

4. Determine the physical infrastructure needs. How much floor space, power, and cooling will you need. Is noise a factor?

5. Determine the operating system (OS) that you will use. Since you bought this book, you have probably selected Linux. Chapter 3 will help you select a particular distribution of Linux as well as understand how to tune Linux for your cluster. The choice of cluster setup software may also influence which distribution of Linux you can use; this is covered in Chapter 6. In choosing the operating system, consider the following:

   • Do your applications run on the chosen system? Many applications and programming models (Part II) run under many operating systems, including Windows, Linux, and other forms of Unix.

   • Do you have expertise with a particular operating system?

   • Are there license issues (cost of acquiring or leasing) software, including the operating system and compilers?

6. Cost tradeoffs. The cost of a node is not linearly related to the performance of that node. The fastest nodes are more expensive per flop (and usually per MByte/sec of memory bandwidth) than are lower-cost nodes. The question is then: Should a cluster use the fastest available nodes regardless of cost, or should it use mid-range or even low-range nodes? The answer depends, as always, on your needs:

   • If price is no object, go with the fastest nodes. This approach will reduce the number of nodes needed for any given amount of computing power, and thus the amount of parallel overhead.

   • If total computing power over time is the goal, then go with mid- or low-end nodes, but replace them frequently (say, every 18 months to two years) with newer nodes. This strategy exploits the rapid advances in node performance; buying two low-end nodes every two years will often give you a greater amount of computing power (integrated over time) than spending the same amount every four years on a high-end node.

   • If a targeted amount of computing power (e.g., for a specific application) is the goal, then analyze the tradeoffs between a greater number of slower (but possibly much cheaper) nodes and a smaller number of faster but individually less cost-efficient nodes.

1.4.2 Setting Up a Cluster

Once you have specified your cluster, you need to assemble the components and set up the initial software. Chapters 2 and 4 cover some of the issues in assembling the hardware for both the nodes and the network. Chapter 20 discusses cluster setup in the context of two generations of clusters used at Argonne National Laboratory.

In the past few years, great strides have been made in simplifying the process of initializing the software environment on a cluster. Chapter 6 covers the most popular packages and provides advice on setting up your new cluster.

At this point, you may wish to benchmark your cluster. Since such benchmarking will require running a parallel program, information on this topic is provided in Part II, Sections 9.10 and 9.10.3. Alternatively, you may prefer to run a prepackaged performance suite, such as the Beowulf Performance Suite (BPS), available at www.plogic.com/bps. BPS contains both single node and parallel performance tests, including the following:

• bonnie++: I/O (disk) performance; www.coker.com.au/bonnie++

• Stream: Memory system performance; www.cs.virginia.edu/stream

• netperf: General network performance; www.netperf.org/netperf/NetperfPage.html

• netpipe: A more detailed network performance benchmark; www.scl.ameslab.gov/Projects/ClusterCookbook/nprun.html

• unixbench: General Unix benchmarks; www.linuxdoc.org/HOWTO/Benchmarking-HOWTO.html

• LMbench: Low-level benchmarks; www.bitmover.com/lmbench

• NAS parallel benchmarks: A suite of parallel benchmarks derived from some important applications; www.nas.nasa.gov/Software/NPB

1.4.3 Developing New Applications

Before deciding to develop new applications, check out what is already available for clusters. Chapter 12 provides a guide to software that is already available, either as full programs or as libraries that can be used to build new applications with little or no explicitly parallel programming. New applications are constantly being developed, so check the web and the Beowulf mailing lists before starting to develop your own application. If what you need is not yet available, then Part II provides an introduction to parallel programming, covering both the most popular tools for building embarrassingly parallel applications as well as the two most popular libraries for parallel programming, MPI (Chapters 8 and 9) and PVM (Chapters 10 and 11). These chapters also contain information on tuning and testing applications. Section 9.10 covers the most common cluster performance benchmarks, including High Performance LINPACK (Section 9.10.3).

1.4.4 Operating a Cluster

Once your cluster is up and running, you (or someone) will need to operate it. Chapter 13 covers the basics of system management for clusters, including account administration, security, monitoring, and file system backups. Chapter 19 discusses parallel I/O in general and the systems administration issues for the Parallel Virtual File System (PVFS) in particular.

One of the biggest decisions is whether the cluster will allow provide interactive use, batch use, or a mixture of both. Since batch use is a common way to use clusters, Chapter 14 provides an overview of the issues and many of the available batch systems. This chapter is followed by chapters on individual batch systems.

1.4.5 Getting Help

Many resources are available to which you can turn for help. One of the best is the Beowulf mailing list. To subscribe, visit www.beowulf.org/mailman/listinfo/beowulf. The Beowulf Web site, www.beowulf.org, also provides much valuable information.

A major strength of Linux is the community. The Beowulf mailing list (<beowulf@beowulf.org>) is a good place to go for answers to all kinds of questions.