Understanding Shell Scripts

Understanding Shell Scripts

Have you ever had a task that you needed to do over and over that took a lot of typing on the command line? Do you ever think to yourself, "Wow, I wish there was just one command I could type to do all this of this"? Maybe a shell script is what you're after.

Shell scripts are the equivalent of batch files in MS-DOS, and can contain long lists of commands, complex flow control, arithmetic evaluations, user-defined variables, user-defined functions, and sophisticated condition testing. Shell scripts are capable of handling everything from simple one-line commands to something as complex as starting up your Red Hat Linux system.

In fact, as you will read in this chapter, Red Hat Linux does just that. It uses shell scripts (/etc/rc.d/rc.sysint and /etc/rc) to check and mount all your filesystems, set up your consoles, configure your network, launch all your system services, and eventually provide you with your login screen. While there are nearly a dozen different shells available in Red Hat Linux, the default shell is called bash, the Bourne-Again shell.

Executing and debugging shell scripts

One of the primary advantages of shell scripts is that they can be opened in any text editor to see what they do. A big disadvantage is that shell scripts often execute more slowly than compiled programs. There are two basic ways to execute a shell script:

  • The filename is used as an argument to the shell ( as in bash myscript). In this method, the file does not need to be executable; it just contains a list of shell commands. The shell specified on the command line is used to interpret the commands in the script file. This is most common for quick, simple tasks.

  • The shell script may also have the name of the interpreter placed in the first line of the script preceeded by a #! (as in #!/bin/bash), and have its execute bit set (using chmod +x). You can then run your script just like any other program in your path simply by typing the name of the script on the command line.

    Cross-Reference?

    See Chapter 4 for more on chmod and read/write/execute permissions.

When scripts are executed in either manner, options to the program may be specified on the command line. Anything following the name of the script is referred to as a command-line argument.

As with writing any software, there is no substitute to clear and thoughtful design and lots of comments. The pound sign (#) prefaces comments and can take up an entire line or exist on the same line as script code. It's best to implement more complex shell scripts in stages, making sure the logic is sound at each step before continuing. Here are a few ways to make sure things are working as expected during testing:

  • Place an echo statement at the beginning of lines within the body of a loop. That way, rather than executing the code, you can see what will be executed without making any permanent changes.

  • To achieve the same goal, you could place dummy echo statements throughout the code. If these lines get printed, you know the correct logic branch is being taken.

  • You could use set +x near the beginning of the script to display each command that is executed or launch your scripts using sh –x myscript.

Understanding shell variables

Often within a shell script, you want to reuse certain items of information. During the course of processing the shell script, the name or number representing this information may change. To store information used by a shell script in a way that it can be easily reused, you can set variables. Variable names within shell scripts are case-sensitive and can be defined in the following manner:

NAME=value

The first part of a variable is the variable name, and the second part is the value set for that name. Variables can be assigned from constants, like text or numbers. This is useful for initializing values or saving lots of typing for long constants. Here are examples where variables are set to a string of characters (CITY) and a numeric value (PI):

CITY="Springfield"
PI=3.14159265

Variables can contain the output of a command or command sequence. You can accomplish this by either enclosing the command in backticks (`) or by enclosing the command in parentheses. This is a great way to get information that can change from computer to computer or from day to day. Here we set the output of the uname -n command to the MACHINE variable. Then we use parens to set NUM_FILES to the number of files in the current directory by piping ( | ) the output of the ls command to the word count command (wc -l).

MACHINE=`uname –n`
NUM_FILES=(/bin/ls | wc –l)

Variables can also contain the value of other variables. This is useful when you have to preserve a value that will change so you can use it later in the script. Here BALANCE is set to the value of the CurBalance variable.

BALANCE=$CurBalance
Note?

When assigning variables, use only the variable name (for example, BALANCE). When referenced, meaning you want the value of the variable, precede it with a dollar sign (as in $CurBalance).

Special Shell Variables

There are special variables that the shell assigns for you . The most commonly used are called the positional parameters or command line arguments and are referenced as $0, $1, $2, $3…$n. $0 is special and is assigned the name used to invoke your script; the remainder are assigned the values of the parameters passed on the command line . For instance, if the shell script named myscript were called as:

myscript foo bar

the positional parameter $0 would be myscript, $1 would be foo, and $2 would be bar.

Another variable, $#, tells you how many parameters your script was given. In our example, $# would be 2. Another particularly useful special shell variable is $?, which receives the exit status of the last command executed. Typically, a value of zero means everything is okay, and anything other than zero indicates an error of some kind. For a complete list of special shell variables, refer to the bash man page .

Parameter expansion in bash

As mentioned earlier, if you want the value of a variable, you precede it with a $ (for example, $CITY). This is really just a shorthand for the notation ${CITY}. Bash has special rules that allow you to expand the value of a variable in different ways. Going into all the rules is probably a little overboard for a quick introduction to shell scripts, but Table 12-1 presents some common constructs that you're likely to see in bash scripts you find on your Red Hat Linux box.

Table 12-1: Examples of bash parameter expansion

Construction

Meaning

${var:-value}

If variable is unset or empty, expand this to value

${var#pattern}

Chop the shortest match for pattern from the end of var's value

${var##pattern}

Chop the longest match for pattern from the end of var's value

${var%pattern}

Chop the shortest match for pattern from the front of var's value

${var%%pattern}

Chip the longest match for pattern from the front of var's value

Try typing the following commands from a shell to test out how parameter expansion works:

# THIS="Example"
# THIS=${THIS:-"Not Set"}
# THAT=${THAT:-"Not Set"}
# echo $THIS
Example
# echo $THAT
Not Set

In the previous examples, the THIS variable is set to the word Example. In the next two lines, the THIS and THAT variables are set to their current values or to Not Set, if they are not currently set. Notice that because we just set THIS to the string Example, when we echo the value of THIS it appears as Example. However, since THAT was not set, it appears as Not Set.

Note?

For the rest of this section, we show how variables and commands may appear in a shell script. To try out any of those examples, however, you can simply type them into a shell as shown in the previous example.

In the following example, we set MYFILENAME to /home/digby/myfile.txt. Next, the FILE variable is set to myfile.txt and DIR is set to /home/digby. In the NAME variable, the file name is cut down to simply myfile, then in the EXTENSION variable the file extension is set to txt. (To try these out, you can type them at a shell prompt as we did with the previous example, then echo the value of each variable to see how it is set.)

MYFILENAME="/home/digby/myfile.txt"
FILE=${MYFILENAME##*/}        #FILE becomes "myfile.txt"
DIR=${MYFILENAME%/*}          #DIR becomes "/home/digby
NAME=${FILE%.*}               #NAME becomes "myfile"
EXTENSION=${FILE#*.}          #EXTENSION becomes "txt"

Performing arithmetic in shell scripts

Bash used untyped variables, meaning it normally treats variables as strings or text, but can change them on the fly if you want it to. Unless you tell it otherwise with declare, your variables are just a bunch of letters to bash. But when you start trying to do arithmetic with them, bash will convert them to integers if it can. This makes it possible to do some fairly complex arithmetic in bash.

Integer arithmetic can be performed using the built-in let command or through the external expr or bc commands. After setting the variable BIGNUM value to 1024, the three commands that follow would all store the value 64 in the RESULT variable:

BIGNUM=1024
let RESULT=$BIGNUM/16
RESULT=`expr $BIGNUM / 16`
RESULT=`echo "$BUGNUM / 16" | bc –l`
Note?

While most elements of shell scripts are relatively freeform (where white space, such as spaces or tabs, is insignificant), both let and expr are particular about spacing. The let command insists on no spaces between each operand and the mathematical operator, whereas the syntax of the expr command requires white space between each operand and its operator. In contrast to those, bc isn't picky about spaces, but can be trickier to use because it does floating-point arithmetic.

To see a complete list of the kinds of arithmetic you can perform using the let command, type help let at the bash prompt.

Using programming constructs in shell scripts

One of the features that make shell scripts so powerful is that their implementation of looping and conditional execution constructs similar to those found in more complex scripting and programming languages. You can use several different types of loops, depending on your needs.

The "if… then" statements

The most commonly used programming construct is conditional execution, or the "if" statement. It is used to perform actions only under certain conditions. There are several variations, depending on whether you're testing one thing, or want to do one thing if a condition is true, but another thing if a condition is false, or if you want to test several things one after the other.

The first if/then example tests if VARIABLE is set to the number 1. If it is, then the echo command is used to say that it is set to 1.

if [ $VARIABLE -eq 1 ] ; then
echo "The variable is 1"
fi

Instead of using –eq, you can use the equals sign (=), as shown in the following example. Using the else statement, different words can be echoed if the criterion of the if statement isn't met ($STRING = "Friday").

if [ $STRING = "Friday" ] ; then
echo "WhooHoo!  Friday!"
else
echo "Will Friday ever get here?"
fi

You can also reverse tests with an exclamation mark (!). In the following example, if STRING is not Monday, then "At least it's not Monday" is echoed.

if [ $STRING != "Monday" ] ; then
   echo "At least it's not Monday"
fi

In the following example, elif is used to test for an additional condition (is filename a file or a directory).

if [ -f $filename ] ; then
   echo "$filename is a regular file"
elif [ -d $filename ] ; then
   echo "$filename is a directory"
else
   echo "I have no idea what $filename is"
fi

As you can see from the preceding examples, the condition you are testing is placed between square brackets [ ]. When a test expression is evaluated, it will return either a value of 0, meaning that it is true, or a 1, meaning that it is false. Table 12-2 lists the conditions that are testable and is quite a handy reference. (If you're in a hurry, you can type help test on the command line to get the same information.)

Table 12-2: Operators for Test Expressions

Operator

What Is Being Tested?

-a file

Does the file exist? (same as –e)

-b file

Is the file a special block device?

-c file

Is the file character special (e.g., a character device)? Used to identify serial lines and terminal devices.

-d file

Is the file a directory?

-e file

Does the file exist? (same as -a)

-f file

Does the file exist, and is it a regular file (e.g., not a directory, socket, pipe, link, or device file)?

-g file

Does the file have the set-group-id bit set?

-h file

Is the file a symbolic link? (same as –L)

??

-k file

Does the file have the sticky bit set?

-L file

Is the file a symbolic link?

-n string

Is the length of the string greater than 0 bytes?

-O file

Do you own the file?

-p file

Is the file a named pipe?

-r file

Is the file readable by you?

-s file

Does the file exist, and is it larger than 0 bytes?

-S file

Does the file exist, and is it a socket?

-t fd

Is the file descriptor connected to a terminal?

-u file

Does the file have the set-user-id bit set?

-w file

Is the file writable by you?

-x file

Is the file executable by you?

-z string

Is the length of the string 0 (zero) bytes?

expr1 -a expr2

Are both the first expression and the second expression true?

expr1 -o expr2

Is either of the two expressions true?

file1 -nt file2

Is the first file newer than the second file (using the modification timestamp)?

file1 -ot file2

Is the first file older than the second file (using the modification timestamp)?

file1 -ef file2

Are the two files associated by a link (a hard link or a symbolic link)?

var1 = var2

Is the first variable equal to the second variable?

var1 -eq var2

Is the first variable equal to the second variable?

var1 -ge var2

Is the first variable greater than or equal to the second variable?

var1 -gt var2

Is the first variable greater than the second variable?

var1 -le var2

Is the first variable less than or equal to the second variable?

var1 -lt var2

Is the first variable less than the second variable?

var1 != var2

Is the first variable not equal to the second variable?

var1 -ne var2

Is the first variable not equal to the second variable?

There is also a special shorthand method of performing tests that can be useful for simple one-command actions. In the following example, the two pipes (||) indicate that if the directory being tested for doesn't exist (-d dirname), then make the directory (mkdir $dirname).

# [ test ] || {action}
# Perform simple single command {action} if test is false
[ -d $dirname ] || mkdir $dirname

Instead of pipes, two ampersands can be used to test if something is true. In the example below, a command is being tested to see if it includes at least three command line arguments.

# [ test ] && {action}
# Perform simple single command {action} if test is true
[ $# -ge 3 ] && echo "There are at least 3 command line arguments."

The case command

Another frequently used construct is the case command. Similar to a switch statement in programming languages, this can take the place of several nested if statements. A general form of the case statement is as follows:

case "VAR" in
   Result1)
      { body };;
   Result2)
      { body };;
   *)
      { body } ;;
esac

One use for the case command might be to help with your backups. The following case statement tests for the first three letters of the current day (case `date +%a` in). Then, depending on the day, a particular backup directory (BACKUP) and tape drive (TAPE) is set.

# Our VAR doesn't have to be a variable,
# it can be the output of a command as well
# Perform action based on day of week
case `date +%a` in
   "Mon")
         BACKUP=/home/myproject/data0
         TAPE=/dev/rft0
# Note the use of the double semi-colon to end each option
         ;;
# Note the use of the "|" to mean "or"
   "Tue" | "Thu")
         BACKUP=/home/myproject/data1
         TAPE=/dev/rft1
         ;;
   "Wed" | "Fri")
         BACKUP=/home/myproject/data2
         TAPE=/dev/rft2
         ;;
# Don't do backups on the weekend.
   *)
         BACKUP="none"
         TAPE=/dev/null
         ;;
esac

The asterisk (*) is used as a catchall, similar to the default keyword in the C programming language. In this example, if none of the other entries are matched on the way down the loop, the asterisk is matched, and the value of BACKUP becomes none.

The "for . . . do" loop

Loops are used to perform actions over and over again until a condition is met or until all data has been processed. One of the most commonly used loops is the for . . . do loop. It iterates through a list of values, executing the body of the loop for each element in the list. The syntax and a few examples are presented here:

for VAR in LIST
do
    { body }
done

The for loop assigns the values in LIST to VAR one at a time. Then for each value, the body in braces between do and done is executed. VAR can be any variable name, and LIST can be composed of pretty much any list of values or anything that generates a list.

for NUMBER in 0 1 2 3 4 5 6 7 8 9
do
   echo The number is $NUMBER
done
   
for FILE in `/bin/ls`
do
   echo $FILE
done
   
# You can also write it this way, which is somewhat cleaner.
for NAME in John Paul Ringo George ; do
   echo $NAME is my favorite Beetle
done

Each element in the LIST is separated from the next by white space. This can cause trouble if you're not careful, since some commands, like ls –l, output multiple fields per line, each separated by whitespace.

If you're a die-hard C programmer, bash allows you to use C syntax to control your loops.

LIMIT=10
# Double parentheses, and no $ on LIMIT even though it's a variable!
for ((a=1; a <= LIMIT ; a++)) ; do
  echo  "$a"
done

The "while . . . do" and "until . . . do" loops

Two other possible looping constructs are the while. . . do loop and the until. . . do loop. The structure of each is presented here:

while condition      until condition
do                   do
   { body }            { body }
done                 done

Here is an example of a while loop that will output the number 0123456789:

N=0
while [ $N –lt 10 ] ; do
   echo –n $N
   let N=$N+1
done

Another way to output the number 0123456789 is to use an until loop as follows:

N=0
until [ $N –eq 10 ] ; do
   echo –n $N
   let N=$N+1
done

Some useful external programs

Bash is great and has lots of built-in commands, but it usually needs some help to do anything really useful. Some of the most common useful programs you'll see used are grep, cut, tr, and sed. Like all the best UNIX tools, most of these programs are designed to work with standard input and standard output, so you can easily use them with pipes and shell scripts.

The General Regular Expression Parser (grep)

The name sounds intimidating, but grep is just a way to find patterns in files or text. Think of it as a useful search tool. Getting really good with regular expressions is quite a challenge, but many useful things can be accomplished with just the simplest forms.

For example, you can display a list of all regular user accounts by using grep to search for all lines that contain the text /home in the /etc/passwd file as follows:

grep /home /etc/passwd

Or you could find all environment variables that begin with HO using the following command:

env | grep ^HO

To find a list of options to use with the grep command, type man grep.

Remove sections of lines of text (cut)

The cut command can extract specific fields from a line of text or from files. It is very useful for parsing system configuration files into easy-to-digest chunks. You can specify the field separator you want to use and the fields you want, or you can break a line up based on bytes.

The following example lists all home directories of users on your system. Using an earlier example of the grep command, this line pipes a list of regular users from the /etc/passwd file, then displays the sixth field (-f6) as delimited by a colon (-d':').

grep /home /etc/passwd | cut –f6 –d':'

Translate or delete characters (tr)

The tr command is a character-based translator that can be used to replace one character or set of characters with another or to remove a character from a line of text.

The following example translates all upper-case letters to lowercase letters and displays the words "mixed upper and lower case" as a result:

FOO="Mixed UPpEr aNd LoWeR cAsE"
echo $FOO | tr [A-Z] [a-z]

In this example, the tr command is used on a list of file names to rename any files in that list so that any spaces contained in a file name are translated into underscores:

for file in * ; do
   d=`echo $file | tr [:blank:] [_]`
   [ "$file" –eq "-d" ] || mv "$file" "$d"
done

The Stream Editor (sed)

The sed command is a simple scriptable editor, and as such can perform only simple edits, such as removing lines that have text matching a certain pattern, replacing one pattern of characters with another, and other simple edits. To get a better idea of how sed scripts work, there's no substitute for the online documentation, but here are some examples of common uses.

You can use the sed command to essentially do what we did earlier with the grep example: search the /etc/passwd file for the word home. Here the sed command searches the entire /etc/passwd file, searches for the word home, and prints any line containing the word home.

sed –n –e '/home/p' /etc/passwd

In this example, sed searches the file somefile.txt and replaces every instance of the string Mac with Linux. The output is then sent to the fixed_file.txt file.

sed –e 's/Mac/Linux/' somefile.txt > fixed_file.txt

You can get the same result using a pipe:

cat somefile.txt | sed –e 's/Mac/Linux/' > fixed_file.txt

By searching for a pattern and replacing it with a null pattern, you delete the original pattern. This example searches the contents of the somefile.txt file and replaces extra blank spaces at the end of each line (s/ *$) with nothing (//). Results go to the fixed_file.txt file.

cat somefile.txt | sed –e 's/ *$//` > fixed_file.txt

Trying some simple shell scripts

Sometimes the simplest of scripts can be the most useful. If you type the same sequence of commands repetitively, it makes sense to store those commands (once!) in a file. Here are a couple of simple, but useful, shell scripts.

A simple telephone list

This idea has been handed down from generation to generation of old UNIX hacks. It's really quite simple, but it employs several of the concepts just introduced.

#!/bin/bash
# (@)/ph
# A very simple telephone list
# Type "ph new name number" to add to the list, or
# just type "ph name" to get a phone number
   
PHONELIST=~/.phonelist.txt
   
# If no command line parameters ($#), there
# is a problem, so ask what they're talking about.
if [ $# -lt 1 ] ; then
   echo "Whose phone number did you want?"
   exit 1
fi
   
# Did you want to add a new phone number?
if [ "$1" = "new" ] ; then
   shift          
   echo $* >> $PHONELIST
   echo $* added to database
   exit 0
fi
   
# Nope. But does the file have anyting in it yet?
# This might be our first time using it, after all.
if [ ! -s $PHONELIST ] ; then
   echo "No names in the phone list yet!"
   exit 1
else
   grep –q "$*" $PHONELIST     # Quietly search the file
   if [ $? –ne 0 ] ; then          # Did we find anything?
      echo "Sorry, that name was not found in the phone list"
      exit 1
   else
      grep "$*" $PHONELIST
   fi
fi
exit 0

A Simple Backup Script

Since nothing works forever and mistakes happen, backups are just a fact of life when dealing with computer data. This simple script will back up all the data in the home directories of all the users on your Red Hat Linux system.

#!/bin/bash
# (@)/my_backup
# A very simple backup script
#
   
TAPE=/dev/rft0
   
# Rewind the tape device $TAPE
mt $TAPE rew
# Get a list of home directories
HOMES=`grep /home /etc/passwd | cut –f6 –d';'`
# Backup the data in those directories
tar cvf $TAPE $HOMES
# Rewind and eject the tape.
mt $TAPE rewoffl
Cross-Reference?

See Chapter 13 for details on backing up and restoring files.




Part IV: Red Hat Linux Network and Server Setup