Home
Programming
Perl tutorial

Chapter 20. Web Automation

The web, then, or the pattern, a web at once sensuous and logical, an elegant and pregnant texture: that is style, that is the foundation of the art of literature.
Robert Louis Stevenson, On some Technical Elements of Style in Literature (1885)

Chapter 1. Strings

Introduction

Recipe 1.1 Accessing Substrings

Recipe 1.2 Establishing a Default Value

Recipe 1.3 Exchanging Values Without Using Temporary Variables

Recipe 1.4 Converting Between Characters and Values

Recipe 1.5 Using Named Unicode Characters

Recipe 1.6 Processing a String One Character at a Time

Recipe 1.7 Reversing a String by Word or Character

Recipe 1.8 Treating Unicode Combined Characters as Single Characters

Recipe 1.9 Canonicalizing Strings with Unicode Combined Characters

Recipe 1.10 Treating a Unicode String as Octets

Recipe 1.11 Expanding and Compressing Tabs

Recipe 1.12 Expanding Variables in User Input

Recipe 1.13 Controlling Case

Recipe 1.14 Properly Capitalizing a Title or Headline

Recipe 1.15 Interpolating Functions and Expressions Within Strings

Recipe 1.16 Indenting Here Documents

Recipe 1.17 Reformatting Paragraphs

Recipe 1.18 Escaping Characters

Recipe 1.19 Trimming Blanks from the Ends of a String

Recipe 1.20 Parsing Comma-Separated Data

Recipe 1.21 Constant Variables

Recipe 1.22 Soundex Matching

Recipe 1.23 Program: fixstyle

Recipe 1.24 Program: psgrep

Chapter 2. Numbers

Introduction

Recipe 2.1 Checking Whether a String Is a Valid Number

Recipe 2.2 Rounding Floating-Point Numbers

Recipe 2.3 Comparing Floating-Point Numbers

Recipe 2.4 Operating on a Series of Integers

Recipe 2.5 Working with Roman Numerals

Recipe 2.6 Generating Random Numbers

Recipe 2.7 Generating Repeatable Random Number Sequences

Recipe 2.8 Making Numbers Even More Random

Recipe 2.9 Generating Biased Random Numbers

Recipe 2.10 Doing Trigonometry in Degrees, Not Radians

Recipe 2.11 Calculating More Trigonometric Functions

Recipe 2.12 Taking Logarithms

Recipe 2.13 Multiplying Matrices

Recipe 2.14 Using Complex Numbers

Recipe 2.15 Converting Binary, Octal, and Hexadecimal Numbers

Recipe 2.16 Putting Commas in Numbers

Recipe 2.17 Printing Correct Plurals

Recipe 2.18 Program: Calculating Prime Factors

Chapter 3. Dates and Times

Introduction

Recipe 3.1 Finding Today's Date

Recipe 3.2 Converting DMYHMS to Epoch Seconds

Recipe 3.3 Converting Epoch Seconds to DMYHMS

Recipe 3.4 Adding to or Subtracting from a Date

Recipe 3.5 Difference of Two Dates

Recipe 3.6 Day in a Week/Month/Year or Week Number

Recipe 3.7 Parsing Dates and Times from Strings

Recipe 3.8 Printing a Date

Recipe 3.9 High-Resolution Timers

Recipe 3.10 Short Sleeps

Recipe 3.11 Program: hopdelta

Chapter 4. Arrays

Introduction

Recipe 4.1 Specifying a List in Your Program

Recipe 4.2 Printing a List with Commas

Recipe 4.3 Changing Array Size

Recipe 4.4 Implementing a Sparse Array

Recipe 4.5 Iterating Over an Array

Recipe 4.6 Iterating Over an Array by Reference

Recipe 4.7 Extracting Unique Elements from a List

Recipe 4.8 Finding Elements in One Array but Not Another

Recipe 4.9 Computing Union, Intersection, or Difference of Unique Lists

Recipe 4.10 Appending One Array to Another

Recipe 4.11 Reversing an Array

Recipe 4.12 Processing Multiple Elements of an Array

Recipe 4.13 Finding the First List Element That Passes a Test

Recipe 4.14 Finding All Elements in an Array Matching Certain Criteria

Recipe 4.15 Sorting an Array Numerically

Recipe 4.16 Sorting a List by Computable Field

Recipe 4.17 Implementing a Circular List

Recipe 4.18 Randomizing an Array

Recipe 4.19 Program: words

Recipe 4.20 Program: permute

Chapter 5. Hashes

Introduction

Recipe 5.1 Adding an Element to a Hash

Recipe 5.2 Testing for the Presence of a Key in a Hash

Recipe 5.3 Creating a Hash with Immutable Keys or Values

Recipe 5.4 Deleting from a Hash

Recipe 5.5 Traversing a Hash

Recipe 5.6 Printing a Hash

Recipe 5.7 Retrieving from a Hash in Insertion Order

Recipe 5.8 Hashes with Multiple Values per Key

Recipe 5.9 Inverting a Hash

Recipe 5.10 Sorting a Hash

Recipe 5.11 Merging Hashes

Recipe 5.12 Finding Common or Different Keys in Two Hashes

Recipe 5.13 Hashing References

Recipe 5.14 Presizing a Hash

Recipe 5.15 Finding the Most Common Anything

Recipe 5.16 Representing Relationships Between Data

Recipe 5.17 Program: dutree

Chapter 6. Pattern Matching

Introduction

Recipe 6.1 Copying and Substituting Simultaneously

Recipe 6.2 Matching Letters

Recipe 6.3 Matching Words

Recipe 6.4 Commenting Regular Expressions

Recipe 6.5 Finding the Nth Occurrence of a Match

Recipe 6.6 Matching Within Multiple Lines

Recipe 6.7 Reading Records with a Separator

Recipe 6.8 Extracting a Range of Lines

Recipe 6.9 Matching Shell Globs as Regular Expressions

Recipe 6.10 Speeding Up Interpolated Matches

Recipe 6.11 Testing for a Valid Pattern

Recipe 6.12 Honoring Locale Settings in Regular Expressions

Recipe 6.13 Approximate Matching

Recipe 6.14 Matching from Where the Last Pattern Left Off

Recipe 6.15 Greedy and Non-Greedy Matches

Recipe 6.16 Detecting Doubled Words

Recipe 6.17 Matching Nested Patterns

Recipe 6.18 Expressing AND, OR, and NOT in a Single Pattern

Recipe 6.19 Matching a Valid Mail Address

Recipe 6.20 Matching Abbreviations

Recipe 6.21 Program: urlify

Recipe 6.22 Program: tcgrep

Recipe 6.23 Regular Expression Grab Bag

Chapter 7. File Access

Introduction

Recipe 7.1 Opening a File

Recipe 7.2 Opening Files with Unusual Filenames

Recipe 7.3 Expanding Tildes in Filenames

Recipe 7.4 Making Perl Report Filenames in Error Messages

Recipe 7.5 Storing Filehandles into Variables

Recipe 7.6 Writing a Subroutine That Takes Filehandles as Built-ins Do

Recipe 7.7 Caching Open Output Filehandles

Recipe 7.8 Printing to Many Filehandles Simultaneously

Recipe 7.9 Opening and Closing File Descriptors by Number

Recipe 7.10 Copying Filehandles

Recipe 7.11 Creating Temporary Files

Recipe 7.12 Storing a File Inside Your Program Text

Recipe 7.13 Storing Multiple Files in the DATA Area

Recipe 7.14 Writing a Unix-Style Filter Program

Recipe 7.15 Modifying a File in Place with a Temporary File

Recipe 7.16 Modifying a File in Place with the -i Switch

Recipe 7.17 Modifying a File in Place Without a Temporary File

Recipe 7.18 Locking a File

Recipe 7.19 Flushing Output

Recipe 7.20 Doing Non-Blocking I/O

Recipe 7.21 Determining the Number of Unread Bytes

Recipe 7.22 Reading from Many Filehandles Without Blocking

Recipe 7.23 Reading an Entire Line Without Blocking

Recipe 7.24 Program: netlock

Recipe 7.25 Program: lockarea

Chapter 8. File Contents

Introduction

Recipe 8.1 Reading Lines with Continuation Characters

Recipe 8.2 Counting Lines (or Paragraphs or Records) in a File

Recipe 8.3 Processing Every Word in a File

Recipe 8.4 Reading a File Backward by Line or Paragraph

Recipe 8.5 Trailing a Growing File

Recipe 8.6 Picking a Random Line from a File

Recipe 8.7 Randomizing All Lines

Recipe 8.8 Reading a Particular Line in a File

Recipe 8.9 Processing Variable-Length Text Fields

Recipe 8.10 Removing the Last Line of a File

Recipe 8.11 Processing Binary Files

Recipe 8.12 Using Random-Access I/O

Recipe 8.13 Updating a Random-Access File

Recipe 8.14 Reading a String from a Binary File

Recipe 8.15 Reading Fixed-Length Records

Recipe 8.16 Reading Configuration Files

Recipe 8.17 Testing a File for Trustworthiness

Recipe 8.18 Treating a File as an Array

Recipe 8.19 Setting the Default I/O Layers

Recipe 8.20 Reading or Writing Unicode from a Filehandle

Recipe 8.21 Converting Microsoft Text Files into Unicode

Recipe 8.22 Comparing the Contents of Two Files

Recipe 8.23 Pretending a String Is a File

Recipe 8.24 Program: tailwtmp

Recipe 8.25 Program: tctee

Recipe 8.26 Program: laston

Recipe 8.27 Program: Flat File Indexes

Chapter 9. Directories

Introduction

Recipe 9.1 Getting and Setting Timestamps

Recipe 9.2 Deleting a File

Recipe 9.3 Copying or Moving a File

Recipe 9.4 Recognizing Two Names for the Same File

Recipe 9.5 Processing All Files in a Directory

Recipe 9.6 Globbing, or Getting a List of Filenames Matching a Pattern

Recipe 9.7 Processing All Files in a Directory Recursively

Recipe 9.8 Removing a Directory and Its Contents

Recipe 9.9 Renaming Files

Recipe 9.10 Splitting a Filename into Its Component Parts

Recipe 9.11 Working with Symbolic File Permissions Instead of Octal Values

Recipe 9.12 Program: symirror

Recipe 9.13 Program: lst

Chapter 10. Subroutines

Introduction

Recipe 10.1 Accessing Subroutine Arguments

Recipe 10.2 Making Variables Private to a Function

Recipe 10.3 Creating Persistent Private Variables

Recipe 10.4 Determining Current Function Name

Recipe 10.5 Passing Arrays and Hashes by Reference

Recipe 10.6 Detecting Return Context

Recipe 10.7 Passing by Named Parameter

Recipe 10.8 Skipping Selected Return Values

Recipe 10.9 Returning More Than One Array or Hash

Recipe 10.10 Returning Failure

Recipe 10.11 Prototyping Functions

Recipe 10.12 Handling Exceptions

Recipe 10.13 Saving Global Values

Recipe 10.14 Redefining a Function

Recipe 10.15 Trapping Undefined Function Calls with AUTOLOAD

Recipe 10.16 Nesting Subroutines

Recipe 10.17 Writing a Switch Statement

Recipe 10.18 Program: Sorting Your Mail

Chapter 11. References and Records

Introduction

Recipe 11.1 Taking References to Arrays

Recipe 11.2 Making Hashes of Arrays

Recipe 11.3 Taking References to Hashes

Recipe 11.4 Taking References to Functions

Recipe 11.5 Taking References to Scalars

Recipe 11.6 Creating Arrays of Scalar References

Recipe 11.7 Using Closures Instead of Objects

Recipe 11.8 Creating References to Methods

Recipe 11.9 Constructing Records

Recipe 11.10 Reading and Writing Hash Records to Text Files

Recipe 11.11 Printing Data Structures

Recipe 11.12 Copying Data Structures

Recipe 11.13 Storing Data Structures to Disk

Recipe 11.14 Transparently Persistent Data Structures

Recipe 11.15 Coping with Circular Data Structures Using Weak References

Recipe 11.16 Program: Outlines

Recipe 11.17 Program: Binary Trees

Chapter 12. Packages, Libraries, and Modules

Introduction

Recipe 12.1 Defining a Module's Interface

Recipe 12.2 Trapping Errors in require or use

Recipe 12.3 Delaying use Until Runtime

Recipe 12.4 Making Variables Private to a Module

Recipe 12.5 Making Functions Private to a Module

Recipe 12.6 Determining the Caller's Package

Recipe 12.7 Automating Module Cleanup

Recipe 12.8 Keeping Your Own Module Directory

Recipe 12.9 Preparing a Module for Distribution

Recipe 12.10 Speeding Module Loading with SelfLoader

Recipe 12.11 Speeding Up Module Loading with Autoloader

Recipe 12.12 Overriding Built-in Functions

Recipe 12.13 Overriding a Built-in Function in All Packages

Recipe 12.14 Reporting Errors and Warnings Like Built-ins

Recipe 12.15 Customizing Warnings

Recipe 12.16 Referring to Packages Indirectly

Recipe 12.17 Using h2ph to Translate C #include Files

Recipe 12.18 Using h2xs to Make a Module with C Code

Recipe 12.19 Writing Extensions in C with Inline::C

Recipe 12.20 Documenting Your Module with Pod

Recipe 12.21 Building and Installing a CPAN Module

Recipe 12.22 Example: Module Template

Recipe 12.23 Program: Finding Versions and Descriptions of Installed Modules

Chapter 13. Classes, Objects, and Ties

Introduction

Recipe 13.1 Constructing an Object

Recipe 13.2 Destroying an Object

Recipe 13.3 Managing Instance Data

Recipe 13.4 Managing Class Data

Recipe 13.5 Using Classes as Structs

Recipe 13.6 Cloning Constructors

Recipe 13.7 Copy Constructors

Recipe 13.8 Invoking Methods Indirectly

Recipe 13.9 Determining Subclass Membership

Recipe 13.10 Writing an Inheritable Class

Recipe 13.11 Accessing Overridden Methods

Recipe 13.12 Generating Attribute Methods Using AUTOLOAD

Recipe 13.13 Coping with Circular Data Structures Using Objects

Recipe 13.14 Overloading Operators

Recipe 13.15 Creating Magic Variables with tie

Chapter 14. Database Access

Introduction

Recipe 14.1 Making and Using a DBM File

Recipe 14.2 Emptying a DBM File

Recipe 14.3 Converting Between DBM Files

Recipe 14.4 Merging DBM Files

Recipe 14.5 Sorting Large DBM Files

Recipe 14.6 Storing Complex Data in a DBM File

Recipe 14.7 Persistent Data

Recipe 14.8 Saving Query Results to Excel or CSV

Recipe 14.9 Executing an SQL Command Using DBI

Recipe 14.10 Escaping Quotes

Recipe 14.11 Dealing with Database Errors

Recipe 14.12 Repeating Queries Efficiently

Recipe 14.13 Building Queries Programmatically

Recipe 14.14 Finding the Number of Rows Returned by a Query

Recipe 14.15 Using Transactions

Recipe 14.16 Viewing Data One Page at a Time

Recipe 14.17 Querying a CSV File with SQL

Recipe 14.18 Using SQL Without a Database Server

Recipe 14.19 Program: gghGrep Netscape Global History

Chapter 15. Interactivity

Introduction

Recipe 15.1 Parsing Program Arguments

Recipe 15.2 Testing Whether a Program Is Running Interactively

Recipe 15.3 Clearing the Screen

Recipe 15.4 Determining Terminal or Window Size

Recipe 15.5 Changing Text Color

Recipe 15.6 Reading Single Characters from the Keyboard

Recipe 15.7 Ringing the Terminal Bell

Recipe 15.8 Using POSIX termios

Recipe 15.9 Checking for Waiting Input

Recipe 15.10 Reading Passwords

Recipe 15.11 Editing Input

Recipe 15.12 Managing the Screen

Recipe 15.13 Controlling Another Program with Expect

Recipe 15.14 Creating Menus with Tk

Recipe 15.15 Creating Dialog Boxes with Tk

Recipe 15.16 Responding to Tk Resize Events

Recipe 15.17 Removing the DOS Shell Window with Windows Perl/Tk

Recipe 15.18 Graphing Data

Recipe 15.19 Thumbnailing Images

Recipe 15.20 Adding Text to an Image

Recipe 15.21 Program: Small termcap Program

Recipe 15.22 Program: tkshufflepod

Recipe 15.23 Program: graphbox

Chapter 16. Process Management and Communication

Introduction

Recipe 16.1 Gathering Output from a Program

Recipe 16.2 Running Another Program

Recipe 16.3 Replacing the Current Program with a Different One

Recipe 16.4 Reading or Writing to Another Program

Recipe 16.5 Filtering Your Own Output

Recipe 16.6 Preprocessing Input

Recipe 16.7 Reading STDERR from a Program

Recipe 16.8 Controlling Input and Output of Another Program

Recipe 16.9 Controlling the Input, Output, and Error of Another Program

Recipe 16.10 Communicating Between Related Processes

Recipe 16.11 Making a Process Look Like a File with Named Pipes

Recipe 16.12 Sharing Variables in Different Processes

Recipe 16.13 Listing Available Signals

Recipe 16.14 Sending a Signal

Recipe 16.15 Installing a Signal Handler

Recipe 16.16 Temporarily Overriding a Signal Handler

Recipe 16.17 Writing a Signal Handler

Recipe 16.18 Catching Ctrl-C

Recipe 16.19 Avoiding Zombie Processes

Recipe 16.20 Blocking Signals

Recipe 16.21 Timing Out an Operation

Recipe 16.22 Turning Signals into Fatal Errors

Recipe 16.23 Program: sigrand

Chapter 17. Sockets

Introduction

Recipe 17.1 Writing a TCP Client

Recipe 17.2 Writing a TCP Server

Recipe 17.3 Communicating over TCP

Recipe 17.4 Setting Up a UDP Client

Recipe 17.5 Setting Up a UDP Server

Recipe 17.6 Using Unix Domain Sockets

Recipe 17.7 Identifying the Other End of a Socket

Recipe 17.8 Finding Your Own Name and Address

Recipe 17.9 Closing a Socket After Forking

Recipe 17.10 Writing Bidirectional Clients

Recipe 17.11 Forking Servers

Recipe 17.12 Pre-Forking Servers

Recipe 17.13 Non-Forking Servers

Recipe 17.14 Multitasking Server with Threads

Recipe 17.15 Writing a Multitasking Server with POE

Recipe 17.16 Writing a Multihomed Server

Recipe 17.17 Making a Daemon Server

Recipe 17.18 Restarting a Server on Demand

Recipe 17.19 Managing Multiple Streams of Input

Recipe 17.20 Program: backsniff

Recipe 17.21 Program: fwdport

Chapter 18. Internet Services

Introduction

Recipe 18.1 Simple DNS Lookups

Recipe 18.2 Being an FTP Client

Recipe 18.3 Sending Mail

Recipe 18.4 Reading and Posting Usenet News Messages

Recipe 18.5 Reading Mail with POP3

Recipe 18.6 Simulating Telnet from a Program

Recipe 18.7 Pinging a Machine

Recipe 18.8 Accessing an LDAP Server

Recipe 18.9 Sending Attachments in Mail

Recipe 18.10 Extracting Attachments from Mail

Recipe 18.11 Writing an XML-RPC Server

Recipe 18.12 Writing an XML-RPC Client

Recipe 18.13 Writing a SOAP Server

Recipe 18.14 Writing a SOAP Client

Recipe 18.15 Program: rfrm

Recipe 18.16 Program: expn and vrfy

Chapter 19. CGI Programming

Introduction

Recipe 19.1 Writing a CGI Script

Recipe 19.2 Redirecting Error Messages

Recipe 19.3 Fixing a 500 Server Error

Recipe 19.4 Writing a Safe CGI Program

Recipe 19.5 Executing Commands Without Shell Escapes

Recipe 19.6 Formatting Lists and Tables with HTML Shortcuts

Recipe 19.7 Redirecting to a Different Location

Recipe 19.8 Debugging the Raw HTTP Exchange

Recipe 19.9 Managing Cookies

Recipe 19.10 Creating Sticky Widgets

Recipe 19.11 Writing a Multiscreen CGI Script

Recipe 19.12 Saving a Form to a File or Mail Pipe

Recipe 19.13 Program: chemiserie

Chapter 20. Web Automation

Introduction

Recipe 20.1 Fetching a URL from a Perl Script

Recipe 20.2 Automating Form Submission

Recipe 20.3 Extracting URLs

Recipe 20.4 Converting ASCII to HTML

Recipe 20.5 Converting HTML to ASCII

Recipe 20.6 Extracting or Removing HTML Tags

Recipe 20.7 Finding Stale Links

Recipe 20.8 Finding Fresh Links

Recipe 20.9 Using Templates to Generate HTML

Recipe 20.10 Mirroring Web Pages

Recipe 20.11 Creating a Robot

Recipe 20.12 Parsing a Web Server Log File

Recipe 20.13 Processing Server Logs

Recipe 20.14 Using Cookies

Recipe 20.15 Fetching Password-Protected Pages

Recipe 20.16 Fetching https:// Web Pages

Recipe 20.17 Resuming an HTTP GET

Recipe 20.18 Parsing HTML

Recipe 20.19 Extracting Table Data

Recipe 20.20 Program: htmlsub

Recipe 20.21 Program: hrefsub

Chapter 21. mod_perl

Introduction

Recipe 21.1 Authenticating

Recipe 21.2 Setting Cookies

Recipe 21.3 Accessing Cookie Values

Recipe 21.4 Redirecting the Browser

Recipe 21.5 Interrogating Headers

Recipe 21.6 Accessing Form Parameters

Recipe 21.7 Receiving Uploaded Files

Recipe 21.8 Speeding Up Database Access

Recipe 21.9 Customizing Apache's Logging

Recipe 21.10 Transparently Storing Information in URLs

Recipe 21.11 Communicating Between mod_perl and PHP

Recipe 21.12 Migrating from CGI to mod_perl

Recipe 21.13 Sharing Information Between Handlers

Recipe 21.14 Reloading Changed Modules

Recipe 21.15 Benchmarking a mod_perl Application

Recipe 21.16 Templating with HTML::Mason

Recipe 21.17 Templating with Template Toolkit

Chapter 22. XML

Introduction

Recipe 22.1 Parsing XML into Data Structures

Recipe 22.2 Parsing XML into a DOM Tree

Recipe 22.3 Parsing XML into SAX Events

Recipe 22.4 Making Simple Changes to Elements or Text

Recipe 22.5 Validating XML

Recipe 22.6 Finding Elements and Text Within an XML Document

Recipe 22.7 Processing XML Stylesheet Transformations

Recipe 22.8 Processing Files Larger Than Available Memory

Recipe 22.9 Reading and Writing RSS Files

Recipe 22.10 Writing XML