1.9 CPAN Modules

The Comprehensive Perl Archive Network (CPAN, http://www.cpan.org) is an impressively large collection of Perl code (mostly Perl modules). CPAN is easily accessible and searchable on the Web, and you can use its modules for a variety of programming tasks.

By now you should have the basic idea of how modules are defined and used, so let's take some time to explore CPAN to see what goodies are available.

There are two important points about CPAN. First, a large number of the things you might want your programs to do have already been programmed and are easily obtained in downloadable modules. You just have to go find them at CPAN, install them on your computer, and call them from your program. We'll take a look at an example of exactly that in this section.

Second, all code on CPAN is free of charge and available for use by a very unrestrictive copyright declaration. Sound good? Keep reading.

CPAN includes convenient ways to search for useful modules, and there's a CPAN.pm module built-in with Perl that makes downloading and installing modules quite easy (when things work well, which they usually do). If you can't find CPAN.pm, you should consider updating your current version.

You can find more information by typing the following at the command line:

perldoc CPAN

You can also check the Frequently Asked Questions (FAQ) available at the CPAN web site.

1.9.1 What's Available at CPAN?

The CPAN web site offers several "views" of the CPAN collection of modules and several alternate ways of searching (by module name, category, full text search of the module documentation, etc.). Here is the top-level organization of the modules by overall category:

Development Support
Operating System Interfaces
Networking Devices IPC
Data Type Utilities
Database Interfaces
User Interfaces
Language Interfaces
File Names Systems Locking
String Lang Text Proc
Opt Arg Param Proc
Internationalization Locale
Security and Encryption
World Wide Web HTML HTTP CGI
Server and Daemon Utilities
Archiving and Compression
Images Pixmaps Bitmaps
Mail and Usenet News
Control Flow Utilities
File Handle Input Output
Microsoft Windows Modules
Miscellaneous Modules
Commercial Software Interfaces
Not In Modulelist

1.9.2 Searching CPAN

CPAN's main web page has a few ways to search the contents. Let's say you need to perform some statistics and are looking for code that's already available. We'll go through the steps necessary to search for the code, download and install it, and use the module in a program.

At the main CPAN page, look for "searching" and click on search.cpan.org. If you search for "statistics" in all locations, you'll get over 300 hits, so you should restrict your search to modules with the pull-down menu. You'll get 25 hits (more by the time you read this); here's what you'll see:

1.  Statistics::Candidates
Statistics-MaxEntropy-0.9 - 26 Nov 1998 - Hugo WL ter Doest

2. Statistics::ChiSquare
How random is your data?
Statistics-ChiSquare-0.3 - 23 Nov 2001 - Jon Orwant

3. Statistics::Contingency
Calculate precision, recall, F1, accuracy, etc.
Statistics-Contingency-0.03 - 09 Aug 2002 - Ken Williams

4. Statistics::DEA
Discontiguous Exponential Averaging
Statistics-DEA-0.04 - 17 Aug 2002 - Jarkko Hietaniemi

5. Statistics::Descriptive
Module of basic descriptive statistical functions.
Statistics-Descriptive-2.4 - 26 Apr 1999 - Colin Kuskie

6. Statistics::Distributions
Perl module for calculating critical values of common statistical distributions
Statistics-Distributions-0.07 - 22 Jun 2001 - Michael Kospach

7. Statistics::Frequency
simple counting of elements
Statistics-Frequency-0.02 - 24 Apr 2002 - Jarkko Hietaniemi

8. Statistics::GaussHelmert
General weighted least squares estimation
Statistics-GaussHelmert-0.05 - 18 Apr 2002 - Stephan Heuel

9. Statistics::LTU
An implementation of Linear Threshold Units
Statistics-LTU-2.8 - 27 Feb 1997 - Tom Fawcett

10. Statistics::Lite
Small stats stuff.
Statistics-Lite-1.02 - 15 Apr 2002 - Brian Lalonde 

11.  Statistics::MaxEntropy
Statistics-MaxEntropy-0.9 - 26 Nov 1998 - Hugo WL ter Doest

12. Statistics::OLS
perform ordinary least squares and associated statistics, v 0.07.
Statistics-OLS-0.07 - 13 Oct 2000 - Sanford Morton

13. Statistics::ROC
receiver-operator-characteristic (ROC) curves with nonparametric confidence bounds
Statistics-ROC-0.01 - 22 Jul 1998 - Hans A. Kestler

14. Statistics::Regression
weighted linear regression package (line+plane fitting)
StatisticsRegression - 26 May 2001 - ivo welch

15. Statistics::SparseVector
Perl5 extension for manipulating sparse bitvectors
Statistics-MaxEntropy-0.9 - 26 Nov 1998 - Hugo WL ter Doest

16. Statistics::Descriptive::Discrete
Compute descriptive statistics for discrete data sets.
Statistics-Descriptive-Discrete-0.07 - 13 Jun 2002 - Rhet Turnbull

17. Bio::Tree::Statistics
Calculate certain statistics for a Tree
bioperl-1.0.2 - 16 Jul 2002 - Ewan Birney

18. Device::ISDN::OCLM::Statistics
OCLM statistics superclass
Device-ISDN-OCLM-0.40 - 02 Jan 2000 - Merlin Hughes

19. Device::ISDN::OCLM::CurrentStatistics
OCLM current call statistics
Device-ISDN-OCLM-0.40 - 02 Jan 2000 - Merlin Hughes

20. Device::ISDN::OCLM::ISDNStatistics
OCLM ISDN statistics
Device-ISDN-OCLM-0.40 - 02 Jan 2000 - Merlin Hughes 

21.  Device::ISDN::OCLM::Last10Statistics
OCLM Last10 call statistics
Device-ISDN-OCLM-0.40 - 02 Jan 2000 - Merlin Hughes

22. Device::ISDN::OCLM::LastStatistics
OCLM last call statistics
Device-ISDN-OCLM-0.40 - 02 Jan 2000 - Merlin Hughes

23. Device::ISDN::OCLM::ManualStatistics
OCLM manual call statistics
Device-ISDN-OCLM-0.40 - 02 Jan 2000 - Merlin Hughes

24. Device::ISDN::OCLM::SPStatistics
OCLM service provider statistics
Device-ISDN-OCLM-0.40 - 02 Jan 2000 - Merlin Hughes

25. Device::ISDN::OCLM::SystemStatistics
OCLM system statistics
Device-ISDN-OCLM-0.40 - 02 Jan 2000 - Merlin Hughes

Let's check out the Statistics::ChiSquare module.

First, click on the link to Statistics::ChiSquare; you'll see a summary of the module, complete with a description, overview, discussion of the method, examples of use, and information about the author.

One of the modules looks interesting; let's download and install it. How big is the source code? If you click on the source link, you'll find that the module is really just one short subroutine with the documentation defined right in the module. Here's the subroutine definition part of the module:

package Statistics::ChiSquare;

# ChiSquare.pm
# Jon Orwant, orwant@media.mit.edu
# 31 Oct 95, revised Mon Oct 18 12:16:47 1999, and again November 2001
# to fix an off-by-one error
# Copyright 1995, 1999, 2001 Jon Orwant.  All rights reserved.
# This program is free software; you can redistribute it and/or
# modify it under the same terms as Perl itself.
# Version 0.3.  Module list status is "Rdpf"

use strict;
use vars qw($VERSION @ISA @EXPORT);

require Exporter;
require AutoLoader;

@ISA = qw(Exporter AutoLoader);
# Items to export into callers namespace by default. Note: do not export
# names by default without a very good reason. Use EXPORT_OK instead.
# Do not simply export all your public functions/methods/constants.
@EXPORT = qw(chisquare);

$VERSION = '0.3';

my @chilevels = (100, 99, 95, 90, 70, 50, 30, 10, 5, 1);
my %chitable = (  );

# assume the expected probability distribution is uniform
sub chisquare {
    my @data = @_;
    @data = @{$data[0]} if @data =  = 1 and ref($data[0]);
    my $degrees_of_freedom = scalar(@data) - 1;
    my ($chisquare, $num_samples, $expected, $i) = (0, 0, 0, 0);
    if (! exists($chitable{$degrees_of_freedom})) {
        return "I can't handle ", scalar(@data), 
        " choices without a better table.";
    foreach (@data) { $num_samples += $_ }
    $expected = $num_samples / scalar(@data);
    return "There's no data!" unless $expected;
    foreach (@data) {
        $chisquare += (($_ - $expected) ** 2) / $expected;
    foreach (@{$chitable{$degrees_of_freedom}}) {
        if ($chisquare < $_) {
             "There's a <$chilevels[$i+1]% and <$chilevels[$i]% chance that this data 
                    is random.";
    return "There's a <$chilevels[$#chilevels]% chance that this data is random.";
$chitable{1} = [0.00016, 0.0039, 0.016, 0.15, 0.46, 1.07, 2.71, 3.84, 6.64];
$chitable{2} = [0.020,   0.10,   0.21,  0.71, 1.39, 2.41, 4.60, 5.99, 9.21];
$chitable{3} = [0.12,    0.35,   0.58,  1.42, 2.37, 3.67, 6.25, 7.82, 11.34];
$chitable{4} = [0.30,    0.71,   1.06,  2.20, 3.36, 4.88, 7.78, 9.49, 13.28];
$chitable{5} = [0.55,    1.14,   1.61,  3.00, 4.35, 6.06, 9.24, 11.07, 15.09];
$chitable{6} = [0.87,    1.64,   2.20,  3.83, 5.35, 7.23, 10.65, 12.59, 16.81];
$chitable{7} = [1.24,    2.17,   2.83,  4.67, 6.35, 8.38, 12.02, 14.07, 18.48];
$chitable{8} = [1.65,    2.73,   3.49,  5.53, 7.34, 9.52, 13.36, 15.51, 20.09];
$chitable{9} = [2.09,    3.33,   4.17, 6.39, 8.34, 10.66, 14.68, 16.92, 21.67];
$chitable{10} = [2.56,   3.94,   4.86, 7.27, 9.34, 11.78, 15.99, 18.31, 23.21];
$chitable{11} = [3.05,   4.58,  5.58, 8.15, 10.34, 12.90, 17.28, 19.68, 24.73];
$chitable{12} = [3.57,   5.23, 6.30, 9.03, 11.34, 14.01, 18.55, 21.03, 26.22];
$chitable{13} = [4.11,   5.89, 7.04, 9.93, 12.34, 15.12, 19.81, 22.36, 27.69];
$chitable{14} = [4.66,   6.57, 7.79, 10.82, 13.34, 16.22, 21.06, 23.69, 29.14];
$chitable{15} = [5.23,   7.26, 8.55, 11.72, 14.34, 17.32, 22.31, 25.00, 30.58];
$chitable{16} = [5.81,   7.96, 9.31, 12.62, 15.34, 18.42, 23.54, 26.30, 32.00];
$chitable{17} = [6.41,  8.67, 10.09, 13.53, 16.34, 19.51, 24.77, 27.59, 33.41];
$chitable{18} = [7.00,  9.39, 10.87, 14.44, 17.34, 20.60, 25.99, 28.87, 34.81];
$chitable{19} = [7.63, 10.12, 11.65, 15.35, 18.34, 21.69, 27.20, 30.14, 36.19];
$chitable{20} = [8.26, 10.85, 12.44, 16.27, 19.34, 22.78, 28.41, 31.41, 37.57];


Some of this code will look familiar; some may not. Check out the use of package, use strict, and require Exporter; they're parts of Perl you've just seen.

You'll also see references to version, Autoloader, use vars, and an initialization of a multidimensional array chitable, which will be covered later. For now, you may want to take a quick read-through of the code and get some personal satisfaction at how much of it makes sense.

Indeed, one of the really nice things about most modules is that you don't really have to read the code very often. Usually you can just install the module, read enough of the documentation to see how to call it from your program, and you're off and running. Let's take that approach now.

1.9.3 Installing Modules Using CPAN.pm

Our next task is to install the module using CPAN.pm. This section contains a log from when I installed Statistics::ChiSquare on my Linux computer using CPAN.pm.

In fact, to make things easy, here's the section of the CPAN FAQ that addresses installing modules:

How do I install Perl modules?

Installing a new module can be as simple as typing

perl -MCPAN -e 'install Chocolate::Belgian'.

The CPAN.pm documentation has more complete instructions on how to use
this convenient tool.  If you are uncomfortable with having something
take that much control over your software installation, or it otherwise
doesn't work for you, the perlmodinstall documentation covers
module installation for UNIX, Windows and Macintosh in more familiar terms.

Finally, if you're using ActivePerl on Windows, the PPM (Perl Package Manager)
has much of the same functionality as CPAN.pm.

The following is my install log. Notice that all I have to do is type a couple of lines, and everything else that follows is automatic!

[tisdall@coltrane tisdall]$ perl -MCPAN -e 'install Statistics::ChiSquare'
CPAN: Storable loaded ok
mkdir /root/.cpan: Permission denied at /usr/local/lib/perl5/5.6.1/CPAN.pm line 2218
[tisdall@coltrane tisdall]$ su
[root@coltrane tisdall]# perl -MCPAN -e 'install Statistics::ChiSquare'
CPAN: Storable loaded ok
Going to read /root/.cpan/Metadata
  Database was generated on Wed, 20 Mar 2002 00:39:29 GMT
CPAN: LWP::UserAgent loaded ok
Fetching with LWP:
Going to read /root/.cpan/sources/authors/01mailrc.txt.gz
CPAN: Compress::Zlib loaded ok
Fetching with LWP:
Going to read /root/.cpan/sources/modules/02packages.details.txt.gz
  Database was generated on Mon, 26 Aug 2002 00:22:07 GMT

  There's a new CPAN.pm version (v1.62) available!
  [Current version is v1.59_54]
  You might want to try
    install Bundle::CPAN
    reload cpan
  without quitting the current session. It should be a seamless upgrade
  while we are running...

Fetching with LWP:
Going to read /root/.cpan/sources/modules/03modlist.data.gz
Going to write /root/.cpan/Metadata
Running install for module Statistics::ChiSquare
Running make for J/JO/JONO/Statistics-ChiSquare-0.3.tar.gz
Fetching with LWP:
CPAN: MD5 loaded ok
Fetching with LWP:
Checksum for /root/.cpan/sources/authors/id/J/JO/JONO/Statistics-ChiSquare-0.3.
     tar.gz ok
Scanning cache /root/.cpan/build for sizes
Deleting from cache: /root/.cpan/build/IO-stringy-2.108 (21.4>20.0 MB)
Deleting from cache: /root/.cpan/build/XML-Node-0.11 (20.8>20.0 MB)
Deleting from cache: /root/.cpan/build/bioperl-0.7.2 (20.7>20.0 MB)
Package seems to come without Makefile.PL.
  (The test -f "/root/.cpan/build/Statistics/Makefile.PL" returned false.)
  Writing one on our own (setting NAME to StatisticsChiSquare)

  CPAN.pm: Going to build J/JO/JONO/Statistics-ChiSquare-0.3.tar.gz

Checking if your kit is complete...
Looks good
Writing Makefile for Statistics::ChiSquare
Writing Makefile for StatisticsChiSquare
make[1]: Entering directory `/root/.cpan/build/Statistics/ChiSquare-0.3'
cp ChiSquare.pm ../blib/lib/Statistics/ChiSquare.pm
AutoSplitting ../blib/lib/Statistics/ChiSquare.pm (../blib/lib/auto/
Manifying ../blib/man3/Statistics::ChiSquare.3
make[1]: Leaving directory `/root/.cpan/build/Statistics/ChiSquare-0.3'
  /usr/bin/make  -- OK
Running make test
make[1]: Entering directory `/root/.cpan/build/Statistics/ChiSquare-0.3'
make[1]: Leaving directory `/root/.cpan/build/Statistics/ChiSquare-0.3'
make[1]: Entering directory `/root/.cpan/build/Statistics/ChiSquare-0.3'
PERL_DL_NONLAZY=1 /usr/bin/perl -I../blib/arch -I../blib/lib -I/usr/local/lib/
     perl5/5.6.1/i686-linux -I/usr/local/lib/perl5/5.6.1 test.pl
ok 1
ok 2
make[1]: Leaving directory `/root/.cpan/build/Statistics/ChiSquare-0.3'
  /usr/bin/make test -- OK
Running make install
make[1]: Entering directory `/root/.cpan/build/Statistics/ChiSquare-0.3'
make[1]: Leaving directory `/root/.cpan/build/Statistics/ChiSquare-0.3'
Installing /usr/local/lib/perl5/site_perl/5.6.1/Statistics/ChiSquare.pm
Installing /usr/local/lib/perl5/site_perl/5.6.1/auto/Statistics/ChiSquare/
Installing /usr/local/man/man3/Statistics::ChiSquare.3
Writing /usr/local/lib/perl5/site_perl/5.6.1/i686-linux/auto/
Appending installation info to /usr/local/lib/perl5/5.6.1/i686-linux/perllocal.pod
  /usr/bin/make install UNINST=1 -- OK
[root@coltrane tisdall]#

This may seem like a confusing amount of output, but, again, all you have to do is type a couple of lines, and the installation follows automatically.

You may get something like the following message when you try to install a CPAN module:

[tisdall@coltrane tisdall]$ perl -MCPAN -e 'install Statistics::ChiSquare'
CPAN: Storable loaded ok
mkdir /root/.cpan: Permission denied at /usr/local/lib/perl5/5.6.1/CPAN.pm line 2218

As you can see, it didn't work, and it produced an error message. On Unix machines, it's often necessary to become root to install things.[2] In that case, use the Unix su command and try the CPAN command again:

[2] You may need to contact your system administrator about getting root permission. The CPAN documentation discusses how to do a non-root installation. If you're not on a Unix or Linux machine and are using ActiveState's Perl on a Windows machine, for instance, you need to consult that documentation.

[tisdall@coltrane tisdall]$ su
[root@coltrane tisdall]# perl -MCPAN -e 'install Statistics::ChiSquare'

Great, it worked. If you look over the rather verbose output, you'll see that it finds the module, installs it, tests it, and logs the installation.

Pretty easy, huh?

It's usually this easy, but not always. Occasionally, errors result, and the module may not be installed. In that case, the error messages may be enough to explain the problem; for instance, the module may depend on another module you have to install first. Another problem is that some modules haven't been tested on, or even designed to work on, all operating systems; if you try to install a Windows-specific module on Linux, it is likely to complain. In extreme cases, the module documentation usually provides the author's email address.

1.9.4 Using the Newly Installed CPAN Module

Now comes the payoff. Let's look again at the documentation for the module and see if we can use it from our own Perl code.

Now that the module is installed, you can see the documentation by typing:

perldoc Statistics::ChiSquare

You can also simply go back to the web documentation found at http://search.cpan.org. Either way, you'll find the following example using this ChiSquare module:

       "Statistics::ChiSquare" - How random is your data?

        use Statistics::Chisquare;
        print chisquare(@array_of_numbers);
        Statistics::ChiSquare is available at a CPAN site near

        Suppose you flip a coin 100 times, and it turns up heads
        70 times.  Is the coin fair?
        Suppose you roll a die 100 times, and it shows 30 sixes.
        Is the die loaded?
        In statistics, the chi-square test calculates "how random"
        a series of numbers is.  But it doesn't simply say "yes"
        or "no".  Instead, it gives you a confidence interval,
        which sets upper and lower bounds on the likelihood that
        the variation in your data is due to chance.  See the
        examples below.

The documentation continues with more discussion and some concrete examples that use the module and interpret the results.

Very often, the SYNOPSIS part of the documentation is all you need to look at. It shows you specific examples of how to call the code in the module. In this case, because it's a very simple module, there is just one subroutine that can be used. As you see from the documentation excerpt, you just need to pass the chisquare subroutine an array of numbers and print out the return value to use the code. Let's try it. We'll take as our input an array of numbers that corresponds to the stops of the Broadway-7th Avenue local subway train on the west side of Manhattan, from 14th Street up to 137th Street in Harlem. (We'll assume you didn't run fast enough and missed the A train.) Let's see how random these stops really are:

use strict;
use warnings;

use Statistics::ChiSquare;

my(@subwaystops) = (14, 18, 23, 28, 34, 42, 50, 59, 66, 72, 79, 86, 96, 103, 110, 
116, 125, 137);

print chisquare(@subwaystops);

This produces the output:

There's a <1% chance that this data is random.

(Knowing firsthand the feelings of long-suffering New York City Subway riders, I predict that this result might provoke some spirited discussion. Nevertheless, we seem to have working code.)

1.9.5 Problems with CPAN Modules

Actually, the sharp-eyed reader may have noticed a problem in our mad dash uptown. In the first line of the SYNOPSIS section, there's the following:

use Statistics::Chisquare;

The name of the module is spelled Chisquare, whereas in all other places in the documentation the module is spelled ChiSquare with a capital S. In Perl, the case of a letter, uppercase or lowercase, is important, and this looks suspiciously like a typographical error in the documentation. If you try use Statistics::Chisquare, you'll discover that the module can't be found, whereas if you try use Statistics::ChiSquare, the module is there. This is a minor bug, but some modules have poor documentation, and it can be a time-consuming problem, especially if you are forced to wade into the module code or try various tests, to figure out how the module works.

Apart from bugs, I've also mentioned the problem that some modules are not tested, or designed, for all operating systems. In addition, many modules require other modules to be present. It's possible to configure CPAN to automatically install all the required modules a requested module uses, as described in the CPAN documentation, but you may need to intervene personally. It's useful to remember that if you have a program that uses a certain module running on one computer, and you move the program to another computer, you may have to install the required modules on the new computer as well.

Saving the worst for last, it's also important to remember that contributing to CPAN is open to one and all, and not all the code there is well-written or well-tested. The heavily used modules are, but counterexamples can be found. So, don't bet the farm on your code just because it uses a CPAN module; you should still carefully read the documentation for the module and test your program.

The CPAN FAQ explains in detail the way to be a good citizen when it comes to testing and reporting bugs that you discover in CPAN code.