Appendix D. A State Machine for Adding Markup to Text

Appendix D. A State Machine for Adding Markup to Text

This book was written entirely in plaintext editors, using a set of conventions I call "smart ASCII." In spirit and appearance, smart ASCII resembles the informal markup that has developed on email and Usenet. In fact, I have used an evolving version of the format for a number of years to produce articles, tutorials, and other documents. The book required a few additional conventions in the earlier smart ASCII format, but only a few. It was a toolchain that made almost all the individual typographic and layout decisions. Of course, that toolchain only came to exist through many hours of programming and debugging by me and by other developers.

The printed version of this book used tools I wrote in Python to assemble the chapters, frontmatter, and endmatter, and then to add graphics/latex.gif markup codes to the text. A moderate number of custom graphics/latex.gif macros are included in that markup. From there, the work of other people lets me convert graphics/latex.gif source into the PDF format Addison-Wesley can convert into printed copies.

For information on the smart ASCII format, see the discussions of it in several places in this book, chiefly in Chapter 4. You may also download the ASCII text of this book from its Web site at <>, along with a semiformal documentation of the conventions used. Readers might also be interested in a format called "reStructuredText," which is similar in spirit, but both somewhat "heavier" and more formally specified. reStructuredText has a semiofficial status in the Python community since it is now included in the DocUtils package; for information see:


In this appendix, I include the full source code for an application that can convert the original text of this book into an HTML document. I believe that this application is a good demonstration of the design and structure of a realistic text processing tool. In general structure, uses a line-oriented state machine to categorize lines into appropriate document elements. Under this approach, the "meaning" of a particular line is, in part, determined by the context of the lines that came immediately before it. After making decisions on how to categorize each line with a combination of a state machine and a collection of regular expression patterns, the blocks of document elements are processed into HTML output. In principle, it would not be difficult to substitute a different output format; the steps involved are modular.

The Web site for this book has a collection of utilities similar to the one presented. Over time, I have adapted the skeleton to deal with variations in input and output formats, but there is overlap between all of them. Using this utility is simply a matter of typing something like:

% "Text Processing in Python" < TPiP.txt > TPiP.html

The title is optional, and you may pipe STDIN and STDOUT as usual. Since the target is HTML, I decided it would be nice to colorize source code samples. That capability is in a support module:
import keyword, token, tokenize, sys
from cStringIO import StringIO

PLAIN = '%s'
BOLD  = '<b>%s</b>'
CBOLD = '<font color="%s"><b>%s</b></font>'
_TEXT    = token.NT_OFFSET+2
COLORS   = { token.NUMBER:     'black',
             token.OP:         'darkblue',
             token.STRING:     'green',
             tokenize.COMMENT: 'darkred',
             token.NAME:       None,
             token.ERRORTOKEN: 'red',
             _KEYWORD:         'blue',
             _TEXT:            'black'  }

class ParsePython:
    "Colorize python source"
    def __init__(self, raw):
        self.inp  = StringIO(raw.expandtabs(4).strip())
    def toHTML(self):
        "Parse and send the colored source"
        raw = self.inp.getvalue()
        self.out = StringIO()
        self.lines = [0,0]      # store line offsets in self.lines
        self.lines += [i+1 for i in range(len(raw)) if raw[i]=='\n']
        self.lines += [len(raw)]
        self.pos = 0
            tokenize.tokenize(self.inp.readline, self)
            return self.out.getvalue()
        except tokenize.TokenError, ex:
            msg,ln = ex [0],ex [1] [0]
            sys.stderr.write("ERROR: %s %s\n" %
                             (msg, raw[self.lines[ln]:]))
            return raw
    def __call__(self,toktype,toktext,(srow,scol),(erow,ecol),line):
        "Token handler"
        # calculate new positions
        oldpos = self.pos
        newpos = self.lines[srow] + scol
        self.pos = newpos + len(toktext)
        if toktype in [token.NEWLINE, tokenize.NL]:  # handle newlns
        if newpos > oldpos:     # send the orig whitspce, if needed
        if toktype in [token.INDENT, token.DEDENT]:
            self.pos = newpos   # skip indenting tokens
        if token.LPAR <= toktype and toktype <= token.OP:
            toktype = token.OP  # map token type to a color group
        elif toktype == token.NAME and keyword.iskeyword(toktext):
            toktype = _KEYWORD
        color = COLORS.get(toktype, COLORS [_TEXT])
        if toktext:             # send text
            txt = Detag(toktext)
            if color is None:    txt = PLAIN % txt
            elif color=='black': txt = BOLD % txt
            else:                txt = CBOLD % (color,txt)

Detag = lambda s: \

if __name__=='__main__':
    parsed = ParsePython(
    print '<pre>'
    print parsed.toHTML()
    print '</pre>'

The module colorize contains its own self-test code and is perfectly usable as a utility on its own. The main module consists of:
"""Convert ASCII book source files for HTML presentation"

Usage: python [title] < source.txt > target.html
__author__=["David Mertz (",]
__version__="November 2002"

from __future__ import generators
import sys, re, string, time
from colorize import ParsePython
from cgi import escape

#-- Define some HTML boilerplate
html_open =\
  .code-sample {background-color:#EEEEEE; text-align:left;
                width:90%%; margin-left:auto; margin-right:auto;}
  .module      {color : darkblue}
  .libfunc     {color : darkgreen}
html_title = "Automatically Generated HTML"
html_close = "</body></html>"
code_block = \
"""<table class="code-sample"><tr><td><h4>%s</h4></td></tr>
#-- End of boilerplate

#-- State constants
          "SUBBODY TERM DEF RULE VERTSPC").split():
    exec "%s = '%s'" % (s,s)
markup = {CHAPTER:'h1', SECTION:'h2', SUBSECT:'h3', SUBSUB:'h4',
          BODY:'p', QUOTE:'blockquote', NUMLIST:'blockquote',
divs = {RULE:'hr', VERTSPC:'br'}

class Regexen:
    def __init__(self):
        # blank line is empty, spaces/dashes only, or proc instruct
        self.blank    = re.compile("^[ -]*$|^  THIS IS [A-Z]+$")
        self.chapter  = re.compile("^(CHAPTER|APPENDIX|FRONTMATTER)")
        self.section  = re.compile("^SECTION")
        self.subsect  = re.compile("^  (TOPIC|PROBLEM|EXERCISE)")
        self.subsub   = re.compile("^  [A-Z 0-9-]+:$") # chk befr body
        self.modline  = re.compile("^  =+$")
        self.pyshell  = re.compile("^ +>>>")
        self.codesamp = re.compile("^ +#[*]?[-=]+ .+ [-=]+#")
        self.numlist  = re.compile("^  \d+[.] ")       # chk befr body
        self.body     = re.compile("^  \S")            # 2 spc indent
        self.quote    = re.compile("^     ?\S")        # 4-5 spc indnt
        self.subbody  = re.compile("^      +")         # 6+ spc indent
        self.rule     = re.compile("^  (-\*-|!!!)$")
        self.vertspc  = re.compile("^  \+\+\+$")

def Make_Blocks(fpin=sys.stdin, r=Regexen()):
    #-- Initialize the globals
    global state, blocks, laststate
    state, laststate = BLANK, BLANK
    blocks = [[BLANK]]
    #-- Break the file into relevant chunks
    for line in fpin.xreadlines():
        line = line.rstrip()            # Normalize line endings
        #-- for "one-line states" just act (no accumulation)
        if r.blank.match(line):
            if inState(PYSHELL):        newState(laststate)
            else:                       blocks[-1].append("")
        elif r.rule.match(line):        newState(RULE)
        elif r.vertspc.match(line):     newState(VERTSPC)
        elif r.chapter.match(line):     newState(CHAPTER)
        elif r.section.match(line):     newState(SECTION)
        elif r.subsect.match(line):     newState(SUBSECT)
        elif r.subsub.match(line):      newState(SUBSUB)
        elif r.modline.match(line):     newState(MODLINE)
        elif r.numlist.match(line):     newState(NUMLIST)
        elif r.pyshell.match(line):
            if not inState(PYSHELL):    newState(PYSHELL)
        elif r.codesamp.match(line):    newState(CODESAMP)
        #-- now the multi-line states that are self-defining
        elif r.body.match(line):
            if not inState(BODY):       newState(BODY)
        elif r.quote.match(line):
            if inState(MODLINE):        newState(MODNAME)
            elif r.blank.match(line):   newState(BLANK)
            elif not inState(QUOTE):    newState(QUOTE)
        #-- now the "multi-line states" which eat further lines
            "stay in this state until we get a blank line"
            "...or other one-line prior type, but shouldn't happen"
        elif r.subbody.match(line):
            "Sub-body is tricky: it might belong with several states:"
            "PYSHELL, CODESAMP, NUMLIST, or as a def after BODY"
            if inState(BODY):           newState(DEF)
            elif inState(BLANK):
                if laststate==DEF:      pass
            elif inState(DEF, CODESAMP, PYSHELL, NUMLIST, MODNAME):
            raise ValueError, \
                  "unexpected input block state: %s\n%s" %(state,line)
        if inState(MODLINE, RULE, VERTSPC): pass
        elif r.blank.match(line): pass
        else: blocks[-1].append(line)
    return LookBack(blocks)

def LookBack(blocks):
    types = [f [0] for f in blocks]
    for i in range(len(types)-1):
        this, next = types[i:i+2]
        if (this,next)==(BODY,DEF):
            blocks[i][0] = TERM
    return blocks

def newState(name):
    global state, laststate, blocks
    if name not in (BLANK, MODLINE):
    laststate = state
    state = name

def instate(*names) :
    return state in names

def Process_Blocks(blocks, fpout=sys.stdout, title=html_title):
    fpout.write(html_open % title)
    for block in blocks:        # Massage each block as needed
        typ, lines = block[0], block[1:]
        tag = markup.get(typ, None)
        div = divs.get(typ, None)
        if tag is not None:
            map(fpout.write, wrap_html(lines, tag))
        elif div is not None:
            fpout.write('<%s />\n' % div)
        elif typ in (PYSHELL, CODESAMP):
        elif typ in (MODNAME,):
            mod = '<hr/><h3 class="module">%s</h3>'%'\n'.join(lines)
        elif typ in (TERM,):
            terms = '<br />\n'.join(lines)
            fpout.write('<h4 class="libfunc">%s</h4>\n' % terms)

#-- Functions for start of block-type state
def wrap_html(lines, tag):
    txt = '\n'.join(lines)
    for para in txt.split('\n\n'):
        if para: yield '<%s>%s</%s>\n' %\

def fixcode(block, style=CODESAMP):
    block = LeftMargin(block)           # Move to left
    # Pull out title if available
    title = 'Code Sample'
    if style==CODESAMP:
        re_title = re.compile('^#\*?\-+ (.+) \-+#$', re.M)
        if_title = re_title.match(block)
        if if_title:
            title =
            block = re_title.sub(", block)  # take title out of code
    # Decide if it is Python code
    firstline = block[:block.find('\n')]
    if'\.py_?|[Pp]ython|>>>', title+firstline):
        # Has .py, py_, Python/python, or >>> on first line/title
        block = ParsePython(block.rstrip()).toHTML()
        return code_block % (Typography(title), block)
    # elif the-will-and-the-way-is-there-to-format-language-X: ...
        return code_block % (Typography(title), escape(block).strip())

def LeftMargin(txt):
    "Remove as many leading spaces as possible from whole block"
    for 1 in range(12,-1,-1):
        re_lead = '(?sm)'+' '*1+'\S'
        if re.match(re_lead, txt): break
    txt = re.sub('(?sm)^'+' '*1, ", txt)
    return txt

def URLify(txt):
    # Conv special IMG URL's: Alt Text:}
    # (don't actually try quite as hard to validate URL though)
    txt = re.sub('(?sm){(.*?):\s*(http://.*)}',
                 '<img src="\\2" alt="\\1">', txt)
    # Convert regular URL's
    txt = re.sub('(?:[^="])((?:http|ftp|file)://(?:[^ \n\r<\)]+))(\s)',
                 '<a href="\\1">\\1</a>\\2', txt)
    return txt

def Typography(txt):
    rc = re.compile     # cut down line length
    MS = re.M | re.S
    # [module] names
    r = rc(r"""([\(\s'/">]|^)\[(.*?)\]([<\s\.\),:;"'?!/-])""", MS)
    txt = r.sub('\\1<i class="module">\\2</i>\\3',txt)
    # *strongly emphasize* words
    r = rc(r"""([\(\s'/"]|^)\*(.*?)\*( [\s\.\),:;'"?!/-])""", MS)
    txt = r.sub('\\1<strong>\\2</strong>\\3', txt)
    # -emphasize- words
    r = rc(r"""([\(\s'/"]|^)-(.+?)-( [\s\.\),:;"'?!/])""", MS)
    txt = r.sub('\\1<em>\\2</em>\\3', txt)
    # _Book Title_ citations
    r = rc(r"""([\(\s'/"]|^)_(.*?)_( [\s\.\),:;'"?!/-])""", MS)
    txt = r.sub('\\1<cite>\\2</cite>\\3', txt)
    # 'Function()' names
    r = rc(r"""([\(\s/"]|^)'(.*?)'([\s\.\),:;"?!/-])""", MS)
    txt = r.sub("\\1<code>\\2</code>\\3", txt)
    # 'library. func() ' names
    r = rc(r"""([\(\s/"]|^)'(.*?)'([\s\.\),:;"?!/-])""", MS)
    txt = r.sub('\\1<i clas    s="libfunc">\\2</i>\\3', txt)
    return txt

if __name__ == '__main__':
    blocks = Make_Blocks()
    if len(sys.argv) > 1:
    Process_Blocks(blocks, title=sys.argv[1])