Hack 94 Cache AWS Responses Locally

You can improve application performance by saving Amazon data locally and updating it on a regular schedule.

If you find your application requesting the same data from Amazon over and over again, you may be able to speed things up by caching the data locally. Why make the same request for live data again and again if you're getting the same data each time? A local version will always be faster. Also, if Amazon's server happens to be down for maintenance, you can rely on your local cache to make up for it.

As mentioned in the previous hack, Amazon requires that the data you display on your site be up to date. The Web Services Licensing Agreement says that data must be updated every 24 hours. If you want to cache data for longer periods of time, you'll need a written agreement from Amazon.

Adding a data cache requires just a few lines of extra code and can make your applications much more efficient. There are many approaches to caching data; the code here shows two different ways to go about it.

94.1 Cache in Memory with ASP

This ASP code stores the Amazon XML response as an Application variable, which means the code is available to the entire application in memory. Storing data in memory makes it available for quick access, but memory is a limited commodity. This is a great solution if you're storing only a few responses locally.

Along with the cached XML, this code sets a DateCached application variable that stores when the data was last saved. A check at the top of the file against the current time (Now( )) plus 24 hours determines whether new data should be cached.

<%
strURL = "http://xml.amazon.com/onca/xml3?t=insert associate tag"
strURL = strURL & "&dev-t=insert developer token"
strURL = strURL & "&type=lite&f=xml&AsinSearch=0596005423"

'If cached XML doesn't exist or is old, make a request for new XML.
If Application(strURL) = "" OR DatePart("h",Application("DateCached")) < 
DateAdd("h",24,Now(  )) Then

    'Make XML/HTTP request
    Set xmlhttp = Server.CreateObject("Msxml2.SERVERXMLHTTP")
    xmlhttp.Open "GET", strURL, false
    xmlhttp.Send(  )
    Set AllXML = xmlhttp.responseXML
    
    'If the XML looks good...
    If AllXML.parseError.ErrorCode = 0 Then

        'Set the XML to an application variable
        Application(strURL) = AllXML.xml
        
        'And set the time it was cached
        Application("DateCached") = Now(  )        

    End If

   Set AllXML = Nothing
   Set xmlhttp = Nothing
End If
response.write "<xmp>" & Application(strURL) & "</xmp>"
%>

This code can be incorporated into any existing ASP file, and the Application variable that holds the XML can be used as if it were a fresh response from Amazon.

94.2 Cache to Disk with Perl

This bit of Perl code by Jeff Barr saves XML responses as a local text file. Before making a request, it checks to see if the cached version has been saved within the last 24 hours. If not, it writes the data to the cache file. Using local text files as your cache is more scalable than saving the responses in memory, especially if you have more than a handful of responses you'd like to cache.

Be sure to set the value of $cache_dir to the local directory you want to store the cache files in.

use LWP;
use Digest::MD5 qw/md5_base64/;

my $LWP_request;
my $LWP = new LWP::UserAgent;

my $cache_dir = "C:\\";
my $URL = "http://xml.amazon.com/onca/xml3?t=insert associate tag [RETURN]
&dev-t=insert developer token&type=lite&f=xml&AsinSearch=0596005423";

# Check for cached data (URL is cache key)
  my $used_cache = 0;
  my $cache_file = $cache_dir . "\\" . md5_base64($URL);
  print $cache_file . "\n\n";
  if (-e $cache_file && (-M $cache_file <= 1.0/24.0))
  {
    # Use the cached data
    if (open(CACHE_FILE, $cache_file))
    {
      my @all_xml = <CACHE_FILE>;
      close CACHE_FILE;

      $xml_text   = join "\n", @all_xml;
      $used_cache = 1;
    }
  }

# If the cache doesn't exist, make the request
  if (!$used_cache)
  {
    # Request the data & process the response
    $LWP_request = new HTTP::Request(GET => $URL);
    my $result = $LWP->request($LWP_request);
    if (!$result->is_success)
    {
      # Handle error. Could use stale cached data if desired.
    }
    else
    {
      $xml_text = $result->content;
    } 
  }

  # Update the cache
  if (!$used_cache)
  {
    if (open(CACHE_FILE, ">$cache_file"))
    {
      print CACHE_FILE $xml_text;      
    }
   
    close CACHE_FILE;
  }

print $xml_text;

This code just prints out the XML, cached or not. But you could easily incorporate this caching method into your existing AWS applications.