Recipe 20.17 Resuming an HTTP GET

20.17.1 Problem

You have part of a file and want to download the rest without refetching the content you already have. For example, your initial download was interrupted, so you want to complete it.

20.17.2 Solution

Use the HTTP 1.1 Range header in your GET request:

use LWP;
$have = length($file);
$response = $ua->get($URL,
                     'Range', "bytes=$have-");
# $response->content hold the rest of the file

20.17.3 Discussion

The Range header lets you specify which bytes to fetch. The 0^th byte is the first in the file, so the range "bytes=0-" fetches the whole file.

You can also specify a range with two endpoints: "0-25", for example, fetches the first 26 bytes of the file. If you want to fetch an interior range, use "26-99".

Some servers don't support ranges, even though they claim to understand HTTP 1.1. In this case you'll be sent the whole file, not the range you asked for. To detect this, use HEAD to see the size of the file and then use a GET with a range to fetch the rest. If the content in the GET response is the same length as the original file, your range was ignored.

Here is the full list of ranges possible in the HTTP 1.1 specification:

The HTTP specification also permits lists of ranges (e.g., "0-5,10-15,20-"). This returns a multipart response.

20.17.4 See Also

LWP documentation; the HTTP spec at http://www.ietf.org/rfc/rfc2616.txt