Automating OS Premium data downloads

This guide takes you though the process of automating the download of OS Premium data downloads.

What you need

• A basic understanding of automatic processing of JSON data and basic procedural (if-then-else) programming.

• OS Downloads API added to an API project in the OS Data Hub with an API Key. See Getting started with an API project for more information.

• A text editor like Visual Studio Code

• A working installation of Node.js and the popular axios module.

Please read the download support documentation before working with data packages.

Downloading data packages

These are generic instructions intended to broadly outline the necessary steps. Please see Sample implementation using Node.js below for one possible implementation.

1. Call the /dataPackages endpoint to obtain a list of data packages and their associated IDs.

   • This is similar to a manual download in the you first need to discover which data packages are available. Be mindful that list items can change.

2. Interrogate the JSON response to identify the data package you are interested in. Each entry in the JSON response provides data package metadata, as well as information about available versions of the data package.

   • You may at this point wish to create a loop that iterates through all available data packages or a subset that you are interested in based on common attributes (for example, vector data in a certain format).

   • You could store the data package id and version ids. That way you can easily check periodically for new versions of the data package being available, simply by comparing the versions we provide with the ones that you hold.

3. Using the data package id from the previous response, make a call to the /dataPackages/{dataPackageId}/versions/latest endpoint.

   • This will return information about the most recent data package version, including a list of the files that are available to download.

4. Once you obtain the download links for the data package version you are ready to download the data.

Sample implementation using Node.js

This example relies on a working installation of Node.js and the popular axios module.

1. Get the list of available data packages including basic information such as their product id and versions.

   const axios = require('axios');
   const apiKey = 'Insert API key here';
   
   async function getList() {
       const dataPackageList = await axios({
           url: 'https://api.os.uk/downloads/v1/dataPackages',
           headers: {
               key: apiKey
           }
       });
       /* For explanation and debugging purposes we display the full response from the API in the console */
       console.log(dataPackageList.data);
       for (const dataPackage of dataPackageList.data) {
           /* This will print out the url of the data package id which can be used in another request. This would be the
              ideal place to call another function which continues the processing or filters down the results to the set required. */
           console.log(dataPackage.url);
       }
   }
   getList();

2. Obtain information for a specific product. While the key parts of this are already provided in step one, this shows another way of getting to specific detail.

   const axios = require('axios');
   
   /* This function will return information about a specific data package.
      You will need to insert both a dataPackageId and API key to complete the implementation. */
   const apiKey = 'Insert API key here';
   const dataPackageId = 'Insert data package id here';
   
   async function getDataPackageDetails() {
       const dataPackageDetails = await axios({
           url: 'https://api.os.uk/downloads/v1/dataPackages/' + dataPackageId,
           headers: {
               key: apiKey
           }
       });
       console.log(dataPackageDetails.data);
       /* At this point we could insert another function to process the results or act on them/download them */
   }
   getDataPackageDetails();

3. Download the data. In this example we hard code the data package id that we are interested in. This can also be provided dynamically into the function allowing the same function to be used for multiple products and formats.

   const fs = require('fs');
   const axios = require('axios');
   
   const apiKey = 'Insert API key here';
   const dataPackageId = 'Insert data package id here';
   
   /* ============================================================
   Function: Uses Axios to download file as stream using Promise
   ============================================================ */
   const download_file = (url, fileName) =>
       axios({
           url,
           headers: {
               key: apiKey
           },
           responseType: 'stream'
       }).then(
           response =>
               new Promise((resolve, reject) => {
                   response.data
                       .pipe(fs.createWriteStream(fileName))
                       .on('finish', () => resolve())
                       .on('error', e => reject(e));
               }
       )
   );
   
   /* ============================================================
   Download all of the files in the latest data package version
   ============================================================ */
   async function downloadFiles() {
       try {
           const downloadInfo = await axios({
               url: 'https://api.os.uk/downloads/v1/dataPackages/' + dataPackageId + '/versions/latest',
               headers: {
                   key: apiKey
               }
           });
           for (const download of downloadInfo.data.downloads) {
               let downloadFile = await download_file(download.url, download.fileName);
               console.log(`Downloaded file ${download.fileName}`);
           }
           console.log('Completed downloading files');
       } catch (error) {
           console.error(error);
       }
   }
   
   downloadFiles();