How can I create automated data downloads?
Ordnance Survey is one of the world’s oldest data organisations. We have gathered information about Great Britain since 1791, organising, analysing and disseminating maps and other information. As information technologies have evolved from tables and charts and pen-and-paper calculations to databases, vector graphics and sophisticated machine learning algorithms, OS has been a leader in adopting and innovating new ways to capture, store, process and share valuable location information.
The challenge – managing data
As the scale of data grows, so does its potential usefulness – and the complexity of managing it. Understanding a small table of numbers is easy, less so for a complex database with dozens of tables and millions of rows. With code though, it has never been easier for programmers to retrieve and manipulate large volumes of data and extract and understand the valuable information they contain.
As every data scientist and web developer knows, well-structured datasets are difficult to find and even harder to create. Data wrangling is a major task and is often overlooked by those less familiar with the data science process. For algorithms to extract accurate and meaningful information from a dataset – especially using a powerful analysis libraries openly available like Python’s pandas – the data needs to be consistently structured and clean.
The answer – automated data downloads
Consistent with our tradition of working to make accurate and authoritative location data about Great Britain useful to our users and customers, we’ve been making a wide range of our datasets available for download over the web. Through the new OS Data Hub, users can connect to a number of APIs including the OS Maps, OS Vector Tile, OS Features, OS Names, OS Linked Identifiers and OS Downloads APIs. These resources range from raster and vector map tiles to vector features, naming and identification data or even full OS datasets.
With the OS Maps, OS Vector Tile and OS Features APIs, users authenticate their request with an API key created on their OS Data Hub account where they can dynamically fetch the data they need when they need it. This is powerfully useful in web maps, data science workflows and desktop GIS applications. These APIs enable developers and spatial analysts to create and customise interactive maps, and to fetch location data with rich attribution – information about a feature useful for visualisation and analysis purposes.
The OS Downloads API makes a number of OS OpenData datasets available for automated download over the internet. This new service makes collecting useful spatial data from Ordnance Survey as easy as pasting a URL, or sending an HTTP GET request. The OS Downloads API documentation has sample code, and we’ve created a tutorial to help JavaScript developers fetch data programmatically with NodeJS.
Unlocking value by improving access
The OS Data Hub APIs are the result of a major effort at OS to make our National Geographic Database easier to access than ever before. A broad range of users in Great Britain from civil servants to NGO workers, academic researchers, entrepreneurs and large companies, could enhance the quality and efficiency of their work with the insights they could derive from OS data. We’re working hard to make sure the path from data to decision-making is as short as possible. Automated data downloads are one of the ways we’re doing that.
We’ve written a tutorial to get you started with automated data downloads from the OS Downloads API using NodeJS – find it here.
Get involved
Do you use OS data and want to learn if the OS Data Hub APIs could help? Or do you think that having access to a high resolution dataset with over 500 million features in Britain would be useful? Sign up for the OS Data Hub or reach out to us on Twitter.
Explore the OS Data Hub
With over 20,000 updates to our databases every day, you have access to a comprehensive and up-to-date set of authoritative data through the OS Data Hub
Our highly accurate geospatial data and printed maps help individuals, governments and companies to understand the world, both in Britain and overseas.