Pages Menu
TwitterRssFacebook
Categories Menu

Posted by on Mar 29, 2019 in GIS, What's new | 1 comment

How-to: harvesting large quantity of data from ArcGIS REST services using a new tool!

How-to: harvesting large quantity of data from ArcGIS REST services using a new tool!

If you have ever worked with ArcGIS REST Services, either through ArcGIS Online or ArcGIS Server, you might have faced a limitation of 1000 features by response, and you wouldn’t be the only one! Googling (or DuckDuckGoing) “1000 limit on ArcGIS REST service” would already give you a good idea of how much this has been a recurring issue for users for quite some time.

This limitation is defined by the maxRecordCount service property and it is set to 1000 by default. As mentioned by Esri, this implies that only that specific number of “records can be returned by the server to a client for any given query operation”. Therefore, as users, we are unable to access a whole dataset with a single query, making us search for workarounds which could allow to perform either a global analysis of the dataset or operations needing access to all records at once, like a “Find Closest Facility”.

Why are services de facto limited?

There are some valid reasons explaining such default value. The main one is to ensure an acceptable performance level for both clients but also servers. As it is explained by Esri and other sources, the data we are handling is quite heavy and take a lot of memory space in one’s browser, especially if we are talking about polygons. Without the 1,000 graphics limit, we could easily reach the capacity of the server to process and send requests, and the user’s browser to smoothly display a map including so many features.

Having said that, some of the complains on the matter are dated from more than five years ago, so one could wonder if the technology hasn’t advanced enough to increase the 1,000 graphics limit. Besides, the limitation was built around the idea of a worst-case scenario, i.e. using a polygon layer, when one often only needs points (a couple of numbers) or simple tabular data. In any case, if you are the data publisher, it is hard to say what could be a reasonable number today, as it depends on each environment and use case.

Yes… but give me all!

One shouldn’t worry! Even if we cannot increase the limitation of the service we are using, either because of technical constraints or because we are not the server maintainers, there are still several methods to query all the data at once.

1- Grouped object IDs approach

One of them, which is mentioned by Esri, has to do with performing sequential queries to groups of 1000 Objectids (or as many Objectids as the limitation is set). Such a process is described by Ken Doman on his blog.

“While there is a limit to the number of features included in the feature set response, there is no limit to the number of object IDs returned in the ID array response. Clients can exploit this to get all the query conforming object IDs by specifying returnIdsOnly=true and subsequently requesting feature sets for subsets of object IDs.” (Esri).

2- The resultOffset approach

A second method that we have been using at CartONG is to take advantage of the resultOffset query parameter (available since 10.3), which allows to “skip the specified number of records and [to return response] starting from the next record” (Esri). If we iterate queries, while each time adding the number of returned records (which should match with the service limitation) to the resultOffset parameter, we then end up obtaining the full dataset. This is exactly what is done by the ArcGIS REST Service export tool created by CartONG and which is introduced in the next section.

Anyone interested on using or consulting the source code of the tool can have a look at its GitHub repository. Special mention to the CartONG.ArcGISService.js file, where the implementation of the method explained above is coded, (CartONG.ArcgisService.prototype.query function).

Introducing the ArcGIS REST Service export tool

Here below is a short introduction on how to best use the tool developed by CartONG to help users extract datasets from ArcGIS services. As of today, the layout is quite simple and limited in terms of options, with only three input options for the user. Yet we hope to keep developing the tool to add further functionalities in the future.

image 2 blog post V2

1- If needed, change the default print format given to any of the available ones (Esri JSON, GeoJSON and CSV as of today). This is the format in which the results will be showing in the grey box below (“Print results”).

2- Optionally, if you don’t need geometries, set Return geometry to “No”, as it will slow down the response time. Keep in mind that polygon services are quite heavy and response could take a long time.

3- Enter the URL of the Map Service layer of interest, including its layer index (the number at the end of the URL below), and click on the search button on the right or press “Enter” on your keyboard.

  • Careful: make sure you don’t add a trailing slash (/).
  • Also, if you don’t know the index of the layer, access the service URL and check the index of available layers.

image 3 blog post V2

4- After a while, if the query is successful, the results will be inputted in the grey box.

image 4 blog post
5- Finally, if necessary, an export option is available in two formats, the elected one at step 1 (or Esri JSON if CSV was chosen) and CSV.

image 5 blog post

ArcGIS services are used in all sectors, be it the private, public or humanitarian one. Overcoming the service limits does thus imply a big advantage for any use case. This is why the ArcGIS REST Service Export tool is a truly cross-cutting support tool for data consumers.

 

1 Comment

  1. hi super good development.
    we are contemplating to use this as it offers to by pass the ma xlimit and we could imagine to built locally gloabal dataset. (eg the whole UNHCR camp position)

    We see a lot of interest as development you could even imagine to position this service via Api: then we read the REST services via your link and your APi and we get a real live data with the whole dataset! ?

    keep going.
    Y

    Best

    Y

Post a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

A propos de l'auteur :

Miguel Moreno Gonzalez

Miguel joined CartONG in 2015. He implements web applications for CartONG's partners. Miguel has a background in geomatics and is passionate about cartography and facilitating the understanding and exploitation of data through tools like GIS and computer programming.