Pages Menu
Categories Menu

Posted by on Mar 29, 2019 in GIS, What's new | 6 comments

How-to: harvesting large quantity of data from ArcGIS REST services using a new tool!

How-to: harvesting large quantity of data from ArcGIS REST services using a new tool!

If you have ever worked with ArcGIS REST Services, either through ArcGIS Online or ArcGIS Server, you might have faced a limitation of 1000 features by response, and you wouldn’t be the only one! Googling (or DuckDuckGoing) “1000 limit on ArcGIS REST service” would already give you a good idea of how much this has been a recurring issue for users for quite some time.

This limitation is defined by the maxRecordCount service property and it is set to 1000 by default. As mentioned by Esri, this implies that only that specific number of “records can be returned by the server to a client for any given query operation”. Therefore, as users, we are unable to access a whole dataset with a single query, making us search for workarounds which could allow to perform either a global analysis of the dataset or operations needing access to all records at once, like a “Find Closest Facility”.

Why are services de facto limited?

There are some valid reasons explaining such default value. The main one is to ensure an acceptable performance level for both clients but also servers. As it is explained by Esri and other sources, the data we are handling is quite heavy and take a lot of memory space in one’s browser, especially if we are talking about polygons. Without the 1,000 graphics limit, we could easily reach the capacity of the server to process and send requests, and the user’s browser to smoothly display a map including so many features.

Having said that, some of the complains on the matter are dated from more than five years ago, so one could wonder if the technology hasn’t advanced enough to increase the 1,000 graphics limit. Besides, the limitation was built around the idea of a worst-case scenario, i.e. using a polygon layer, when one often only needs points (a couple of numbers) or simple tabular data. In any case, if you are the data publisher, it is hard to say what could be a reasonable number today, as it depends on each environment and use case.

Yes… but give me all!

One shouldn’t worry! Even if we cannot increase the limitation of the service we are using, either because of technical constraints or because we are not the server maintainers, there are still several methods to query all the data at once.

1- Grouped object IDs approach

One of them, which is mentioned by Esri, has to do with performing sequential queries to groups of 1000 Objectids (or as many Objectids as the limitation is set). Such a process is described by Ken Doman on his blog.

“While there is a limit to the number of features included in the feature set response, there is no limit to the number of object IDs returned in the ID array response. Clients can exploit this to get all the query conforming object IDs by specifying returnIdsOnly=true and subsequently requesting feature sets for subsets of object IDs.” (Esri).

2- The resultOffset approach

A second method that we have been using at CartONG is to take advantage of the resultOffset query parameter (available since 10.3), which allows to “skip the specified number of records and [to return response] starting from the next record” (Esri). If we iterate queries, while each time adding the number of returned records (which should match with the service limitation) to the resultOffset parameter, we then end up obtaining the full dataset. This is exactly what is done by the ArcGIS REST Service export tool created by CartONG and which is introduced in the next section.

Anyone interested on using or consulting the source code of the tool can have a look at its GitHub repository. Special mention to the CartONG.ArcGISService.js file, where the implementation of the method explained above is coded, (CartONG.ArcgisService.prototype.query function).

Introducing the ArcGIS REST Service export tool

Here below is a short introduction on how to best use the tool developed by CartONG to help users extract datasets from ArcGIS services. As of today, the layout is quite simple and limited in terms of options, with only three input options for the user. Yet we hope to keep developing the tool to add further functionalities in the future.

image 2 blog post V2

1- If needed, change the default print format given to any of the available ones (Esri JSON, GeoJSON and CSV as of today). This is the format in which the results will be showing in the grey box below (“Print results”).

2- Optionally, if you don’t need geometries, set Return geometry to “No”, as it will slow down the response time. Keep in mind that polygon services are quite heavy and response could take a long time.

3- Enter the URL of the Map Service layer of interest, including its layer index (the number at the end of the URL below), and click on the search button on the right or press “Enter” on your keyboard.

  • Careful: make sure you don’t add a trailing slash (/).
  • Also, if you don’t know the index of the layer, access the service URL and check the index of available layers.

image 3 blog post V2

4- After a while, if the query is successful, the results will be inputted in the grey box.

image 4 blog post
5- Finally, if necessary, an export option is available in two formats, the elected one at step 1 (or Esri JSON if CSV was chosen) and CSV.

image 5 blog post

ArcGIS services are used in all sectors, be it the private, public or humanitarian one. Overcoming the service limits does thus imply a big advantage for any use case. This is why the ArcGIS REST Service Export tool is a truly cross-cutting support tool for data consumers.



  1. hi super good development.
    we are contemplating to use this as it offers to by pass the ma xlimit and we could imagine to built locally gloabal dataset. (eg the whole UNHCR camp position)

    We see a lot of interest as development you could even imagine to position this service via Api: then we read the REST services via your link and your APi and we get a real live data with the whole dataset! ?

    keep going.



    • Hi Yann! Thanks a lot for your feedback and sorry for the slow answer. I’m happy this tool is useful for you. We will try to make some updates in the future to improve it.
      Regarding your suggestion of having a service like this, it is not in our plans so far, but it is of course a good idea to have in mind.

    • Hi, yes, I get the error as well… It seems it’s blocked because the service is under http and not https. Is it your own ArcGIS Server? Could it be upgraded?
      EDIT: I see that your service has ‘maxRecordCount’ parameter set to 50,000 features. Please have in mind that if your table does not contain more features than that, you could directly use ArcGIS REST API, without going through our export tool.

  2. I get the following error:

    Service error: Error performing query operation. Service description:
    “version”: 10.8,
    “type”: “Feature Layer”,
    “capabilities”: “Map,Query,Data”,
    “supportedQueryFormats”: “JSON, geoJSON”,
    “maxRecordCount”: 1000

    • Hi Gleb,
      Thank you for using the tool.
      Could you please tell us which is the service you are using so that we could replicate the issue?

Post a Reply

Your email address will not be published. Required fields are marked *

🚚 📦 We're moving!

[French below]

The content of the CartoBlog was transferred to the Information Management Resource Portal in May 2021:

Please note that the CartoBlog will be decommissioned in early 2022. From now on, we warmly encourage you to consult the tutorials, benchmarking studies and other resources directly onto the new portal.

Thank you!

🚚 📦 On déménage !

Le contenu du CartoBlog a été transféré sur l’IM Resource Portal en mai 2021 :

Veuillez noter que le CartoBlog sera désactivé au début de l’année 2022. Nous vous invitons, dès à présent, à prendre l’habitude de consulter les tutoriels, benchmarkings et autres ressources directement sur le nouveau portail.

Merci !