Pages Menu
TwitterRssFacebook
Categories Menu

Posted by on Mar 11, 2016 in Information Management, Mobile Technologies, What's new | 2 comments

Tutorial: Migration path on ODK-based platforms

Tutorial: Migration path on ODK-based platforms

Let’s say you have a few projects currently running on an ODK-type platform and you would like to switch to another one. This could be for example older forms on Formhub, that you would now migrate to a supported and actively developed platform such as Kobo or ONA, or you may want to transfer data from an offline server (such as Nafundi’s VM) to an online centralized account. How can you accomplish this fairly quickly and ensure that your data travel with your form, without forcing a break and having to keep track of them in 2 different places?

Chances are you can accomplish this using Briefcase.

The general steps are the same across platform, although details will vary. The only real prerequisite is that both the original server and the target server must be Briefcase compatible. This isn’t very restrictive, since most of the platforms in the ODK environment are. You can find here the URL to use with Briefcase in each case:

  • Kobo: https://kc.humanitarianresponse.info/MY_ACCOUNT_NAME
  • ONA: https://odk.ona.io/MY_ACCOUNT_NAME
  • Aggregate (VM or other installations): will depend on your installation
  • Formhub: https://formhub.org/MY_ACCOUNT_NAME

Warning: as with any major transfer of data, ensure that you have everything backed-up. Keep copies of your data outside the platforms where you will be performing the migration. Also, as a general rule, you should test the result before you call it a success and start submitting real data – what about filling a test form and trying to submit to your new account before you tell everyone to submit already? You can always delete that test submission once you’re sure that it worked. You should also check that the number of submissions on your new server matches the number of submission on the old one for each form.

General steps outline

  1. Pulling data from your original data repository
    1. Use Briefcase to download your data locally.
    2. If the original server is ONA, Kobo or Formhub, you should retrieve the XLS form definition as well, because Briefcase only deals with the XML form definition, which is harder to edit.
  2. Upload the XLS forms to the new platform (so that it is still “attached” to the data and doesn’t have to be managed separately)
  3. Push the raw data using Briefcase
  4. Repeat for all forms of interest

Detailed steps – From Formhub to Kobo

This will be our case for demonstration, however the general procedure would be very similar between other ODK platforms – the only real difference will be the URL you enter in Briefcase when you connect it to your server to pull or push data (and the username/password for each account). If you are moving to or from Aggregate, however, you would not be able to keep the XLS form definition on the account because Aggregate only works with the XML format. You can then ignore all the steps involving the XLS form below and will have to manage it separately.

Pulling original data

Using Briefcase (preferably the latest version, available here), you’ll have to enter the URL and credentials to log into your former account. The URL will vary according to the platform (see above for references), but generally speaking it would have the same structure:

  • https://ROOT_URL/MY_ACCOUNT_NAME
1

Figure 1: Connecting to your former account

Hint: you may have to try http or https. Most platforms don’t need a “/” at the end after “MY_ACCOUNT_NAME”. Some will work even if you add one, however others will refuse the connection. Just replace “MY_ACCOUNT_NAME” by your actual username.

If successful, Briefcase will tell you:

2

Figure 2: Resounding success!

 

The following step may be optional, but quite useful if you want to avoid, a few months down the road, to have to write frantic emails to former co-workers who may then be half a world away in order to retrieve the XLS forms because you really need to make a new version and just can’t seem to find that XLS form and your XML skills just aren’t what they used to be… If you are working with Formhub, you might have experienced that the user interface is slow if the traffic is high. However, you don’t need to click on the screen to recover the data: you can simply copy a correctly formatted URL in any web browser, which will result in the file appearing in your download folder. The general format of that URL is:

  • https://formhub.org/MY_ACCOUNT_NAME/forms/MY_FORM_ID/form.xls
    • Where “MY_ACCOUNT_NAME” is the same name that was entered in Briefcase earlier
    • “MY_FORM_ID” also appear in Briefcase earlier, in our demonstration it was “Laboratory_Evalutaion_v1_15”. It is the same ID that appear under the “Settings” tab of your XLS form

3

4

Figure 3: Our URL to download the form definition with a minimal amount of clicking involved.

 

Figure 4: A successful download on Chrome.

You can repeat this with any other forms for which you would like to obtain the XLS form definition.

The next detail is important if you want this plan to work: you must upload this XLS form definition to your new account first, before you push the data you got with Briefcase. On Kobo, after you upload it and go to your project page:

Figure 5: Our XLS form has been uploaded (Kobo)

We have no data (the “0” on the right is the number of submissions). We’re now going to push the data using Briefcase on our new account, to this new form definition that currently is empty. We’ll go to the “PUSH” tab, then click “Connect…” but this time enter the credentials and URL of our new account. Then, select the forms you want to push, and click “Push” (bottom right).

Figure 6: Pushing to Kobo

 Briefcase will warn us that the upload is only partially successful:

Figure 7: « Partially successful », but everything we want is there.

 There’s no need to worry: Briefcase thinks it is only partially successful, because when it tried to push the form definition (XML) to our new account, the new account refused it because it already had one such form definition named “Laboratory_Evaluation_v1_15” (the XLS form we prefer). But we can see that the data has indeed been submitted, as is evidenced below by the submission number (on the right) that went from 0 (above) to 10:

Figure 8: All our 10 submissions have been added (right)

Similar steps can also be taken to consolidate data for the same form, but that was stored on different servers – for example if you are working with Nafundi’s VM offline and would like to centralize those (local) data. You would only need to replace the URL used to connect to the server with Briefcase. For example, if using ONA:

Figure 9:Connecting Briefcase with ONA

For Nafundi’s VM, you would enter the URL that the black screen shows on startup, likely similar to this:

Figure 10:Connecting Briefcase with Nafundi’s VM

Depending on how you have setup your users on Aggregate (when you log into the browser to access Aggregate), you may be able to forget username & password.

Good luck with your new account! :)

2 Comments

  1. My server I use Nafundi’s odk aggregate VM. And it keeps telling me that the VM does not have IP. How do I access my data otherwise.

    • Hey Alfa,

      Well there can be a few reason why this happens. But generally it should be linked to the specific wifi network you are using. For example, are you trying to start the VM on a new network that you have not used before? If so, is it a larger office network? Because sometimes network will have security settings that may prevent the VM from obtaining an IP address (such as 192.168.0.111 or similar) to which you can connect.

      If not, it may be linked to your computer’s firewall. You may try to temporarily disable your firewall (make sure to start it again after). How to do this depends on your specific computer, but you can certainly find information on this online.

      Hopefully we can help solve your problem,

      Francis

Post a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

A propos de l'auteur :

Francis Vachon

With a background physics engineering, Francis is working as information management at CartONG... when he's not managing reforestation teams in Québec!