Tutorial: Migration path on ODK-based platforms
Let’s say you have a few projects currently running on an ODK-type platform and you would like to switch to another one. This could be for example older forms on Formhub, that you would now migrate to a supported and actively developed platform such as Kobo or ONA, or you may want to transfer data from an offline server (such as Nafundi’s VM) to an online centralized account. How can you accomplish this fairly quickly and ensure that your data travel with your form, without forcing a break and having to keep track of them in 2 different places?
Chances are you can accomplish this using Briefcase.
The general steps are the same across platform, although details will vary. The only real prerequisite is that both the original server and the target server must be Briefcase compatible. This isn’t very restrictive, since most of the platforms in the ODK environment are. You can find here the URL to use with Briefcase in each case:
- Kobo: https://kc.humanitarianresponse.info/MY_ACCOUNT_NAME
- ONA: https://odk.ona.io/MY_ACCOUNT_NAME
- Aggregate (VM or other installations): will depend on your installation
- Formhub: https://formhub.org/MY_ACCOUNT_NAME
Warning: as with any major transfer of data, ensure that you have everything backed-up. Keep copies of your data outside the platforms where you will be performing the migration. Also, as a general rule, you should test the result before you call it a success and start submitting real data – what about filling a test form and trying to submit to your new account before you tell everyone to submit already? You can always delete that test submission once you’re sure that it worked. You should also check that the number of submissions on your new server matches the number of submission on the old one for each form.
General steps outline
- Pulling data from your original data repository
- Use Briefcase to download your data locally.
- If the original server is ONA, Kobo or Formhub, you should retrieve the XLS form definition as well, because Briefcase only deals with the XML form definition, which is harder to edit.
- Upload the XLS forms to the new platform (so that it is still “attached” to the data and doesn’t have to be managed separately)
- Push the raw data using Briefcase
- Repeat for all forms of interest
Detailed steps – From Formhub to Kobo
This will be our case for demonstration, however the general procedure would be very similar between other ODK platforms – the only real difference will be the URL you enter in Briefcase when you connect it to your server to pull or push data (and the username/password for each account). If you are moving to or from Aggregate, however, you would not be able to keep the XLS form definition on the account because Aggregate only works with the XML format. You can then ignore all the steps involving the XLS form below and will have to manage it separately.
Pulling original data
Using Briefcase (preferably the latest version, available here), you’ll have to enter the URL and credentials to log into your former account. The URL will vary according to the platform (see above for references), but generally speaking it would have the same structure:
Hint: you may have to try http or https. Most platforms don’t need a “/” at the end after “MY_ACCOUNT_NAME”. Some will work even if you add one, however others will refuse the connection. Just replace “MY_ACCOUNT_NAME” by your actual username.
If successful, Briefcase will tell you:
The following step may be optional, but quite useful if you want to avoid, a few months down the road, to have to write frantic emails to former co-workers who may then be half a world away in order to retrieve the XLS forms because you really need to make a new version and just can’t seem to find that XLS form and your XML skills just aren’t what they used to be… If you are working with Formhub, you might have experienced that the user interface is slow if the traffic is high. However, you don’t need to click on the screen to recover the data: you can simply copy a correctly formatted URL in any web browser, which will result in the file appearing in your download folder. The general format of that URL is:
- Where “MY_ACCOUNT_NAME” is the same name that was entered in Briefcase earlier
- “MY_FORM_ID” also appear in Briefcase earlier, in our demonstration it was “Laboratory_Evalutaion_v1_15”. It is the same ID that appear under the “Settings” tab of your XLS form
You can repeat this with any other forms for which you would like to obtain the XLS form definition.
The next detail is important if you want this plan to work: you must upload this XLS form definition to your new account first, before you push the data you got with Briefcase. On Kobo, after you upload it and go to your project page:
We have no data (the “0” on the right is the number of submissions). We’re now going to push the data using Briefcase on our new account, to this new form definition that currently is empty. We’ll go to the “PUSH” tab, then click “Connect…” but this time enter the credentials and URL of our new account. Then, select the forms you want to push, and click “Push” (bottom right).
Briefcase will warn us that the upload is only partially successful:
There’s no need to worry: Briefcase thinks it is only partially successful, because when it tried to push the form definition (XML) to our new account, the new account refused it because it already had one such form definition named “Laboratory_Evaluation_v1_15” (the XLS form we prefer). But we can see that the data has indeed been submitted, as is evidenced below by the submission number (on the right) that went from 0 (above) to 10:
Similar steps can also be taken to consolidate data for the same form, but that was stored on different servers – for example if you are working with Nafundi’s VM offline and would like to centralize those (local) data. You would only need to replace the URL used to connect to the server with Briefcase. For example, if using ONA:
For Nafundi’s VM, you would enter the URL that the black screen shows on startup, likely similar to this:
Depending on how you have setup your users on Aggregate (when you log into the browser to access Aggregate), you may be able to forget username & password.
Good luck with your new account! 🙂