Tuto: Conceiving a survey in the humanitarian and development field – Part 2
This post is the follow up to a previous blog post ( What to do before you start listing the questions!) that you can find here.
Part 2: Now you can start listing the questions!
“If you do not know how to ask the right question, you discover nothing.” (W. Edward Deming)
Now, and only now, however much you wanted to before (!) you can take a sheet of paper and start listing the questions that are needed to fill in your desired indicators. Think them through carefully! Make sure that your survey flows logically from one question to another and also from one group of questions to another.
Some of the following might seem obvious but it can sometimes be a little hard to take the necessary distance when one is conceiving a survey- here is therefore a checklist to keep in mind when one thinks out a survey’s questions and answers:
Beyond this check list, here are three aspects that I’d like to zoom into (so I can say “I told you” some day 😉 ):
Avoid making mandatory questions a bad habit…
Making questions mandatory is always a big debate and the only answer one can give is that it depends on the context. Some surveys can have most or all questions mandatory without it being a problem (for example a KAP – Knowledge, Attitude and Practices – survey that has a scoring system to evaluate a health center’s progress over time). However, always think about it first to make sure it’s an informed decision, as there are different reasons why you might sometimes deliberately want not to make a question mandatory:
- If for technical reasons there might be situations where the data can’t be captured (e.g. GPS points can sometimes be problematic independently of your enumerator…);
- If you are not sure that your possible answers to a question are comprehensive – although this can be bypassed by adding options such as “don’t know”, “none of the above”, “other”, “N/A” that fit the case. Don’t forget to plan for the future if this is a long-term survey, by leaving ways of opting out: for example, a list of enumerators or beneficiaries can change – if you have not worked out an adequate coding system, make sure that you have an “other” option that a new staff can tick to be identified easily;
- If the filling in of your survey depends on different people, that cannot all always be available at the proper moment, which can be frequent during Household surveys (another solution is to make sure that your tool’s settings render it possible to skip a mandatory question at least until you validate the form, so as to be able to save it and come back later);
- People who run into issues with particular questions, especially those required to complete the survey, may provide false information just to get past your survey error.
If for a given question that you can’t in fact make mandatory for one of the reasons above you want to encourage the enumerator to get an answer nonetheless, you can also add a mandatory question like “does the household have a hand-washing station?” before asking a mandatory “Please take a picture” with an adequate skip pattern (if you have an enumerator that can’t be bothered to answer a question it’ll make it harder for him to skip a question if he deliberately has to give a false answer)
Do what you can to make the unique identifier sexy
A lot of MDC tools have an automatically created unique identifier. This can be sufficient in some type of surveys, when you have no need to link back towards secondary data or an external list of elements. When you do, having a humanly-intelligible identification system to avoid duplications of data that will completely bias your analysis and also to double check through triangulation can be very interesting. This one might already exists in your secondary data (using P-codes to identify villages for example – or else an existing list of beneficiaries that you or one of your partner has) or else, if it does not (which can be the case for beneficiaries, households, key informants, points of interests that are sometimes only identified by normal text information…) take the necessary time to set up a system before starting the survey, making sure that you have done your utmost to reduce the risk of errors in the data capture.
Here are a few elements to keep in mind:
What not to do | What to do |
Use a code that has no meaning for the enumerators | Use a meaningful, short code that can be understood by the enumerator to facilitate data capture |
Use free text to capture a unique identifier (be it a name or a code) | Use drop down lists or even better, set up a barcode system where the code is scanned from a card or a printed Excel table.If you don’t use barcodes, don’t hesitate to have the code created automatically based on previous question answers, and to have a prompt informing the enumerator of the calculated code for confirmation |
Put in a list of hundreds of codes that can be picked from in a drop-down list | Set up filters on the drop down lists (e.g. filter by geographic area, by point of interest, etc.) |
We could write for hours on the question of security – but, on this particular ID related question, if you are talking of sensitive data, using only the identifier rather than the name or address of your beneficiaries can be a way of reducing the risk of your data being used against your will if a phone is stolen or the survey results fall into wrong hands (check out chapter 6 of the following ICRC document on Professional standards for Protection Work to see humanitarian guidelines on managing sensitive protection information), while still being able to easily link back to your central database. However make sure that the identification system in your form works well as all your enumerators’ data capture will then be based on this ID.
Don’t forget whatever happens to remind your enumerators regularly why this particular information is capital for your analysis!
Give your enumerators the incentive to collect data of good quality!
It makes perfect sense to do everything you can for the survey results to be as accurate as possible. Let us note that to achieve this, rather than making your survey completely independent of your enumerator by closing it up as much as possible in terms of content, you must learn to adapt the survey also to the type of context that the enumerators will be working in. As they will be key actors in the success of your survey, involve them in the content also, in clarifying definitions or questions and take into consideration their comments and suggestions before finalizing the forms. It will make them all the more willing to fill in the survey to the best of their ability.
One must therefore work on data quality and enumerator satisfaction at the same time, like for example by informing the enumerator if he has done an error when writing down a number or email (in XLS forms – the coding standard that is used the most by mobile data collection tools – this can be done with the following formulas: regex(., ‘[0-9]{0,15}’) or regex(., ‘[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,4}’)) or else using pictures rather than text in some questions to make his capture easier. However it is important for them to continue to view the survey and data quality checks as a help and not a hindrance. This is particularly the case when you have profiles of enumerators that have been specialized in a given field for numerous years, working perfectly well without MDC, who must not be made to feel that the mobile tool replaces their skill but adds to it – I’ve sometimes seen lists of things to check before a cash transfer can be done for example that can nearly be humiliating for some highly qualified social workers .
This can be done for example by favoring interaction rather than technology creating a barrier in the communication established with the person surveyed. Further than explaining why one uses mobiles rather than paper forms (which is essential to create trust in most contexts), some NGOS, that are advanced in the use of MDC, in particular with children, can use the phone or tablet as a way of drawing or showing what a child feels or describing their environment rather than talking about it . Although this is not something that can be generalized to all survey types, finding the balance between open and closed survey processes is essential for a survey where both human and technological intelligence are necessary. Any social worker will tell you that discussing the question of vulnerabilities whilst answering a survey using a mobile phone is completely out of place- technology needs to be put to good use and used only when it really adds to the data collection process.
To conclude on this, it is capital to show experienced data collectors the added value of MDC (how easily the results can be visualized, how the standardization of the responses facilitates the analysis, how their photos can help understand a situation better than words, how GPS points can help make beautiful maps…) and how it will not be a hindrance to their work in the field. Proving to them that their role and the quality of their data collection is even more important now that the data is used more thoroughly empowers the staff using MDC and makes them deliberate actors of the survey’s success.
All things considered, if after you’ve followed these rules you ever hear of an enumerator who deliberately wants to go back to pen and paper after using a well thought-out MDC for a while, let us know 😉 !
Further reading:
- http://data.unhcr.org/imtoolkit/: Information Management Toolkit ( UNHCR Handbook for response within the first two to four months of a refugee emergency)
- http://www.acaps.org/img/documents/h-humanitarian-needs-assessment-the-good-enough-guide.pdf: ACAPS’ great “Humanitarian Needs Assessment- The Good Enough Guide”
- ttp://www.parkdatabase.org/ Profiling and Assessment Resource Kit, readily available methodologies, tools and other practical resources used in previous profiling and assessment exercises (Initiated by the Joint IDP Profiling Service (JIPS) and managed jointly with the Assessment Capacities Project (ACAPS))
- ttps://www.humanitarianresponse.info/en/programme-cycle/space/page/assessments-useful-links: different links on assessment and to the Kobo toolbox
PS: If you’d like one of the aspects discussed in this blog post to be developed in more detail in another post, leave us a comment!
І jᥙst lіke the valuable іnformation you supply іn youг articles.
I wilⅼ bookmark үour blog and test agɑin rіght һere regularly.
І am sligһtly ceгtain I ᴡill be informed а lot of new stuff rіght roght һere!
Best oof luck for the next!
Glad to hear you find it useful!
Maeve