Tuto: Conceiving a survey in the humanitarian and development field – Part 1
Part 1: Your todo list BEFORE you start jotting down questions!
I find that, in my experience of implementing Mobile Data Collection (MDC) for our humanitarian partners, I see more and more people well drilled in the technical aspects of setting up a survey (which is great news by the way!)– however, one of the key aspect that is often missing in the capacities of the field staff I’ve worked with (be it project managers, Monitoring & Evaluation staff, etc.) is how to actually design a survey. The interesting part is that they often don’t realize how important reading up or being properly trained on this is, until they have implemented it themselves in real situations… O you, who thought you didn’t need to read this article: hope you find it useful in the end! 🙂
Note: If you want an interesting read on the whole assessment cycle, beyond just the designing of a survey, don’t hesitate to look into ACAPS’ great “Humanitarian Needs Assessment- The Good Enough Guide” that you can find here or this related training document.
Most people start defining the content of a survey by taking a sheet of paper and writing down questions (we’ve all done it, haven’t we?!)- however, this should in fact be the last step… Why is that? Because the content of a MDC also derives from a reflection concerning a whole lot of aspects that need to be thought out carefully first. Here’s a summary of those that we encounter daily in the mobile data collections we set up or follow:
Most importantly, looking into the measurable goals and listing the decisions which the survey will help you inform will not only help you in your choices later on in the conception but it will also help give credibility to the choices you make with regard to other stakeholders in the project. It will also help streamline your output indicators to avoid having too many, too few or inadequate questions which is a much-too-frequent phenomenon. The more you will think all this out early on, the less you will lose time in the implementation of your survey, and the more your project will sound valid to those involved. Remember: “If the statistics are boring, you’ve got the wrong numbers” (Edward Tufte)
The risk of not considering all these aspects early on in the project can be more or less serious. Some mistakes entail mostly loss of time and energy and perhaps a dubious data quality. Yet one frequently hears of organizations abandoning a survey completely after it got deployed to the field for the first time- for example because a couple of different partners set up the survey together, but each had different priorities that they didn’t manage to discuss thoroughly and therefore the list of questions kept on getting longer and longer: without a clear measurable goal in terms of filling in the survey in a given time frame for example, one can easily be lead into a catastrophic pilot site where the whole project is undermined by lack of adequate decision-making in the designing. We’ve had partners that tell us when we bring up the subject “Filling in the survey will take the time it’ll take” which usually does not bode well with enumerators, the people answering the survey, and, for that matter, also for the person analyzing the huge quantity of data that it entails!
Using existing data (be it secondary data, baseline data, etc.) can be a great resource to target the content of your survey, to verify the quality of the data collected through triangulation (by comparing with past or other geographical contexts) or to contextualize your results and make them more relevant.
Context and constraints
The context and constraints regroup an incalculable number of things to look into: you will want to check the tech literacy of the teams in the field (before setting up an advanced barcode system for example), to check that the context is secure enough for your enumerators, to think out the data collection frequency and scope… and adapt all this to the budget and different types of resources available (which are of course never sufficient to cover all one wants to cover!).
Methodology, workflow, SOPs…
Now that you have all these elements worked out you are ready to work on the organization you will set up to match your now well-defined needs, your methodology, the workflow concerning the data processing and analysis (number of enumerators, survey managers, your monitoring plan, but also the daily standard operating procedures, data quality checks, etc.). Now most of these aspects would require a whole manual to detail, and can be very specific to a situation, therefore we will not look at them in detail here, however important they are. Let us just mention that once the survey is ready to be deployed, all these must be thoroughly tested in real-life situations to ensure that they are adequate and avoid nasty surprises that emerge too late to be dealt with! Most of all, keep at the top of your priority list the testing of your analysis to make sure that it will be easily exploitable.
Choosing the tools
The choice of the tool and of the phones or tablets will depend on all these aspects that have been thought out: all the collection, analysis or visualization tools do not have the same internet access requirements, the same capacities or ease-of-use in terms of analysis, will not work with all types of surveys (a case management system for example –although feasible- will not be easily set up with a traditional survey tool such as -the otherwise fabulous- Kobo and ODK in their present form) and do not manage data security (encryption for example) in the same way; all phones do not have the same possibilities in terms of screen size, robustness and battery life either. (Check out the NOMAD project website if you need tips in choosing your MDC tool)
Your analysis tools must also be thought out carefully, depending on the decisions the survey will inform. There are a multitude of tools out there (some are embedded in survey management tools, otherwise Excel (being the most frequently used in the humanitarian world!), geographical information systems, statistical analysis tools, Business Intelligence tools…). Just make sure you know why you use a given tool before you use it!
If you need reassurance at this stage, let’s just say that if you’ve done all this properly, the rest of the conception will be a matter of course… which we’ll look into in part 2 available here 😉 !