What Are Some Data Collection Challenges and How Do You Overcome Them? (Part 1)

Getting it right or as close to right as possible is critical when collecting data.  The consequences of failing to properly collect data include the inability to answer your research questions, inability to validate the results, distorted findings, wasted resources, misleading recommendations and decisions, and harm to participants.  

There are many things that could go wrong when collecting data, which may ultimately compromise the trustworthiness of your findings.  To help you think through potential challenges and how to address them, here are some challenges and advice to consider as a nonprofit organization. 

This article focuses on the challenges present Before Data Collection.

• Challenge: Complicated forms can lead to nonresponse or respondents filling in answers at random simply to complete the survey.

Consideration: Keep the language and format of surveys simple.  When possible, word questions so that the response options are the same and try to eliminate or reduce reverse worded questions.  Always validate your surveys before using them. 

• Challenge: Insufficient identifying information for matching pre/post responses.

Consideration: Ask for the first 3 letters of respondent’s last name and their birthdate (MMYY).  These elements are consistent and not forgotten by respondents.  In some contexts, these identifiers are problematic because respondents do not actually know them – seek advice from experienced international data collection teams. 

• Challenge: Literacy comprehension barriers.

Consideration: Survey responses can be read to participants.  Mobile devices can also be utilized to ‘read’ to respondents who select non-read response options (face expressions, colors, etc.). 

• Challenge: Language comprehension barrier.

Consideration: Hire professional translators to translate questions and then have another translator back-translate to original language to ensure intended meaning not lost.  Pilot test the survey with a variety of people to ensure intent is understood. 

• Challenge: Insufficiently trained staff.

Consideration: There should always be a team of people involved with data collection and a leader identified. Clear methodologies are critical for your team’s success.  Clear protocols for the “what if” scenarios are crucial.  Role play various situations in which the team may or will find themselves: gaining approvals from authorities, giving explanations to community leaders/teachers, implementing surveys, fielding questions from respondents, etc. 

• Challenge: Lack of understanding of context.

Consideration: Sometimes before you can quantify anything, you have to learn more about the context of a respondent group.  This may require qualitative research to answer key questions that can then be used for quantitative research; without it, you risk missing truth altogether. 

• Challenge: Necessary approvals.

Consideration: Every survey will require approvals at one stage or another.  Adult respondents must agree to be a part; child respondents must have their parents’ approval.  If conducted in school or in any organized environment, educators/leaders will need to approve the endeavor.  Ensure sufficient time is built into your timeline to acquire all necessary approvals.

• Challenge: Useful resources.

Consideration: Data collection can be expensive.  Collaborating with like-minded nonprofits to conduct mutually beneficial research is a good way to collect good data at half the cost.  Hiring and training a local data collection team will be cheaper (and often more effective at obtaining data) than bringing in non-locals to do the work.  

• Challenge: Inappropriate sampling.

Consideration:  To generalize your findings, you must have an appropriate sample that reflects the population you are hoping to generalize.  There are sample calculators that can estimate the size you need, but you also need to consider statistical power.  This power, provided by a statistician, determines the level to which an effect is established.  In order for your findings to be relevant, you must have statistical power. 

• Challenge: Participation in research.

Consideration: Generally speaking, all participation in research is voluntary.  Where possible, ensure anonymity; this will almost always increase participation.  Keep qualitative research around 45-60 minutes in time and survey research to less than 20 minutes.  Provide incentives such as gift cards, coupons or discounts, raffle options, etc.

• Challenge: Untrusted data. Technically this is an analysis issue, but to correct it, it should be considered before collecting your data. 

Consideration: Build social desirability scales into your surveys to check (in analysis) whether responses can be trusted. 

Take every effort to set up appropriate measures before data collection begins (quality assurance) and then to follow the protocols during and after data collection (quality control).  

Be diligent in the process.  It is the only way you will be able to trust your findings.