Data sources and collection methods Ken Mease Cairo, June 2009
What types of Data? A thorough assessment may well include: • Archival and secondary data • Survey data • Quantitative and qualitative approaches and data • It also will likely include de jure and de facto information
Qualitative and Quantitative data • There are basically two types of data: qualitative and quantitative • Qualitative data are usually text or words and quantitative data - numbers • Qualitative approaches, if conducted in a rigorous manner, require more skill than most quantitative approaches
Qualitative Data and Approaches • Qualitative approaches provide text data, but increasingly audio, video and images • They are more time consuming to analyze • Text management software, such as NVIVO, AtlasTI and AnSWR (available free at http://www.cdc.gov) • Coding is a very subjective process and open to various problems, such as investigator bias or a lack of inter-coder reliability
Quantitative Data and Approaches • Quantitative approaches provide increased rigor by investigating relationships at known levels of probability • They are easier to analyze because researchers use standard, replicable techniques • Common software includes SPSS, Stata and SAS
Sources and Types of Governance Data • • • • • • • • •
Country reports/desk studies Cross-country (region)comparative surveys Expert assessments Government data Household surveys Mass opinion surveys Key Stakeholder Surveys Media Indicators Business surveys
Combining different sources and methods • Combining methodologies and types of data often provides the most useful results. • Combining archival information and administrative data with original qualitative and quantitative survey data allows for triangulation • This approach can increase the level of professionalism, credibility and legitimacy.
Administrative, archival and secondary These data, both qualitative and quantitative, objective, reported events, perception and proxies come from a variety of sources: •Narrative reports, administrative data and other information routinely collected by government ministries •The constitution, laws, policies and legislation •Statistics and data gathered by NGOs, international organizations and academics.
Data Mapping • It is the best way to identify existing data accessibility, quality and gaps • One tool available to assist with this process is the IMF’s Data Quality Assessment Framework • A senior academic conducted the data mapping exercise in Zambia and was very valuable
Survey Data • Qualitative and Quantitative • Sample sizes can range from 20 to 20,000 or more • Costs can range accordingly • There are a range of options for who collects the data – independent surveys firms, academics, NSOs • Perception and reported events data
Different Types of Interviewing • Structured interviews use an identical instrument for each respondent, • Interviewers are trained and have explicit instructions. • This technique uses primarily structured questions with fixed response sets and very few open-ended questions
Types of Interviewing • Semi-structured interviews use a written list of questions that need to be covered in a particular order • The questions are often developed from informal discussions and focus groups. • They can include open-ended and/or more structured questions • Ideal when working with elites, managers, bureaucrats and other people who have limited time
Types of surveys and data collection methods • Face-to-face data collection is likely the best option in most developing countries • It is also the most expensive and timeconsuming • It requires professional management of trained interviewers, sampling and other aspects of the study
Mail surveys • Work well only if the postal system is reliable • The questionnaire must be carefully designed for self-administration, and there should not be too many language issues • The cost is usually quite reasonable • It misses the homeless and other vulnerable groups who may not have a valid mailing address
Internet surveys • Require special planning and design • Very good for certain populations • These surveys can suffer from low response rates. • Developing and implementing internet surveys has become very reasonable in terms of cost • Very reasonably priced internet services are available - Survey Gizmo at www.surveygizmo.com
Telephone surveys • May have coverage issues in most countries. • In many countries, people do not have a phone in their home, and if they do have a phone, it is usually a mobile or cell phone • Most cell phone providers do not make telephone numbers available for use in random-digit-dial surveys.
Focus groups • Can generate information about the background conditions surrounding governance issues. • Focus groups are usually efficient in terms of time and money. • They are highly participatory and have the potential of generating solutions to the problems identified by the group members • It is very demanding and requires highly skilled coordinators.
Focus groups • Make individual ratings insignificant • Accuracy can suffer, as some individuals may not feel comfortable to speak up in public, while others are hard to keep quiet • Focus groups yield less systematic results • Focus groups are best used to identify issues and develop surveys rather than as the only source of data
Who collects the data • Outside contractors – surveys and desk studies • Local survey researchers • Local academics for desk studies • National Statistical Offices • Government ministries • NGOs/CSOs
Data collection and dissemination exercise • Pick a Governance issue in your country • Choose the levels and types of the data you will collect – Macro, Micro, De jure and De facto • Identify the types and sources of data • Choose the data collection method/s • Identify who will collect the data • Select several options for sharing the results with different audiences