BML246
Research Skills Session 3:
Survey Design and Introduction to Bristol Online Survey Tutors: Dr Andy Clegg and Dr Jorge Gutic
Learning Outcomes Aims: To discuss the key elements of effective, valid and reliable survey
design
To discuss the required skills for writing individual questions and
designing questionnaires
To examine the specific requirements of face-to-face and online
surveys
To identify the key aspects of survey design that allow the effective
interrogation and analysis of the collected data
To demonstrate the stages of building and launching an online
survey in BOS
Step 3: Questionnaire Design
3
Questionnaire Design
What are the key elements of effective questionnaire design?
Questionnaire Design
Deciding on your methodology Is a survey the best way to collect the information you need to
achieve your research objectives
Assessment of the respective merits and limitations Alternatives / qualitative approaches / mixed methods Time frame and available resources to aid completion Sampling strategy – characteristics of the population you are
sampling (understanding the characteristics of your respondent) Piloting
Questionnaire Design
Decide on the delivery format Face-to-face or online survey Interviewee-completion or respondent-completion? Level of instruction and guidance Length of the survey Influences on your response rate
Questionnaire Design
Key Considerations Avoid prejudicial language – questions that annoy, irritate or
insult will influence the way people respond (if at all!)
Imprecision – avoid vague phrases that are likely to be interpreted
in different ways by different respondents
Leading questions – avoid a questions that suggest a possible
answer and hence promote bias
Assumptive questions – avoid questions that make assumptions
about people’s beliefs or behaviours
[Arksey and Knight, 1999]
Questionnaire Design
Key Considerations Knowledge – make sure the group is targeted has the knowledge
to actually do so
Memory recall – remember that people often have difficultly in
remembering the most recent of events
[Arksey and Knight, 1999]
Questionnaire Design
Question-wording Use simple language: Bad Example:
- What is your frequency of utilisation of retail travel outlets?
Improved Example:
- How often do you use travel agents?
[Arksey and Knight, 1999]
Questionnaire Design
Question-wording Avoid Ambiguity: Bad Example:
- Do you play sport very often?
Improved Example:
- Have you played any of the following sports within the last four weeks?
[Arksey and Knight, 1999]
Questionnaire Design
Question-wording Avoid leading questions: Bad Example:
- Are you against the extension of Heathrow airport?
Improved Example:
- What is your opinion on the extension of Heathrow Airport? Are you for it, against it or not concerned?
[Arksey and Knight, 1999]
Questionnaire Design
Question-wording Ask one question at a time: Bad Example:
- Do you use the Hub and if so what do you think of its facilities?
Improved Example:
- Do you use the Hub – yes/no? - What do you think of the facilities in the Hub?
[Arksey and Knight, 1999]
Questionnaire Design
Your questions
Adapt - questions used in other research Adopt - questions used in other research Develop - your own questions
Questionnaire Design
Layout and ordering of questions Clear presentation and instructions on about how to respond Guide the respondent clearly through the process Use pages and sections in BOS Place broad and general questions at the beginning of the
questionnaire followed by more specific questions Using funnel questions
Questionnaire Design
Sequencing
[Gray, 2017, p. 375]
Questionnaire Design
Types of Variable Attributes – things a respondent possesses – e.g. age / gender /
marital status
Behaviour – what respondents do Opinion – how respondents feel about something and what they
think or believe is true
Questionnaire Design
Layout and ordering of questions Numeric coding Avoid placing survey questions out of order or out of context Format and layout of questions – e.g. horizontal or vertical
presentation
BOS allows you to insert a variety of media into your questions
(e.g. images)
Questionnaire Design
Layout and ordering of questions Double questions - don’t ask two questions in one – e.g. is your job
interesting and well paid cannot be answered with a simple yes or no
Attention to detail – overlapping categories such as age ‘30-35,
35-40’
Questionnaire Design
Avoid Bias Be careful with the use of wording – the tone and wording of
questions can have a significant impact on your results (e.g. might/should/could)
Capture the Voice Even within quantitative surveys open-ended questions provide an
opportunity to ‘capture’ the voice of the respondent
Questionnaire Design
Be Specific and Exact Unclear survey questions produce answers that lack meaning and
relevance
e.g. How regularly do you watch TV – ‘regularly’ needs defining Make sure multiple choice questions are mutually exclusive so that
a clear choice can be made
Questionnaire Design
Be Considerate Avoid intrusive questions (ethics!) Consider your audience – ensure they understand your language,
terminology and the question being asked
Keep questions short and to the point – respondents will not
complete long and tedious surveys
Incentive – what do I get for completing your survey?
Questionnaire Design
Choose the Right Question Your choice of question will influence the type of data you get and
opportunities for subsequent analysis
Do the question and answer formats provide enough robustness to
meet analysis requirements?
The type of data will influence the nature and scope of your
analysis (and ability to meet assessment criteria!)
Questionnaire Design
— Clear and logical structure of the
presentation/poster demonstrating progression from basic to advanced statistical techniques referencing a specific aspect of the research process/results
Data Requirements Accurate - there is no ‘systematic bias’ in measurements Precise - they are measured with sophisticated instruments /
methodologies
Reliability – data is comparable over time and/or
spatially/geographically (longitudinal analysis)
Valid - they are a ‘true’ representation of the underlying more
complex phenomenon, e.g. ‘quality of life’ or ‘economic well-being’ etc (the extent to which the data reflects what they are meant to reflect)
Data Requirements Fit for purpose – provide the basis for basic descriptive to
advanced statistical analysis
Informed – methodologies have been drawn from a
review/critique of the available literature – background reading!
Services
Bristol Online Survey