BML246 TOPIC 4: SURVEY DESIGN AND BOS

BML246

Research Skills Session 3:

Survey Design and Introduction to Bristol Online Survey Tutors: Dr Andy Clegg and Dr Jorge Gutic

Learning Outcomes Aims: To discuss the key elements of effective, valid and reliable survey

design

To discuss the required skills for writing individual questions and

designing questionnaires

To examine the specific requirements of face-to-face and online

surveys

To identify the key aspects of survey design that allow the effective

interrogation and analysis of the collected data

To demonstrate the stages of building and launching an online

survey in BOS

Step 3: Questionnaire Design

3

Questionnaire Design

What are the key elements of effective questionnaire design?

Questionnaire Design

Deciding on your methodology Is a survey the best way to collect the information you need to

achieve your research objectives

Assessment of the respective merits and limitations Alternatives / qualitative approaches / mixed methods Time frame and available resources to aid completion Sampling strategy – characteristics of the population you are

sampling (understanding the characteristics of your respondent) Piloting

Questionnaire Design

Decide on the delivery format Face-to-face or online survey Interviewee-completion or respondent-completion?

Level of instruction and guidance Length of the survey Influences on your response rate

Questionnaire Design

Key Considerations Avoid prejudicial language – questions that annoy, irritate or

insult will influence the way people respond (if at all!)

Imprecision – avoid vague phrases that are likely to be interpreted

in different ways by different respondents

Leading questions – avoid a questions that suggest a possible

answer and hence promote bias

Assumptive questions – avoid questions that make assumptions

about people’s beliefs or behaviours

[Arksey and Knight, 1999]

Questionnaire Design

Key Considerations Knowledge – make sure the group is targeted has the knowledge

to actually do so

Memory recall – remember that people often have difficultly in

remembering the most recent of events

[Arksey and Knight, 1999]

Questionnaire Design

Question-wording Use simple language: Bad Example:

- What is your frequency of utilisation of retail travel outlets?

Improved Example:

- How often do you use travel agents?

[Arksey and Knight, 1999]

Questionnaire Design

Question-wording Avoid Ambiguity: Bad Example:

- Do you play sport very often?

Improved Example:

- Have you played any of the following sports within the last four weeks?

[Arksey and Knight, 1999]

Questionnaire Design

Question-wording Avoid leading questions: Bad Example:

- Are you against the extension of Heathrow airport?

Improved Example:

- What is your opinion on the extension of Heathrow Airport?

Are you for it, against it or not concerned?

[Arksey and Knight, 1999]

Questionnaire Design

Question-wording Ask one question at a time: Bad Example:

- Do you use the Hub and if so what do you think of its facilities?

Improved Example:

- Do you use the Hub – yes/no?

- What do you think of the facilities in the Hub?

[Arksey and Knight, 1999]

Questionnaire Design

Your questions

Adapt - questions used in other research Adopt - questions used in other research Develop - your own questions

Questionnaire Design

Layout and ordering of questions Clear presentation and instructions on about how to respond Guide the respondent clearly through the process Use pages and sections in BOS Place broad and general questions at the beginning of the

questionnaire followed by more specific questions Using funnel questions

Questionnaire Design

Sequencing

[Gray, 2017, p. 375]

Questionnaire Design

Types of Variable Attributes – things a respondent possesses – e.g. age / gender /

marital status

Behaviour – what respondents do Opinion – how respondents feel about something and what they

think or believe is true

Questionnaire Design

Layout and ordering of questions Numeric coding Avoid placing survey questions out of order or out of context Format and layout of questions – e.g. horizontal or vertical

presentation

BOS allows you to insert a variety of media into your questions

(e.g. images)

Questionnaire Design

Layout and ordering of questions Double questions - don’t ask two questions in one – e.g. is your job

interesting and well paid cannot be answered with a simple yes or no

Attention to detail – overlapping categories such as age ‘30-35,

35-40’

Questionnaire Design

Avoid Bias Be careful with the use of wording – the tone and wording of

questions can have a significant impact on your results (e.g. might/should/could)

Capture the Voice Even within quantitative surveys open-ended questions provide an

opportunity to ‘capture’ the voice of the respondent

Questionnaire Design

Be Specific and Exact Unclear survey questions produce answers that lack meaning and

relevance

e.g. How regularly do you watch TV – ‘regularly’ needs defining Make sure multiple choice questions are mutually exclusive so that

a clear choice can be made

Questionnaire Design

Be Considerate Avoid intrusive questions (ethics!)

Consider your audience – ensure they understand your language,

terminology and the question being asked

Keep questions short and to the point – respondents will not

complete long and tedious surveys

Incentive – what do I get for completing your survey?

Questionnaire Design

Choose the Right Question Your choice of question will influence the type of data you get and

opportunities for subsequent analysis

Do the question and answer formats provide enough robustness to

meet analysis requirements?

The type of data will influence the nature and scope of your

analysis (and ability to meet assessment criteria!)

Questionnaire Design

— Clear and logical structure of the

presentation/poster demonstrating progression from basic to advanced statistical techniques referencing a specific aspect of the research process/results

Data Requirements Accurate - there is no ‘systematic bias’ in measurements Precise - they are measured with sophisticated instruments /

methodologies

Reliability – data is comparable over time and/or

spatially/geographically (longitudinal analysis)

Valid - they are a ‘true’ representation of the underlying more

complex phenomenon, e.g. ‘quality of life’ or ‘economic well-being’ etc (the extent to which the data reflects what they are meant to reflect)

Data Requirements Fit for purpose – provide the basis for basic descriptive to

advanced statistical analysis

Informed – methodologies have been drawn from a

review/critique of the available literature – background reading!

Services

Bristol Online Survey