The Consensus Group Technique A U sers' Manual
Dennis List
Original Books Wellington, New Zealand
2005
Original Books Head office: POBox 6637 Marion Square Wellington New Zealand Publisher: Niel Wright email: nielwright@xtra.co.nz Quick Guide to Audience Research
ISBN 1-86933-726-3 Edition of September 2005 copyright Š Dennis List, 2005 author's email: dennis@audiencedialogue.org International distributor Audience Dialogue 1 East Tee Nailsworth Adelaide 5083 South Australia email:audial@ftml.net
Consensus Groups Manual
3
The Consensus Group Technique: A Users' Manual
CONTENTS
Introduction
................................................................................. 5
Recruitment phase Stage 1
Choose sampling points
8
Stage 2
Organize a venue
9
Stage 3
Prepare a screening questionnaire
11
Stage 4
Find participants
14
Discussion phase Stage 5
Set up discussions
18
Stage 6
Introductions
23
Stage 7
Agenda-setting
24
Stage 8
Main discussion
25
Stage 9
Establish the level of consensus
'" 30
Consolidation phase Stage 10
Consolidate the findings
34
Stage 11
Reporting
35
Further information
37
Appendixes Notes for moderators and secretaries
38
Checklist of materials needed
40
Example of preliminary questionnaire
41
4
Consensus Groups Manual
Introduction
This simple method, originally developed for audience research, uses verbal skills (as commonly possessed by media staff) rather than the numerical skills needed for traditional survey research. It is midway between a focus group and a public meeting. It includes elements of other techniques: nominal groups, search conferences, the Delphi method, citizen juries, and consensus conferences. How does a consensus group differ from a survey? • In any survey, the questionnaire ensures that everybody is asked the same questions. The only variation can be in the number of people giving each answer to each question. Therefore, though surveys begin with words (questionnaires), the results are always expressed in numbers. • Consensus groups work in the opposite way: the numbers remain roughly constant, but instead of questions there are statements. In discussion, the wording of each statement is adjusted until either the great majority of participants agree - or it becomes clear that they can't agree. And how does a consensus group differ from a focus group? • Focus groups normally have a moderator, but no secretary. Consensus groups need both. • Focus groups work best with about 7 or 8 participants. Consensus groups work best with 10 to 12. • A focus group resembles a large (sometimes unruly) conversation. A consensus group is more like a small meeting - a little more formal than a focus group. • With a focus group, the moderator is firmly in control - of both the agenda and the interpretations. With consensus groups, participants help set the agenda, and vote on preferred statements. • Results from a focus group are based on the moderator's perceptions. Results from a consensus group are decided by the participants: the organizers have much less power. Though this manual uses radio audience studies as an example, consensus groups can be used for any kind of population. The method has also worked well in assessing the adoption of innovations, and in determining priorities for voluntary organizations.
Consensus Groups Manual
5
A common misunderstanding about consensus groups is that they are a means of building consensus. This is not so. The consensus group technique is primarily a research method, not a conflict management method (though it can serve that purpose, indirectly). The purpose of the method is to find out to what extent consensus already exists - to detect consensus, not to create it. As the method uses representative sampling, there is no point in using the group process to create a consensus among a tiny proportion of a population. If this were done, the findings would no longer represent the population. The technique has three main phases: recruiting participants, holding discussions, and consolidating findings But before participants are recruited, the same strategic concerns apply as with any other social research method. In short, the organizers must decide exactly what is to be covered, and among what population. Once the scope of the study has been decided, three main phases follow: recruitment, discussion, and consolidation. Within these three phases are ten stages of the process: Recruitment 1. Within the area to be studied (e.g. a radio station's coverage area), three sampling points are chosen, contrasting as much as possible.
2. At each of the sampling points, a venue is arranged. All you need is a suitable room.
3. A short screening questionnaire is prepared. The purpose of this is to find people eligible to take part in the consensus groups. 4. At each sampling point, people are interviewed using the screening question-naire. These interviews find people who are both eligible and willing to attend a consensus group. Discussions 5. The group meets, either immediately after the interviewing, or up to several weeks later. This meeting will run for about two hours. At each meeting, there would be approximately 12 participants and 2 organizers: a moderator and a secretary.
6. At the beginning of the meeting, each participant briefly introduces himself or herself to the others, giving some basic information about their habits - e.g. radio listening.
6
Consensus Groups Manual
7. Possible sub-topics are discussed, and everybody votes on their level of interest in each topic. The chosen topics, in descending order of popularity, are discussed by all participants. The moderator steers the discussion around the arranged scope of the study, and the secretary takes notes.
8.
In the final stage of the meeting, consensus is determined. On a whiteboard (etc.) the secretary writes down statements which most participants are expected to agree with. Statements are modified, depending on what participants say. When each statement is ready, participants vote on it. Typically, around 30 statements are agreed on by most of the participants. This list of statements is the main outcome of the meeting. 9.
Consolidation 10. After three meetings have been held (one for each sampling
point found in stage 1), the three lists of statements are compared. Any statements shared by at least two groups are the outcome of the study. 11. Report on the findings.
Now let's look at each of the above stages in more detail...
Consensus Groups Manual
7
Stage 1. Choose sampling points
A sampling point is simply a geographical area, where a small survey can be carried out. It can be either a group of homes, or a public place where passers-by are recruited for the study. For consensus group sampling to work effectively, at least three sampling points are needed. They are chosen to be as different as possible. For example, if your study is being done on behalf of a radio station, the population will probably be all people who live in that radio station's coverage area. Within this area, you should identify three contrasting localities. For example, if the station covers a city and outlying rural areas, you might choose: • An inner-city area • An area near the outer edge of the city • A rural area. Another way in which localities often vary is in wealth. Therefore it would be useful to choose one wealthy area, one poor area, and one middle-income area. Sampling points need not be geographical. Where the population is clustered, and prior information is available - e.g. a database of clients - three distinct groups of potential participants can be defined from that prior information. For example, in a recent study we did, planning a community information system, the three groups were information providers, information end-users, and intermediaries between the other two. Sometimes, people of different age groups and genders may inhibit each others' responses. When that is likely to occur, groups can be separated by age and gender - as in a study we did with Aboriginal people in central Australia: the groups were of younger men, younger women, older men, and older women. Whatever the basis on which the three (or more) groups are selected, the main goal is to make them as different as possible. This is a type of maximum-variation sampling. Three sampling points is a minimum, but there can be more. We have· found that each additional sampling point adds less and less variation in the statements. However, in some situations, more than three sampling points are needed to adequately cover the variance in a population. For example, a technique we have often used compares three groups of a station's potential listeners with three groups of its current listeners. This requires choosing two groups at each 8
Consensus Groups Manual
sampling point. If both the sampling points and the type of person invited to a group are different, you cannot make clear conclusions about the causes of any differences in the results - hence the need for a lot more groups in this case.
Stage 2. Organize a venue
A venue for a consensus group is a space that will hold about 15 people and is free from interruptions. It need not even be a room; in Papua New Guinea we held some groups outdoors, with no problems. In Australia, many hotels have rooms available for hire for meetings, and we often use these. We have also used clubrooms of voluntary organizations, office areas after working hours, schools, libraries, restaurants, shops, and so on. Another possibility is to use a private house, paying the owner a small fee for the use of their room. It is usually best not to use the premises of the organization sponsoring the research, as people may be reluctant to criticize an organization when they are on its premises. For example, if your research is for a radio station, avoid using its studios as a venue. But this depends on the type of people attending, and on the organization. Here are some factors to take into account when choosing a venue: •
•
•
•
•
We usually provide something to eat and drink for the participants - it helps them to relax. This is a factor should be borne in mind when choosing a venue: hotels and restaurants have an advantage here. A venue should be easy to find, specially for people who have never been there before. Places such as hotels are usually well known to people living in the area. A venue should not be a place that some people would not want to visit. In some cultures, for example, women will not go to hotels. A venue should be fairly quiet, particularly if the meeting is to be recorded on audio tape. Noisy air-conditioning can be an unexpected problem, making voices much more difficult to understand - even when participants are hardly aware of the background noise. And of course, the venue must be close to the area from which the sample is drawn.
Consensus Groups Manual
9
We have found that people are often reluctant to travel for more than about ten minutes to a group discussion. When distances are large, some people don't turn up, and others (disruptively) arrive much too late. Sometimes you may have to change a sampling point, because no suitable venue can be found nearby.
10
Consensus Groups Manual
Stage 3.
Prepare a screening questionnaire
The main purpose of the preliminary screening survey is to find participants for the consensus groups. They must be both eligible and willing to participate. If, for example, you are assessing the programs on a radio station called FM99, you may decide there is little point in inviting people who don't listen to the station. In this case, the key question on the screening questionnaire would be "Do you listen to FM99 at least once a week?"
(This is better wording than asking simply "Do you listen to FM99?" If no time limit is included, people who listen only very rarely to the station would answer Yes, and would not be able to discuss the programs in detail.) If you are interested in increasing the station's audience, you may want to speak to potential listeners to the station. There are several ways to define potential listeners on a questionnaire. Our own studies have found that when a station increases its audience, this is usually because people who already listened to the station infrequently began to listen more often. So most of your potential listeners are probably listeners already - but not very often. In a screening survey, you could ask "How often do you listen to FM99: at least once a week, very occasionally, or never?"
All those giving the middle answer would be considered potential listeners. If you are trying to assess the audience to something that does not yet exist (such as a planned new radio station), you will need to define the potential listeners in another way. This can be done from two starting points: • A demographic group, such as "rich people living near the city." If these were defined as the target audience, the purpose of the study would be to find the kind of radio program that most appealed to these people. • A program concept, for example "a radio station specializing in jazz, blues, and reggae." In this case, the purpose of the study would be to estimate how many people are interested, what type of people they are, and exactly what program content would most interest them.
Consensus Groups Manual
11
When you have found a respondent who is eligible to take part in a consensus group, the next step is to find out if they will attend. We normally use wording like this...
"With the answers you've given, you're eligible to take part in a group discussion, to talk about news and current affairs in more detail. We'd like to invite you to come to this discussion next Tuesday night, the 15th of October. We're holding it at the Flinders Lodge, in Dequetteville Terrace, Kent Town, starting at 7 pm, going for up to two hours. People usually find these meetings very interesting, and if you attend we'll pay you forty dollars to cover your expenses. Would you like to come along?"
D D D
Not interested Interested, but can't come Agreed to come ~ (get name and address, and say we'll send a
letter) Name
.
Address
.
The essential points included in the above question are: • Invitation to attend • Very approximate description of the subject • Date, time, and place of meeting • Maximum duration of meeting • Incentives, such as payment to respondents, offer to feed them, meet interesting people, and so on. The third type of question that can be included in a screening questionnaire is the demographic question: their sex, their age group, and perhaps their occupation and other similar data. There are two reasons for obtaining this demographic information: • To help ensure a proper balance of sexes, age groups, etc. in the consensus groups. • To help find out the differences between the type of people who attend the groups and those who do not. As a screening questionnaire will have few questions, you can save paper by not having a separate sheet of paper for each interview.
12
Consensus Groups Manual
Using a single page, with a few lines for each person's answers, can be more convenient.
Consensus Groups Manual
13
Stage 4. Find participants for the discussions
The soundest method of finding participants is to carry out a random screening survey on the population to be studied. However, this is not always possible, so other recruitment methods may need to be used. Though many informal selection methods work well, it's essential to avoid any method in which only a tiny proportion of those contacted volunteer to take part (such as a mail survey). Such an outcome almost guarantees that the findings will not be typical of the population. Unless the group discussions are to be held as soon as enough eligible respondents are found, it is best to use only one or two interviewers at each sampling point. Because you will be aiming for 12 participants in each group, a large number of interviews will not be required - unless those eligible to take part are only a small percentage of the population. In order for 12 people to turn up, you will probably need more than 12 to agree. In Australia, even when we send a letter to confirm the details of the discussion, and recontact each participant the day before the discussion, usually 10% to 20% of those who accepted fail to turn up. Therefore, we usually get acceptances from one more person than we really want: if we want 12, we obtain 13 or 14 acceptances. In other countries, the proportion that fails to turn up may be different from the 10% to 20% figure we have experienced. In Australia, we have found that unless we confirm each person's attendance (by a follow-up letter and/ or telephone call) many more than 20% do not turn up. We have also found that people who say they "might" come hardly ever do. One of the worst things you can do when organizing a group is to extend a weak invitation to a lot of people. As this is very easy, it may seem tempting. If you are running a radio station, you may think "Why not advertise on air that we are doing a research study, and invite listeners to come along?" The problem is that you have no control over the number of people who turn up. It could be nobody, and it could be hundreds. We have found that 12 people is about the ideal number for a consensus group. With fewer than about 8 participants, there is too much danger of the responses being atypical - one very persuasive person might influence the others. With more than about 15, the group either runs on for far too long, or else it is abbreviated so much that many participants don't have a full chance to offer their thoughts. 14
Consensus Groups Manual
In Australia, usually less than 50% of people eligible to attend a group will agree to do so. Bearing in mind those who are not eligible, the eligible people who do not want to come to the group, and those who say they will come but do not, sometimes it takes a lot of interviews to fill one group of 12. For example, if one person in 10 is eligible, and a third of those attend a group, that's 30 interviews for each person who attends, or 360 interviews to fill a group. It is therefore a good idea to make the eligibility criterion fairly broad. Another step you can take to reduce the number of interviews is to offer an incentive to attend. If you can persuade two thirds of the eligible people to attend instead of one third, only half as many interviews will be needed. Therefore if you pay participants generously, this can greatly reduce the total cost. We normally do the screening surveys between one week and two weeks before the discussion. If given more than two weeks' notice, people tend to forget. With less than a week's notice, many people can't attend, because they have already made plans to do other things at the time. But it is not essential to wait that long. Another completely different approach is to hold consensus groups on the spot. For example, in a training session at Hanoi, Vietnam, a number of trainees were sent out to recruit people in the street outside the venue. All people who met the criteria and had an hour to spare were invited to a group discussion then and there. It took only about 10 minutes to find enough participants. Of course, this only works when a large number of eligible people are nearby, but if your organization is one that has a lot of visitors (such as a museum) this could be an easy way to find participants. Affinity groups
The above description of screening questionnaires involves a separate questionnaire for each person. People are interviewed individually. In the resulting groups, each participant will usually not know any of the other participants - except in small towns, where many people already know each other. In some ways unfamiliarity can be an advantage, because participants will not offend their friends by giving opinions they feel the friends might disagree with. But in other ways, it can be disadvantageous to hold a discussion among strangers: participants may feel unwilling to reveal their opinions to people they do not know and cannot trust.
Consensus Groups Manual
15
Which of these two disadvantages is the stronger will vary from country to country. In Australia, my experience is that when the participants in a group already know each other well, they tend to express their feelings more freely. Sometimes it is best to restrict each group to similar people - not asking all eligible persons to attend, but only some of them. For example, when obtaining opinions of listeners to a radio station, in a country where, men and women do not mix socially with strangers of the opposite sex, you might need to organize four groups instead of three: two of men only, and two of women only. Sometimes it is better to have separate groups of younger and older people. In some countries, it may be best not to mix supporters of different political parties (or different religions) in the same group. This separation can be done partly by careful selection of sampling points, and partly through screening questionnaires. Just bear in mind that the purpose of restricting a group to a particular type of person is to enable the members of a group to speak more freely. It's usually best to dissuade husbands and wives from coming together. They tend to inhibit each other. Often only one of them will participate in the discussion. As each group is quite small, it would be better to invite two people who would give separate opinions. A completely different approach, which can also work well, is to organize a discussion among a group of people who already know each other, such as members of a sporting club, a group of people who work together, or a group of students. And of course, the groups need not have a common purpose: they can simply be groups of friends or neighbours. However, I advise against using a group made up of members of a family. There is too strong a chance that they will not be typical of the population, because the entire study would then be limited to a few families. These groups (of people who already know each other) are called affinity groups. When affinity groups are used for a study, each group needs to be as different as possible from each other (replacing the three sampling points). For example, don't choose three sporting clubs, or three groups of neighbours: This type of sampling is most effective when there is largest possible contrast between the types of person in each consensus group. ' _:'. : :.:;- :~, ':;_
' ' :1
One problem which restricts the use of affinity groups is that not everybody in an affinity group may be eligible. If a radio station is
studying its listeners, it does not matter if a few people in an affinit y group are not listeners, but if most participants are non-listeners, the group will not provide useful information. Also, if people are not interested in the topic being studied, they are likely to disrupt the discussion by talking among themselves. Therefore affinity groups are best when all or most people in the population are eligible to be interviewed.
Consensus Groups Manual
17
Stage 5. Set up the discussions '.
Each group needs a discussion leader, or moderator. This person (preferably not a broadcast presenter, in whose presence people may withhold criticism) feeds the group with stimuli, or material to react to, as well as encouraging the reticent to speak up, and discouraging the talkative from dominating the proceedings. And when the topic is radio, one essential function of the moderator is to occasionally remind the participants to stop talking about television! Each group should have a second person from the organizing team, to act as secretary. Though it is (just) possible for an experienced moderator to fulfil both functions, it is valuable to have a second organizer present, so that opinions can be compared after the participants have left. But if too many people from the organizing team are present, participants are likely to feel inhibited. If there are 12 participants, there should not be more than 4 organizers present at once. Apart from the moderator and secretary, the other organizers should hardly speak at all. These other people who may be present include: • a video-camera or tape-recorder operator. • somebody to provide drinks or food for the participants. • an observer or two from the client organization. Choosing a moderator
Ideally, the moderator should have some experience with facilitating small groups, as well as a good understanding of the topic area. A good moderator will balance the pressures of keeping to time against the value of bringing out new viewpoints. If one participant is being disruptive, a good moderator will be able to silently enlist the other participants to put pressure that participant. A good moderator will also encourage everybody to join the discussion on equal terms. People with extensive experience at chairing meetings often make good moderators. Choosing a secretary
The qualities of an ideal secretary include being able to write in large letters neatly and quickly, but above all to interpret vague statements into clear and unambiguous wording. These qualities are rarer than those found in a good moderator. We've found that teachers and exteachers often make excellent secretaries. Preliminary questionnaires It hardly ever happens that all participants in a group arrive at the
same time. Even when we ask people to be sure to arrive at the 18
Consensus Groups Manual
advertised starting time, some usually arrive ten minutes late, and others arrive much too early. As soon as they arrive, participants are keen to know what will be happening- - but it can be tiresome to repeat this over and over again, as each new participant arrives a minute apart, while you're getting ready for the discussion. To give participants something to do, we usually have a preliminary questionnaire, which they can fill in as soon as they arrive. Those who arrive late can fill in their questionnaires while the discussion takes place. As well as giving participants something to do while they are waiting for others to arrive, these questionnaires can collect useful information. The other use of preliminary questionnaires is to help settle the agenda. This can be done either directly ("In this meeting about FM99, what topics would you like to discuss?") or by asking questions which the people can think about and discuss later, such as "If you could make one change to FM99, what would it be?" These questionnaires should be short and simple. I try to restrict them to a single sheet of paper, with questions only on one side. As some people prefer to communicate in writing, we let participants keep their questionnaires throughout the discussion, and invite them to write their thoughts and comments on the blank back of the questionnaire, and hand it up at the end.
Seating arrangements A table is not essential, but very useful. The best type of room arrangement is normally a large table (or several smaller tables pushed together), around which the participants sit in chairs. The tabletop is useful for filling in questionnaires, and for food, drink, and microphones, and also serves as a kind of psychological protection for participants from the organizers. In some cultures, specially in Asia and Africa, people prefer to sit on the floor. This is no problem, but if the participants are going to fill in questionnaires, you will need to provide writing surfaces, such as clipboards. Displaying data
An essential part of the consensus group technique is the display wall, on which all the results from the group are displayed. Originally we used butchers' paper, but later found it more Consensus Groups Manual
19
convenient to use many small sheets. On these, the secretary writes the findings of the group, in large letters so that all participants can read them. (If some or most are illiterate, symbols can be used as well as words.) It's also possible to use a blackboard or whiteboard, but sheets of paper are best because they can be taken away and kept as the record of the discussion. This diagram shows a room layout that works well:
Wall of statements Secretary
Moderator
Participants
The secretary needs a small table, on which to write statements, and should be close to the wall, for modifying statements when necessary. The moderator should not have a table, to reduce formality and seem more approachable. The participants are arranged in a slight semi-circle, so that they can see each other, with the wall (covered in statements) easily visible. Any observers should sit behind the participants, to keep out of the way. The minimum space required for this layout is about 5 metres deep, and 8 metres wide - but (in Australia) that's a tight squeeze. Electronic recording
For later reference, the whole discussion should be recorded, either on videotape or audiocassette - though this is not necessary, because the whole proceedings of a consensus group are recorded on paper. Video has many advantages, but it also has the disadvantage of requiring a camera operator. Though it is possible to set up a video camera and simply point it at a group of people sitting in a semicircle, most detail is lost unless there's an operator to zoom in on each person as they speak, and to film the reactions of others. You might expect that a video operator would greatly distract the participants, but we have found this doesn't happen. After the first few minutes, participants usually stop noticing the video camera. 20
Consensus Groups Manual
(You know this when replaying a video recording, because participants hardly ever look directly at the camera.) Of course, the video operator should intrude as little as· possible, and stand back at a reasonable distance. Bear in mind that, to focus on a whole group of 12 or more people, you will often need a larger room than you might expect - or a camera whose lens can zoom out to an unusually wide angle. If a video camera is not available, the next best thing is to record the
discussion on audiocassette. This does not require a separate operator - the secretary or moderator can work it. Some requirements for successful audiotaping are:
• With a video recording, you can see who is speaking, but with audio, many voices may sound the same. To help identify voices on the tape, the moderator should address participants by name as often as possible. The moderator may also need to describe other things that are happening which will not be recorded on the tape. For example, if a participant shows the others a printed advertisement, the moderator should describe what is happening (and explain why, to the participants).
• A
high-quality microphone is essential. After much experimentation, we discovered the PZM microphone, also known as a boundary microphone. These are made by several manufacturers, and look like a flat, thick piece of metal, about the size of a person's hand. It sits on a tabletop or other flat surface, and records all sounds reaching that surface. The microphones built into cassette recorders are usually of too Iowa quality to pick up voices dearly, specially when several people are speaking at once, and the microphone is more than a few metres away. PZM microphones work well up to about 10 metres range, as long as there's not much background noise. Two microphones are best, because then you can record in stereo (making it easier to identify voices based on where people were sitting), and if one malfunctions you still have the other.
• Double-check, before anybody arrives, that everything is working! There are many ways to go wrong with a tape recording. Batteries go flat (in either the tape recorder or the microphones), plugs may not be inserted fully, the tape can be set to "pause" and forgotten, the volume control can be turned right down, the machine can be switched to "play" instead of "record", and so on. I prefer a tape recorder with an indication of the volume (flashing LED lights or a VU meter), and a clear window Consensus Groups Manual
21
so that you can check that the tape is turning. If the light is flashing and the tape is turning, the recording will probably be OK.
22
Consensus Groups Manual
'.
Stage 6. Introductions
When almost all participants have arrived, the discussion can begin. In this stage, the moderator asks each participant in turn to introduce himself or herself. Here we are looking for factual information which will help others understand the opinions which will be stated later by that person. For example, if the topic is a radio station, I find it helpful to ask participants to describe their households, their daily routine, and their radio listening habits. The other purpose of this stage is to help each participant gain confidence in speaking to others in the group. Not a lot of detail is needed here. I have found it best for the moderator and secretary to introduce themselves in the style needed, as an example, before asking participants to do the same. Each participant is given a voting card - a bright coloured piece of cardboard, about the size of a playing card. These are used when voting in the final stage (easier to count than hands that are halfraised), but are also used to indicate agreement with what the current speaker is saying. This introductory stage usually takes between one and two minutes per person, or 15 to 20 minutes for a group of 12 people. Identifying participants
We often use name tents to help identify the participants to each other and to the organizers. A name tent is a folded piece of cardboard, on which is written the name of the participant sitting at that place on the table.
If the name tents are put on the table in advance, this can be used as a way of planning where people sit. If several of the participants are from the same family, they are likely to distract the other participants by whispering if seated next to each other, so it's a good idea to separate them. If the participants are not sitting at a table, an alternative to using name tents is to draw a map of the seating arrangements, marked with all the participants' names - similar to the diagram above. This can be displayed on the wall for all to see. Consensus Groups Manual
23
Stage 7. Agenda-setting
Participants will already have been told the broad topic of the discussion. On the preliminary questionnaires they'll have had a chance to think about what they want to discuss, within that broad topic. Of course, the organizers have a say in this too. They will have prepared their own list of topics in advance, writing these on a large sheet of paper. The purpose of dividing the subject into topics is to help ensure that nothing is inadvertently overlooked. There don't need to be a lot of topics - 10 is usually more than enough (though 5 is often too few). For example, in a discussion about the radio station FM99, the broad topic announced to participants would simply be " radio in your area." All would have been selected through a screening questionnaire as being either current or potential listeners to FM99. The topics initially listed could be: o quality of reception o information about programs o music on radio o news on radio o announcers o advertising and promotion on air. The moderator would ask each participant in turn if there was anything they wanted to discuss that wasn't covered in that list of topics. Anything that could be included within an already-listed topic is written next to that topic by the secretary. Anything not already covered is added to the foot of the list. When each participant has had a chance to nominate more topics, the agenda is voted on. Before voting, the moderator points out that there will not be enough time to .cover all topics, so participants should not vote for everything. This vote is the first use of the green cards. The moderator announces each topic in turn, asking participants to raise their cards if they would like to discuss that. The secretary counts the number of votes and writes that number beside each topic. The number of votes then becomes the sequence of the agenda. No topic is specifically excluded, but those with the fewest votes are most likely to not be discussed if time runs out.2
4
Consensus Groups Manual
Stage 8. Main discussion
After the introductions and agenda setting, the moderator begins the main discussion, by introducing each topic in turn. The moderator needs to keep an eye on the clock, so that all statements that the majority of participants want to talk about can be covered. If the participants don't already know which organization is conducting the research, it may be best not to tell them just yet, so that their opinions will be more unbiased. In some cultures (much of Asia, for example), when participants know who the research is for, they may be reluctant to criticize it in the presence of people from that organization. But even if they're not told the organization's name at the beginning, they'll usually work it out later. This means that the early parts of the discussion may be the only opportunity to hear unbiased opinions.
When participant take turns to speak, round-robin style, this ensures that everybody has a say, but using this method for an entire session can make conversation very awkward. My preference is to begin and end the discussion phase by asking each participant in turn to speak. For the rest of the discussion phase, anybody can speak, as they wish. Sometimes the moderator may need to intervene to prevent one or two people from dominating the discussion, or to encourage the shyer participants to have their say. Participants can be encouraged to make notes on the back of their preliminary questionnaires. The organizers should not be drawn into the discussion. If a participant asks the moderator "what do you think of this matter?" the moderator should answer "it's your opinions that are important here, not mine. I don't want to influence you." The purpose of the meeting is for the listeners to provide information to the organization that is conducting the research. But sometimes participants try to reverse the flow of information, and begin questioning the organizers in detail. If this gets out of hand, much of the meeting time can be wasted. The moderator should handle such questions by stating that there will be a "question time" at the end of the meeting. The discussion itself can be either structured or unstructured (the difference is explained below), and will typically run for between 30 minutes and 2 hours. The duration can usually be controlled by the moderator. While discussion takes place, the secretary notes down Consensus Groups Manual
25
statements that most of the participants seem to agree with (e.g. if several are waving their voting cards, or indicating agreement with their body language). These statements are not posted on the wall yet, but kept on the secretary's table. Posting statements on the wall too soon seems to stop participants from thinking further about a topic. If anything is posted on the wall at this stage, it should be a question or a topic, not a tentative conclusion. Unstructured discussions
An unstructured discussion is one in which the moderator merely introduces the broad topic, and sits back to let everybody else talk. The advantage of this approach is that participants feel unfettered by restrictions, and may provide useful and unexpected insights. The disadvantage is that much of the discussion may be irrelevant, and will not provide useful information. Therefore it is normal for the moderator to interrupt when the discussion drifts away from the stated topic (for example, if the stated topic is radio, the discussion will often drift onto television). An unstructured discussion will often cover most topics on the agenda, without the moderator having to introduce the topics one at a time. In that case, the moderator should check off the topics as they are covered, by ticking them on the list on the wall. Towards the end of the discussion period, the moderator can ask specifically about high-priority topics that have not been discussed. Structured discussions
With a structured discussion, the moderator introduces topics one at a time, and asks the participants to discuss them. The moderator needs to avoid asking the participants any questions which can be answered Yes or No. (If this is the type of information you need, it's better to do a survey.) In fact, it's best not to ask questions at all. Instead, the moderator should say things like: - "Tell me some of the things you like about the breakfast program on FM99." - "Why do you no longer listen to that program?"
Both the above prompts are the type that seek detailed responses, and are loose enough to allow for unexpected answers. If the questions asked are too specific, you risk not finding out the key facts. Another type of structured discussion is one that involves playing excerpts of programs from a prepared tape. Reactions to specific programs are often more useful than generalized comment. For this 26
Consensus Groups Manual
purpose, we usually prepare tapes of 10 to 20 short program extracts, each around one minute - just long enough to illustrate a particular point, or jog the memories of people who are not sure whether they have heard or seen a program before. Play one item at a time, then let people talk about it for a few minutes.
Many participants in consensus groups have a tendency to vagueness, making comments such as "I'm not very impressed with the news, on FM99." Such statements are not useful, and need to be followed up with questions from the moderator ("What are the things that you don't like about the news?") or asking them to recall some specific examples. Doing this a few times near the beginning of each discussion will show the other 'participants what sort of comment is wanted. The Devil's Advocate
Some societies, particularly in south-east Asia, place a high value on social cohesion. People strongly want to agree with each other, and particularly with participants of high status. This was particularly noticeable in the small city of Bogor, Indonesia, when one participant was a retired general. Most other participants were young, and didn't dare disagree with him. Whatever he said, everybody agreed with. So after that, I created a Devil's Advocate role. In each group, some innocuous-seeming person, often the youngest woman, was given a poster of an ugly monster (a chupacabra, or Mexican goatsucker) taped to a ruler.
She was told that her role would be to notice when something was not being said, or some obvious point was being glossed over - "the elephant in the living room" as some have called it. In such a case she would brandish the chupacabra, and state the overlooked point not necessarily agreeing with it, just stating it. When she spoke without holding up the chupacabra, that was as herself, not in the Devil's Advocate role. The moderator assists by stopping proceedings to let the chupacabra speak. This tactic turned out to be very successful: when assigned that power, meek girls became fireConsensus Groups Manual
27
breathing chupacabras. After hearing observations from the Devil's Advocate, a group will often swung around to a different viewpoint. Types of statement
The secretary will be looking for a mixture of statement types: • • • •
Statements of attitude - often including the word should. Statements about behaviour, such as participants' habits. Statements about knowledge and awareness. Statements about characteristics of participants.
Some statements of type 4 are more easily collected as question in the preliminary questionnaire - e.g. "Do you have a radio at home?" The strength of consensus groups lies in negotiating statements, and statements so simple that no negotiation is needed are not a good use of the group's time.
28
Consensus Groups Manual
A break
By now, discussion will have been going for an hour or so. To conclude stage 8 (the longest) it's time-to take a break of 5 to 10 minutes. During the break, participants can have refreshments, while the secretary and moderator post the statements on the wall. Here's an example of how the wall might look...
(This was after the voting, which is why the statements have counts such as 9/9 at the lower right.)
Consensus Groups Manual
29
Stage 9. Establish the level of consensus
The final stage of a consensus group involves finding out to what extent consensus exists on the issues discussed in the second phase of the session. Taking each statement on the wall in turn, the moderator first asks participants if they understand exactly what it is saying: "Does everybody clearly understand what this statement means, so that you are able to say whether you agree with it or not?" Participants vote by displaying their cards. If even one person expresses doubt, that probably means a lot of others are also doubtful, but don't like to admit it. So unless the vote is unanimous, the moderator now calls for suggestions to clarify the wording. This often results in a statement being split into two. After the clarifying vote, the real vote takes place. The purpose of this type of research is to describe any consensus that exists: in other words, a statement phrased in such a way that the great majority of participants are able to agree with it. Statements in the form initially offered will often need to be altered somewhat so that as many people as possible can agree with them. We regard it as consensus if no more than two people in a group (of up to 12) don't agree with it - this corresponds to around 80%. If only a few people do not agree with a statement, they are asked why. "Could you agree with it if a few words were changed?" the moderator will ask. The principle is that when a statement receives too few votes, it needs to be softened. If a statement expressed in an extreme way is watered down a little, more people will agree with it. This is usually possible. For example, not many participants might agree that: People who do not listen to FM99 should have a loudspeaker set up in the street outside their home and be forced to listen to FM99 all day.
A few more might agree that People who do not listen to FM99 should be given a free radio that receives FM99.
30
Consensus Groups Manual
And even more might agree that: People who do not listen to FM99 should be informed that it exists.
Participants sometimes ask, "What exactly do you mean by agree? Do you mean mild agreement, strong agreement, or what?" One answer: if you're not sure whether you agree or not, then you don't agree. Agreement is meaningless unless it is positive and fairly strong. When consensus has been found, the secretary writes the modified statement on the poster, together with the numbers who agree. For about a quarter of the statements (on average) it is not possible to reach consensus, and the group will divide into two sections, with no common ground. In such a case, try to get two statements, each with as many as possible agreeing, and record both statements, with appropriate annotation. After the secretary has finished going through the statements she or he noted down during the discussion stage, participants can be asked to add statements that they feel most others will agree with. Each of these statements is then modified and voted on in the same way as the others - but usually this happens much more quickly. Finally, the secretary asks the moderator to offer further statements for evaluation. Also at this stage, participants in each group can be asked to vote on statements from previous groups. The second group can vote on statements agreed in the first group, and the third group on statements from the first two groups. Because the wordings of these statements have already been argued over, such voting is normally very quick. The success of consensus groups depends very much on the quality of these statements. There are some statements which most people will agree with, yet which offer no insight into a topic. For example, everybody may agree that the sky is blue - but how does that help? With experience, the moderator can steer the group towards statements that are on the border of being controversial. That is, statements with which most people agree, but they only just agree. When a statement is agreed unanimously, this may mean that it is too weak, and needs to be strengthened. So if 12 out of 12 agree that the sky is blue, that vote stands, but the moderator can suggested a tightened version - perhaps "the sky is always blue." To save Consensus Groups Manual
31
rewriting a statement, the secretary can use a different colour pen to make minor changes to wording, and write (also in that new colour) the revised vote. Imagine all possible statements being spread out, as if on a map. An irregular shape on this map might define the statements with which most people agree. At the centre of the map are the statements which are so obvious that they are hardly worth stating. For example, all regular listeners to a radio station might agree with "I usually like what I hear when I listen to FM99." (Otherwise, they probably wouldn't listen.) Towards the edge of the irregular outline on the map are the borderline statements, at the boundaries of agreement of the station's listeners. An example of a borderline statement might be "Though I usually listen to FM99, I listen to other stations when I don't like the FM99 program." These borderline statements tend to be more interesting, and less obvious. The consensus-seeking stage of the discussion will typically last between 30 minutes and one hour, depending on how many statements are offered for discussion. Sometimes a group will split into two factions, which hardly agree on anything. When that happens (which is rare) the consensus stage will take much longer. Why separate the discussion and consensus stages?
You may wonder why discussing issues and reaching consensus are presented as two separate stages of the discussion. Wouldn't it be more efficient to take each topic one at a time, reach consensus on that, then move on to the next topic? I have tried this, but found it severely impedes the flow of discussion. Also, returning to a topic at the consensus stage gives people more time to gather their thoughts, and consensus seems more easily assessed after a time gap. The exception to this sequence is when the discussion is structured, by being divided into a number of clear parts - for example, when a lot of short program excerpts are being played to the participants, and they reach agreement on each one separately. Insights at the end
Sometimes the most difficult part of running a consensus group is persuading the participants to leave at the end of it. With some groups, nobody wants to go home, and most participants may sit around and talk for an hour or two longer. I find these late sessions very useful. By that time, the participants know each other (and the organizers) much better and may offer insights that they did not
32
Consensus Groups Manual
mention in the formal session. So if you are recording the session, don't turn the tape recorder off until the last person leaves. Format of statements
The outcome of each group is a set of statements. Typically, there are between 20 and 40 statements, each written in large letters on a separate sheet of paper - as shown in the photo on page 29 above. Each statement should be a single sentence, avoiding the words and and or , no more than about 20 words. It's OK for the statement to be qualified in some way - restricting its applicability - but it should be completely clear and unambiguous - very much like a question in a survey. Below the statement is a number, showing how many people (out of those present) agreed with it. For example, this statement was agreed with by 10 of the 12 participants present:
The midday news on FM99 should have much more local content 92
10/12
g2 at lower left means group 2 (of the three groups). When the
findings for all groups are being compared, it will be necessary to distinguish statements from the different groups. Group numbers can be added later, on the conclusion of each group. An alternative is to use different coloured paper for each group - as long as that colour is recorded somewhere!
Consensus Groups Manual
33
Stage 10.CConsolidate the findings
After three group sessions have been held, you will have three sets of statements. One reason for holding at least three groups is that one group may develop its thoughts in quite a strange way, perhaps due to one or two powerful personalities. With two groups, if they produce quite different findings, you won't know if one group is atypical. But with three groups, if the results from one are very different from the other two, this will be obvious. As mentioned above, three is the minimum number of groups: sometimes it's appropriate to hold four or more groups. When consolidating statements from three groups, a lot of paper is involved. As the average group produces around 30 statements, that's about 100 statements to compare. As this would need a very large wall, and only a few people are involved in the consolidation, it's more convenient to rewrite each statement on a small card, and work with the 100-odd cards on a large table, laying them out in a sequence that best aids understanding of the findings as a whole. When several groups agree on the same statement, those cards can be overlaid. Though three groups never come up with exactly the same set of statements, we have always found strong similarities. Any statements that are substantially similar for at least two of the three groups can be safely assumed to be representative of the whole audience sampled - provided that the third group didn't agree on an opposing statement. No matter how differently the three groups are selected, we usually
find a lot of agreement between the lists of statements when the discussions have been conducted in the same way. Observing the similarities will provide confidence that the results are true of the entire population studied, not only of the three disparate sampling points.
34
Consensus Groups Manual
Stage 11. Reporting on the findings
At this point, one advantage of consensus groups over focus groups becomes very obvious. A typical focus group will last about 90 minutes, but it usually takes a full day to transcribe the discussion, and another day per group to analyse the text and produce a report. Though unskilled moderators can usually manage to make a focus group work (after a fashion), high-quality analysis requires much skill and experience. Much of the high cost of focus groups is incurred at the analysis stage. Consensus groups are very different. It's only an hour or two's work to combine statements from three groups: Have the combined list of statements typed out, and this will be the basis of a report resulting from the discussions. Add a description of any taped stimulus material, the criteria used for selecting listeners, and findings from the preliminary questionnaires, write a one-page summary of the process, and that should be sufficient. If you have taped the discussions (whether on audio or videotape) you can extract relevant comments to illustrate each of the agreed statements, and include these in a more detailed report - but that's a lot more work.
Improving with practice
The consensus group technique, though simple, is not simplistic. It can be used even by highly experienced researchers, and analysed in the same as focus groups. As people become more experienced at conducting consensus groups, these improvements occur... o Moderators get better at teasing information out of participants in the allotted time, ensuring that everybody's voice is heard, and reducing disruptive conflicts between participants. o Secretaries get better at writing out the agreed statements in such a way that these statements are more usable by the organization which has commissioned the research, as well as being an accurate reflection of the group's ideas. o The output need not be just a long list of statements: it can be structured into a map-like format, showing (for example) the ways in which people might change their behaviour - such as beginning to listen to a new radio station. The 2004 article by Mike Metcalfe and myself in Technological Forecasting and Social Consensus Groups Manual
35
Change (detalis in Further Information, page 37 below) explains one way of doing this. Extending the method
Consensus groups can also be used as part of a larger study. One extension of the consensus group to a broader situation is the codiscovery conference, which involves about 30 people, takes a full day, and consists of a number of small-scale consensus groups. This is used for two groups of people - e.g. "staff" and "audience"- to infuse a strong tacit understanding of each others' needs. The implicit message of a set of consensus groups (like a survey, or focus groups, or almost any other "research" method) 'is "we, the organizers, are researching you, the passive recipients." In contrast, the implicit message of a co-discovery conference "we are both researching each other, so that we can develop programs that satisfy us both." The first half-day develops mutual understanding, while the second half-day builds on that to plan mutually acceptable programs. The method can also resolve conflicts due to misunderstanding - as long as these are not strongly entrenched. Further details can be found in chapter 14 of my book Know Your Audience, and online at www.audiencedialogue.org/kya14.html
Conclusion
In summary, the consensus group technique is one that can be used by inexperienced researchers with reasonable safety. It is more difficult to draw misleading conclusions with this technique than with other types of qualitative research. And because consensus group statements are generated by the audience involved, consensus group findings can be more relevant and directly usable than are findings from surveys.
36
Consensus Groups Manual
Further information
Online
The Audience Dialogue website at www.audiencedialogue.org has information on related methodologies - currently not a lot on consensus groups that isn't already in this manual, but more will be added in the next few months, including a page on the idea-space mapping of statements from consensus groups. Books
Audience Survey Cookbook, by Dennis List. ABC, Sydney, 1997 (out of print, but available in some libraries). Also available in Vietnamese, . Khmer, and Lao translation. Know Your Audience, by Dennis List. Original Books, Wellington, NZ, 2005. The earlier edition is available in Bahasa Indonesia as Memahami Khalak Anda, published by UNESCO, Jakarta, 2001. Articles in academic journals
"The Consensus Group Technique in Social Research", by Dennis List. Field Methods, vol. 13, no. 3, pp. 277-290. "Sourcing forecast knowledge through argumentative inquiry", by Dennis List and Mike Metcalfe. Technological Forecasting and Social Change, vol. 71, no. 5, pp. 525-535. Contact details
Email: dennis@audiencedialogue.org or audial@ftml.net or dennis.list@unisa.edu.au Mail: Dennis List, Audience Dialogue, 1 East Tee, Nailsworth SA 5083, Australia Telephone: +61882695321 or 0405983891
Consensus Groups Manual
37
Consensus groups -notes for moderators , The main duties for a moderator are 1. Introduce the topics for discussion 2. Manage the time available 3. Encourage the least talkative people to speak 4. Stop the most talkative people from dominating the conversation Sequence of work Welcome everybody and introduce the broad topic. Explain use of voting cards (hold them up when you agree with the present speaker), as secretary hands them out. 1. Introduce everybody (about 10 minutes) Moderator begins by describing his or her own habits - about 1 minute Ask each person in turn. If the first few people don't give enough detail, ask for more. 2. Select agenda. (Draft agenda is on wall.) Explain the draft agenda, then ask each person in turn to nominate an area of interest - something they want to talk about, that's within the broad topic, but not covered in the draft agenda. [Secretary writes down new ones.] When everybody has had a chance to add something, vote on priorities. Explain that this will mean that the least popular items will be discussed last, and may not be reached at all. Ask participants not to vote for everything. Time: about 10 minutes. 3. Main discussion. Introduce each sub-topic in descending order of popularity on the agenda. Ask each person in turn to comment, round-robin style. After several roundrobins, discussion can become more free-form. At the end of each sub-topic, do another round-robin. Time: about 40 minutes to 1 hour. Break Have a 5 minute break, while the secretary and moderator attach the draft statements to the wall where everybody can read them. Voting (40 minutes to 1 hour) Point to each statement in turn, and read it aloud. Ask first: "Please raise your card if you are sure you completely understand the meaning of this statement." If everybody raises their card, vote on that statement. If even one person is uncertain, ask them how the wording might be improved (just to clarify it, not to change the meaning). Everybody else can now discuss this. When the wording is agreed, the secretary changes it. Then vote again on (a). When everybody agrees they know what it means, they can vote: "Now I'd like you to vote on whether or not you agree with this statement. Just hold up your green card if you agree."
38
Consensus Groups Manual
Consensus groups -Notes for secretaries
'.
Introduction (about 10 minutes) Give each person a voting card. (The moderator will explain their use.) Make a simple map of the seats in the room, and write on the map the name of each participant. Put the draft agenda on the wall. Setting agenda (about 10 minutes) Moderator will ask everybody in turn to suggest a topic. If the topic is not already on your list, add it to the foot of the list - large letters, so everybody can read it. If it comes under an existing topic, write it beside that topic. When moderator asks for vote, write number of votes for each topic next to that topic. This forms the order of discussion. Main discussion (about 40 minutes to 1 hour) Listen to whoever is speaking but also watch everybody else. If 3 or more seem to be agreeing (e.g. holding up their voting cards, nodding heads, etc) summarize the statement and write it in large letters on a sheet of A4 paper -one statement per sheet. Try to limit statements to about 10-15 words. Make sure the statement is about a single concept, and is not ambiguous. If it contains the words "and" or "or" it might need to be split into 2 statements. Break (about 5 minutes) Using masking tape, stick all the statements to the wall where everybody can see them clearly. Use same sequence as discussion. Voting (40 minutes to 1 hour) Discuss each statement in turn. The moderator first asks if there are any questions about the meaning of the statement. If so, everybody is asked to hold up their card. If even one card is not held up, the statement needs to be clarified. In that case, the moderator will ask for suggestions to improve the wording, then voting is repeated. The secretary writes the changed wording. If only one or two words are changed, the secretary crosses out the changed words and writes in new ones. If more words are changed, it's usually easier to write a whole new statement. When everybody is clear about the meaning, that statement can be voted on. The moderator asks all participants if they agree - if so, they should hold up their cards. Count the number voting for each statement, and write that number on the same page as the statement, e. g. 11/12 means 11 out of 12 people present voted for it. If a statement is weak and everybody agrees, you or the moderator can suggest stronger wording. If only a few words change, make changes with a different colour pen and record votes on the same sheet. At the end Write the group number or location on each statement sheet, e.g. at bottom left. Consensus Groups Manual
39
Materials that may be needed for a consensus group o Blank sheets of A4 paper - up to about 100 of them. Can use a different colour paper for each group. o Thick marking pens, two dark colours (more than one pen of each colour - they don't last long). o Roll of masking tape, 15 to 30 mm wide. o Voting cards (thin card, bright green· or other bright colour· about as big as the palm of a hand - 1 per person, plus a few spare) o Cardboard for name tents (4 name tents per A4 sheet - 1 tent per person) o List of names of those who have agreed to attend (check off as they arrive) o Preliminary questionnaires (1 per person, plus a few spare) o Cheap pens (1 per person, plus a few spare)
Optional extras o Ethics/privacy/data use statement (1 per person, plus a few spares) o Information sheets for· participants o Copies of stimulus material (1 per person) o Recording sheet (1 per person, plus a few spare) o A roll of small sticky dots for voting (can measure off a strip for each person) o Money to pay participants o Gifts for participants o Envelopes for payment money o Receipt/s to be signed by participants o Clipboards to rest questionnaires on (if people aren't sitting at tables) o Name tags (instead of name tents· if people aren't sitting at tables) o Camera, and media for it (film, tape, or digital cards). Check batteries!
Recording materials (if session is recorded) o Recorder- cassette, video, minidisc, or digital voice recorder. The secretary (or whoever is operating it) needs to be thoroughly familiar with its operation. o Microphone- omnidirectional, not unidirectional or cardioid. Safer to bring 2. Preferably boundary or PZM type, with flat metal base. Some video cameras have adequate omnidirectional microphones built in. o Battery tester (if recorder is battery operated • need to test batteries before each group) o Spare batteries for recorder and microphone o Long extension leads- to allow for distant power outlets o Recording medium (e.g. blank tapes): enough for at least double the expected duration of the group discussion. 40
Consensus Groups Manual
'.
Example of preliminary questionnaire Date .. / ../ ..
Time .....
Place. . . . . . . . ... . . . Your initials ....
Please tick all the boxes that apply to you: [] Male
[] Female
[] Age 18 to 24
[]
[] Working full time [] Home duties
[] Working part time [] Retired
Age 25 to 34
[] Age 35 to 54 [] Not working
[] Age 55 or over [] Student
Which suburb or locality do you live in? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . In a typical week, about how many days do you read a newspaper? Which newspapers do you read, regularly or occasionally?
Are there any particular aspects of newspapers that you'd like to bring up in this discussion?
Any other comments you'd like to add?
Consensus Groups Manual
41
Dennis List is principal of the communications evaluation consultancy Audience Dialogue, as well as being a Research Fellow at the School of Management at the University of South Australia, where he has recently completed a PhD. He has been working in media research and associated areas for more than 20 years, with the New Zealand and Australian Broadcasting Corporations, and has done consulting work for international agencies including the Swedish Media Development Office, the Media Development Loan Fund, and several other high-profile international organizations. He has published several books on media research methods, including Know Your Audience: A Practical Guide to Media Research (2002, 2005) and Participative Marketing for Local Radio (2003), both available from
this publisher. The text of most chapters of those books is available on the Audience Dialogue website, www.audiencedialogue.org
42
Consensus Groups Manual