Learning in Conferences by Olivier Serrat

Knowledge Solutions

December 2011 | 110

Learning in Conferences By Olivier Serrat

No Loose Change

The true value of a conference lies in its effects on participants. Conferences are to generate and share knowledge that impacts behavior and links to results: this will not happen if the state-of-the-art of conference evaluation remains immature and event planners do not shine a light on the conditions for learning outcomes.

Lest we forget, a conference is a purposeful gathering of people aiming to pool ideas on at least one topic of joint interest or needing to achieve a common goal through interaction (and, naturally, relation). They are face-to-face, sometimes virtual,1 venues for situated learning dedicated to the generation and sharing of knowledge, usually to reach agreement, in formal or informal (yet planned) settings. Conferencing, then, is an age-old technique for reasoning and problem solving, aka sense making, the process by which people give meaning to experience through spoken and written narratives. Certainly, the Socratic Method—a debate between individuals with opposing views that used effective questions to stimulate critical thinking—was a form of it (and the oldest known way of teaching). Nowadays, new modes of transport and communication mean that conferences can take many forms including (i) conventions—large meetings of delegates, industries, members, professions, representatives, or societies seeking concurrence on certain attitudes or routines, such as processes, procedures, and practices; (ii) forums—broad occasions for open discussion, as a rule among experts but now and then involving audiences; (iii) seminars—prolonged and sometimes repeated meets for exchange of results and interaction among a limited number of professionals or advanced students engaged in intensive study or original research; (iv) workshops—brief educational programs for small groups of peers focusing on techniques and skills in a particular field; (v) retreats—periods of group withdrawal from regular activities for development of closer relationships, instruction, or self-reflection; and (vi) meetings2—sundry instances of coming together for business, civic, courtship, educational, government, health and wellness, leisure, religious, social, sports, and other functions. Minds that have nothing to confer find little to Conventions, forums, seminars, perceive. workshops, retreats, and meetings— —William Wordsworth to which the emerging practice of

Conferences ordinarily occur face-to-face but information and communication technology now also enables virtual mediation—through telephone and, increasingly, video—between people who are geographically separated. In the workplace, one-time, recurring, or series of meetings that regularly draw on the services of a chairperson include briefings, advisory meetings, committee meetings, council meetings, and negotiations. The Knowledge Solutions on conducting effective meetings describe the nature and challenges of each. See ADB. 2009. Conducting Effective Meetings. Manila. Available: www.adb.org/documents/information/knowledge-solutions/ conducting-effective-meetings.pdf. Other types are ad hoc, investigative, one-on-one, team, and work meetings.

Knowledge Solutions

“unconferencing”3 should hereafter be added—are a pervasive form of interaction. The resources allocated to their organization, conduct, and attendance—of which the opportunity cost incurred from taking part is no loose change—must surely be astronomical. Even so, we seldom assess their relative value to either participants Meetings are a great trap. Soon you find or event planners. (Run-of-the-mill, end-of-session surveys yourself trying to get agreement and then requesting participants to jot down what they enjoyed the people who disagree come to think they or disliked—namely, to log reactions—will no longer have a right to be persuaded. However, they do.) Granted that conferences serve different purposes, are indispensable when you don’t want to do these Knowledge Solutions concentrate on gatherings that anything. are ostensibly designed to generate and share (relevant, —John Kenneth Galbraith effective, and therefore valued) knowledge4 and leverage related networking in support, such as forums and seminars. [That said, given the claims that other meetings make about knowledge generation and sharing—pace the disconnect between their means and ends, it stands to reason (and would indeed be logical) that these Knowledge Solutions also apply there.]

The Poverty of Conference Evaluations

Questionnaires are synonymous with conference evaluation. (Indeed, few other tools seem to be used.) In all probability, the language that event planners employ to allegedly gauge conference satisfaction—they hardly ever dare establish outcome and impact—will read: “Thank you for taking time to complete this survey. Your opinion is important: it will inform plans for the next event.” All too predictably when some pretense at conference evaluation is in fact made, the following “key” questions will be posed: Did the event lead to its goal?5 True genius resides in the capacity for What were its main strengths and weaknesses? What did you evaluation of uncertain, hazardous, and value in the event?6 Were the sessions7 relevant to the subject conflicting information. —Winston Churchill matter? How well did they align with your expectations? Can you rate the quality of the presenters? Has your knowledge of the subject matter increased as a result of the event? Will the event set in motion changes in the way you work in the future?8 What undertakings can you now initiate? How might the event be improved? (For sure, there will also be open fields inviting further suggestions for improvement.) Be these as they may, the politically incorrect question must be asked: What might data compiled from chiefly formative, not summative, quizzes possibly help validate or change in any meaningful way?9 Stating the obvious, feedback that cannot be used should not be sought. Ironically, since a dog’s tail should not wag its owner, what practical recommendations

Unconferences are participant-driven meetings. The label has been ascribed to a wide range of gatherings—made possible by the spread of personal computers and the internet over the last 25 years—based on the premise that in any professional gathering the people in the audience, not just those who speak on stage, also have know-what, know-how, and know-why to communicate. The tools, methods, and approaches of unconferences include Barcamp, Birds of a Feather, Conferences That Work, Everyday Democracy, Fishbowl Conversation, Future Search, Knowledge (or World) Café, Lightning Talks, Open Space Technology, Pecha Kucha, Speed Geeking, and TeachMeet. 4 Here, it may be pertinent to distinguish two types of knowledge: instrumental (or procedural) knowledge that can be exploited in a reasonably well defined and specific way; and conceptual knowledge that shapes the way people consider issues but that may not have an obvious, direct impact. 5 It goes without saying that the absence of a clearly stated goal and related objective(s) complicates evaluation. (From the onset, it also makes it well-nigh impossible to decide on the best learning strategy, resources, and logistics.) 6 As you would expect, since surveys more often than not deliberately require that names and professional titles be recorded—not counting demographic and other information captured during the registration process or before that through short surveys to gather baseline data and perhaps help shape the event—positive responses to this question will help craft testimonials that, with social proof, are intended to lend credence to the marketing campaign for the next event. 7 This is shorthand: conferences can bring into play a mix of keynote addresses, presentations, panel discussions, roundtable discussions, breakout sessions, workshops, hands-on labs, and luncheons. 8 The selection of the conference goal and its objective(s) should permit event planners to identify the terminal, applied behavior of participants by name, viz., the kind of behavior that would be accepted as evidence that the conference has achieved its objective(s). Therefore, the definition of the desired behavior should describe the important conditions under which it would be presumed to occur. Criteria of acceptable performance should also be specified. 9 The areas that would stand to benefit from evaluation are the event planner’s operations and conference programs. The first includes aspects such as planning processes, decision-making procedures, personnel, physical facilities, public relations, and administration and management. The second would include objectives, clientele, methods and techniques, materials, and quality of learning outcomes. 3

Learning in Conferences

for improvement are proposed will probably be turned down, with thanks, as great but simply not possible given this or that constraint. There has, of course, been much debate over the near-universal reliance on questionnaires for conference evaluation. Detractors wonder if they really provide worthwhile information;10 adherents research how to obtain a representative cross-section of attendants since, more often than not, there is no strong motivation to respond—they remark that surveys can (at low cost) ensure One test of the correctness of educational at least summarily uniform coverage of all information areas procedure is the happiness of the child. deemed essential; provide an opportunity to triangulate —Maria Montessori results using different techniques; and allow the same questions to be submitted in the same way year after year so that evaluation results can be compared against a baseline. Innovators advocate "recent life histories" that highlight the event's influence on selected individuals, for example in terms of education, networking, professional development, and application of knowledge gained; or "roving reporters" who would converse with participants throughout the event with a mix of demographic, short-answer, Figure: Linking Conferences to Results and open-ended questions.11 Put bluntly, the value that conference evaluations add is incongruously scant. In declining order of interest—with variations depending on the sector, theme, and discipline addressed, evaluations home in on (i) the overall reactions of participants, (ii) conference strengths and weaknesses, (iii) ratings of sessions and presentations, (iv) ratings of the extent to which the needs of participants were met, (v) areas for improvement, (vi) financial return on investment, (vii) participant learning in the short term, and (viii) new behaviors in the medium term. The case must be made that the last two areas demand more attention. And, there surely is scope for Donald Kirkpatrick’s four levels of learning evaluation, even if they were developed in 1959 for the evaluation of training programs.12 With minor modifications to adapt them to the context of conferences, the levels are as follows: • Reaction—To what degree do participants react favorably to an event? • Learning—To what extent do participants acquire the intended knowledge, skills, attitudes, confidence, and commitment based on their participation in the event? • Behavior—To what degree do participants apply what they learned during the event when they return to their job? • Results—To what extent do targeted outcomes occur as a result of the event and subsequent interaction and relation? Source: Author.

Questionnaires are complicated instruments: when well designed, a repeated mistake is to try to read too much into responses; when badly designed, they mislead. If the response rate is high they can describe the broad characteristics of a large population but, all in all, the quality of data is not as high as with alternative methods of data collection such as structured and semi-structured individuals and focus group interviews. Certainly, they are an unsuitable means of evaluation in complex and chaotic contexts where probing and sensing are required—there is no real possibility to follow up on the responses they elicit. 11 In addition to questionnaires, quantitative and qualitative methods of data collection for conference evaluation can—yet too infrequently— include (i) individual face-to-face or telephone interviews; (ii) focus group interviews; (iii) online surveys; (iv) printed surveys; (v) structured observations of conference sessions and areas; (vi) examination of conference programs and online resources; (vii) review of statistical data on conference registration, abstracts, etc.; (viii) review of statistical data and evaluation findings from previous events; (ix) rapporteurs; (x) “instant” feedback systems; (xi) social network analysis; (xii) analysis of media coverage; and, more recently, (xiii) scrutiny of posts left by participants on conference display boards, web logs, Facebook pages, and Twitter accounts. 12 Donald Kirkpatrick and James Kirkpatrick. 2006. Evaluating Training Programs: The Four Levels. Berrett-Koehler Publishers. 10

Knowledge Solutions The Poverty of Learning in Conferences

Most conferences are called to achieve a shared goal—that, ultimately, being collaborative learning that links to results—yet dispense at best information; they do not generate knowledge. Participants depart with their own learning13—that, as noted above, is rarely evaluated and, in the first instance, not necessarily shared. This is because most conferences funnel programmed information; they do not know what potential collaborative learning, if any transpired, could enrich theory, research, and practice in their domain as a whole. Why? Diane Chapman No grand idea was ever born in a conference, et al.14 remark that event planners assert they want to create but a lot of foolish ideas have died there. spaces for learning but do not evaluate if that, and the changes —F. Scott Fitzgerald in behavior linking to results it should conduce, actually did occur. Rather they aggregate individual responses, thereby missing opportunities for subtler analyses of more diverse inputs. Helpfully, Chapman et al. remind us that, from an etymological perspective, to evaluate is to ascertain or fix the value of something; more profoundly, and typically after careful appraisal and study, evaluation helps establish its significance, worth, or condition. The first definition suggests determination of positive or negative effects; the second embraces the idea of determination of condition, which removes the requirement to assign worth. Evaluation techniques that rest on the first definition serve accountability; those that spring from the second propel learning. Chapman et al. posit a three-pronged “New Learning” conceptual framework integrating notions of learning organizations,15 communities of practice,16 and knowledge creation17 to facilitate learning in conferences—not forgetting their evaluation—which uncovers fertile ground for research and practice. Table: Characteristics of New Learning and Evaluation Characteristic

New Learning

Evaluation

Analysis of information

Involves capturing and codifying

Involves judging, valuing, and rating

Comparative nature

Is relative to the event

Is comparative in nature

Time

Focused on the future

Focused on the past

People

Focused on learners

Focused on presenters

Type of information

Focused on unknowns

Focused on what is known

Guiding questions

Why? How?

Good? Bad?

Impact on environment

Globally situated

Locally situated

Breadth of analysis

Holistic and on-going

One-time snapshot

Use of results

Ability to build from new learning to new learning

Usually limited to one use

Use

Change

Status quo

The world over, people attend conferences, take notes, and hardly ever (if at all) study them again. (Back-to-office reports are another ineffective receptacle of findings and conclusions.) Never mind collaborative learning: to boost their individual learning, participants might, for example (i) draw 2–3 objectives, personal or organizational; (ii) listen actively; (iii) ask questions; (iv) plan a networking strategy; (v) tap by design the ideas of fellow participants; and (vi) take time to reflect and innovate, away from the rigors of daily schedules. 14 Diane Chapman, Colleen Aalsburg Wiessner, Julia Storberg-Walker, and Tim Hatcher. 2006. New Learning: The Next Generation of Evaluation? North Carolina State University. pp. 1–8. 15 The Knowledge Solutions on the dimensions of a learning organization and how it might be built present useful models associated with learning and change and elucidate ways to support and energize organization, people, knowledge, and technology for learning. See ADB. 2009. Dimensions of the Learning Organization. Manila. Available: www.adb.org/documents/information/knowledge-solutions/dimensionslearning-organization.pdf; and ADB. 2009. Building a Learning Organization. Manila. Available: www.adb.org/documents/information/ knowledge-solutions/building-a-learning-organization.pdf 16 The Knowledge Solutions on building communities of practice explain how groups of like-minded, interacting people filter, amplify, invest and provide, convene, build, and learn and facilitate to ensure more effective creation and sharing of knowledge in their domain. See ADB. 2008. Building Communities of Practice. Manila. Available: www.adb.org/documents/information/knowledge-solutions/buildingcommunities-practice.pdf 17 Ikujiro Nonaka’s SECI Approach, for one, is relevant: it isolates four processes through which tacit and explicit knowledge interact: (i) socialization, (ii) externalization, (iii) combination, and (iv) internalization. See Ikujiro Nonaka. 1994. A Dynamic Theory of Organizational Knowledge Creation. Organization Science. Vol. 5, No. 1, pp. 14–37. The Knowledge Solutions on intellectual capital enumerate tools, methods, and approaches for value creation besides others for value extraction and value reporting. See ADB. 2011. A Primer on Intellectual Capital. Manila. Available: www.adb.org/documents/information/knowledge-solutions/primer-intellectual-capital.pdf 13

Learning in Conferences

Characteristic

New Learning

Evaluation

Critical nature

Constructive, embracing

Judgmental, discriminatory

Source of information

Focused on both the planned and unplanned and on both the

Focused on the planned and the formal

formal and informal Audience

Everyone

Formal stakeholders

Purpose

What can we do?

How did we do?

Social implication

Promotes collaboration

Promotes confrontation

Impact

Aimed at field, profession, research, and practice

Aimed at events, programs, and presentations

Source: Adapted from Diane Chapman, Colleen Aalsburg Wiessner, Julia Storberg-Walker, and Tim Hatcher. 2006. New Learning: The Next Generation of Evaluation? North Carolina State University. pp. 1–8.

The nascent practice of unconferencing, cited earlier, bodes well too. Summarizing, the shortcomings of conferences are that: (i) conference programs are set by event planners and do not predict well what sessions are actually wanted; (ii) a distinction is made between presenters (teachers) and participants (learners); (iii) sessions are dominated by presenters; participants receive predetermined information passively; (iv) logistics revolve around general and breakout sessions; (v) content is broadcast in long, uninterrupted sessions; and (vi) chances to network are restricted to meals and social gatherings outside sessions. In contrast, some characteristics of unconferences are that: (i) the culture of unconferences is participatory, not passive; (ii) the intellectual capital of participants, not presenters, is harnessed; (iii) unconferences give time for individualized knowledge sharing and learning: the intent is not just to work toward the goal of the event; (iv) knowledge sharing and learning happen in I believe we are going to move into a situation where the more effective conferences small groups rather than in sessions; (v) interaction is put center stage; (vi) participants have greater input and control will be smaller, more specialized, more over sessions and are thus more apt to engage in knowledge focused, with occasional large gatherings to sharing and learning that help realize the goal of the event; get the attention of the larger world. —Maurice Strong (vii) teaching and learning roles are not fixed; and (viii) sessions can be created on the spot. To note, however, event planners still do not take advantage of unconferencing despite improved connectivity; the chief explanation is fear that unconferences will not work, fuelled by understandable concern over loss of control over one's event and general unfamiliarity with associated facilitation requirements, technical and logistical considerations, and revenue models.