Formative evaluation 3

U8. El Modelo ASSURE para el Diseño Instruccional: Evaluación y Revisión.

3. - Formative Evaluation Related Textbook Readings • •

Dick, Carey and Carey (2001) Chapter 10 - Designing and Conducting Formative Evaluations.

Dick, Carey and Carey (2001) Chapter 11 - Revising Instructional Materials

Introducción al Diseño Instruccional

Introduction A fundamental component of any instructional design effort is formative evaluation.

In the 1960’s, problems with using untested instructional materials became apparent with the arrival of large curriculum development projects. At that time, "evaluation" was defined as the determination of effectiveness of an innovation in comparison to other existing products. Such studies found relatively low student achievement with new curriculum materials. In reviewing the results, Cronbach and Scriven (cited in Dick, Carey, & Carey, 2001) concluded that we must expand our notion of evaluation. They proposed that developers conduct what is now known as formative evaluations the collection of data and information during the development of instruction to improve the effectiveness of the instruction. Research indicates that by evaluating materials prior to releasing a final version of the instruction makes a significant difference in the effectiveness of materials.

However, it is equally important to conduct formative evaluations on existing materials as well as newly developed materials. Dick, Carey, and Carey (2001) do an excellent job describing and providing examples of how to conduct expert reviews, one-to-one, small group, and field test formative evaluations. So, be sure to read the related sections in the course textbook. I will only briefly summarize a few of the key concepts here. In addition, I will: • • •

Present a model illustrating the relationships between basic formative evaluation concepts; Identify limitations associated with the Dick, Carey, and Carey procedures; and Examine usability engineering techniques.

U8. El Modelo ASSURE para el Diseño Instruccional: Evaluación y Revisión.

Formative Evaluation Model

Introducción al Diseño Instruccional

Figure 10.1 Model identifying relationship between key formative evaluation concepts Figure 10.1 presents a model that may help to increase your understanding of formative evaluation. It views formative evaluation as a process that occurs within and is bounded by the goals and constraints of a specific instructional project (Bordonaro, McAlpine, & Weston, 1995). The model defines two primary stages of the process: data collection and revision. Data collection occurs when draft instruction is tried out and feedback is collected. Revision is when evaluation is used to improve the instruction. The model represents these stages as overlapping to show that these stages may occur concurrently. From theses stages the participants of the evaluation are defined. The participants exist on a continuum from novice to expert based upon their initial knowledge of the learning domain.

Roles are then assigned to the participants. Roles focus on the tasks given to participants during the process. From the roles, methods and situations are determined. Methods are defined as the procedures, techniques, and instruments used during the formative evaluations. The situations define the context in which the formative evaluation takes place. These situations occur individually, in dyads, small groups, or large groups. Bordonaro intend this model to be a generic description of the formative evaluation process. It is not designed to dictate a specific path through the process, but to merely describe the landscape.

Review of Key Formative Evaluation Concepts In Chapters 10 and 11, Dick, Carey, and Carey (2001), describe four types of formative evaluations and discuss techniques for analyzing results and revising instruction. To help focus your attention on key concepts, I will review the four basic phases of the formative evaluation process, including: • • • •

Expert Reviews, One-to-One Evaluations, Small Group Evaluations, and Field Trial.

To gain apt knowledge of the formative evaluation process, please be sure to read the assigned chapters in the course textbook.

U8. El Modelo ASSURE para el Diseño Instruccional: Evaluación y Revisión.

Expert Reviews

Introducción al Diseño Instruccional

Although it is assumed that the instructional designer is knowledgeable about the content area, or is working with content specialist, there are reasons why you may want other experts to review your instruction before implementation. First, as most of you already know, it is difficult to review and edit your own work. Second, particularly with the use of technology-based delivery systems, there are a number of design issues that the instructional designer or content specialist ma y not have the expertise to address.

In general, you may want to elicit help from three areas of expertise: (a) subject matter, (b) learning and (c) media. Subject matter experts (or SMEs) should be asked to comment on the currency, accuracy, and adequacy of the information provided within the instruction. Learning specialists may be asked to review the instructional strategy and assessment methods. Media specialists are particularly important when developing video, computer, or web-based instruction. They may be asked to examine quality of the user interface, navigation scheme, video, and/or media resources. Typically, experts are asked to review prototypes or initial drafts of the instruction, as well as design documents, such as goal, instructional, learner and context analysis reports. Revisions may be made before presenting the instruction to target learners for one-to-one evaluations.

One-to-One Evaluations The purpose of the one-to-one evaluation is to identify and remove the most obvious errors in the instruction and to obtain initial reactions to the content from learners. During this stage, course developers work directly with the person evaluating the materials. Direct interactions distinguish this phase of formative evaluation from others. Observation forms, interview questions, attitudes surveys, and achievement tests are the most frequently used instruments for gathering data during this phase. Before achievement tests are used to evaluate student achievement, they should also be formatively evaluated to help ensure validity and reliability.

Participants for the one-to-one evaluations typically include one learner who is above average in ability, one who is average, and one who is below average. The instruction is revised after each evaluation and you may choose to conduct more one-to-one evaluations if necessary. The primary criteria and decisions to be made during one-to-one evaluations include: Clarity: Is the message, or what is being presented, clear to individual learners?

Impact: What is the impact of the instruction on individual learner’s attitudes and achievement of the objectives and goals?

Feasibility: how feasible is the instruction given the available resources (time/context)?

(Dick & Carey, 2001, p. 286)

U8. El Modelo ASSURE para el Diseño Instruccional: Evaluación y Revisión.

I also recommend that you review Table 10.1 on page 287 to review the types of information that may be gathered and analyzed for each criterion. Methods for analyzing data from the one-to-one evaluations are covered in Chapter 11 of the course textbook. Figure 10.2 illustrates slightly modified version of how Smith and Ragan (1999) recommends displaying one-to-one data is illustrated below.

Introducción al Diseño Instruccional

Target

Data Source

Information Gained

Revision Decision

Figure 10.2. Sample table format for illustrating one-to-one formative evaluation data.

In Figure 10.2, Target refers to the component of the instruction that was reviewed (e.g., achievement of a particular objective, user interface, icon, syllabus). Data source refers to the method and/or instrument that were used to gather information. Information Gained summarizes the data gathered from the one-to-one, and Revision Decision describes the actions to be taken based on the information gained.

Small Group Evaluations The purposes of small group evaluations are to determine the effectiveness of changes made following the one-to-one evaluation, to identify any remaining learning problems that students may have, and to determine if learners can use the instruction with little to no interaction with the course developers or the instructor.

The basic procedure used for small group evaluations differ from the one-to-one evaluations in that after a giving a preliminary overview, you administer the instruction in an environment that closely resembles its intended setting. The evaluator intervenes only if equipment fails, or if for some reason, learners get stuck and cannot continue. During this phase, pretests are often used to measure

U8. El Modelo ASSURE para el Diseño Instruccional: Evaluación y Revisión.

learners’ prerequisite skills and posttests are used to measure performance on the primary instructional objectives. Attitude questionnaires and follow-up interviews are also conducted. Feasibility of the instruction is also evaluated at this stage by determining the time required by learners to complete the instruction and required performance measures, estimating the costs and viability of delivering instruction in the intended format and environments, and assessing the attitudes of those implementing and managing the instruction. In ideal situations, participants would be randomly selected so that your findings could be generalized to the target learner population. When random selecting is not possible, subgroups (based on ability, knowledge of the content matter, and experience with the delivery system, among others) are often selected from the target population.

A sample of key questions that should be answered during small group evaluations, along with intended outcomes is noted on page 293 of the course textbook.

Methods for analyzing, illustrating and interpreting small group evaluation data are provided in Chapter 11.

Introducción al Diseño Instruccional

Field Trial This final stage of the formative evaluation is where you will attempt to apply the instructional materials in a learning environment that mimics its intended setting.

The purpose of this stage is to decide if the alterations in the instruction made after the small group stage were effective and if the instruction can be used under targeted conditions. To answer these questions, all materials should be ready to go.

If an instructor is involved in executing the instruction, the designer should not play this role.

The procedure and instruments used during field trials should be very similar to that used for the small group evaluations. The primary change is in your role as a designer. If you are not the instructor, you should only observe the process. At this stage the primary concern in feasibility. You should also measure learners’ attitudes and performance; observations and interviews with learners and instructors are valuable.

A more detailed description of field trial evaluations is provided on pages 294-295 of the Dick, Carey, and Carey (2001) course textbook. Specific methods for analyzing, reporting, and revising materials-based field trial data are also provided in Chapter 11. I also recommend that you review the concise listing of formative evaluation activities on pages 305-310 and the examples of evaluations instruments provided on subsequent pages of the textbook. Refer to the list and consider using similar instruments when generating your formative evaluation plans.

Dick, Carey, and Carey (2001), also describe additional important formative evaluation concepts that are only listed here. Before you complete this chapter, be sure to note (a) why it may be important to consider formatively evaluating the transfer of learned skills and knowledge to the performance context, (b) issues related to the formative evaluation of selected materials, and (c) techniques for

U8. El Modelo ASSURE para el Diseño Instruccional: Evaluación y Revisión.

formatively evaluating instructor-led instruction. Also, be sure to note that the tools and techniques for formatively evaluating instructor-led materials may be readily adapted to the formative evaluation of self-instructional print.

While Dick and Carey provides an extensive treatment of formative evaluation methods, their techniques may leave you, and other instructional designers wanting. Can you think of some reasons why?

Limitations

Introducción al Diseño Instruccional

While Dick, Carey, and Carey’s (2001) methods are seen as effective for addressing self-instructional print, video, and instructor-led materials, it is believed that they may not be adequate for formatively evaluating technology-based instructional delivery systems. Specifically, they neither identify nor address key issues related to computer interface design. To extend your knowledge of formative evaluation techniques, I will introduce several usability engineering tests that are aimed at improving computer systems by removing errors and making interfaces easier and more pleasant to use. I also recommend that you to consider rapid prototyping as a prima ry formative evaluation technique (previously discussed in Chapter 9) Usability Engineering and Testing Usability is a general term for a large number of attributes related to users’ performance with computers. Usability engineering is a way of measuring these attributes. It is playing an increased role with software development as evidenced by the growing budgets for usability testing. In 1971, Shakel (1971) established that a reasonable share for usability budgets for non- military computer systems was about 3%. Later, Wasserman (1989) found that leading companies were allocating about 4-6% of their development budget to usability testing. Neilsen (1993) surveyed 31 development projects and found that the median share of their budgets allocated to usability engineering was 6% (Neilsen, 1993).

Inadequate use of usability engineering methods in software development has been estimated to cost the US economy $30 billion per year in lost productivity (Neilsen, 1977). Neilsen estimates that bad web design will cost $50 billion per year for the world economy in 2001. Applying usability testing techniques during your formative evaluation should help you avoid costly errors. I will outline six specific usability techniques: • • • • • •

Card Sorting Icon Intuitiveness Test Card Distribution to Icons Thinking Aloud Page Walk Through Heuristic Evaluation Scenario-Based User Testing

U8. El Modelo ASSURE para el Diseño Instruccional: Evaluación y Revisión.

Card Sorting The Card Sorting usability engineer technique analyzes the overall organization and structure of a web site or other computer program. This test estimates the distinction between a novice and an expert user's understanding of the contents and organization of a system. It is often used as a tool for building menu trees.

Also, card sorting can be used to identify cognitive models of an information space.

The audience, procedure, and outcome involved in this testing process are described: Audience A sample of 3-5 users who are representative of the target population along three main dimensions of experience (Figure 10.3): •

Introducción al Diseño Instruccional

• •

general knowledge about computers, specific system expertise, and understanding of the task/subject matter domain

Impacto

Claridad

Factibilidad

Figure 10.3 Dimensions of User Experience

Procedure 1.

2.

3.

4.

5.

Define the purpose and procedure for the evaluation.

Develop necessary instruments.

Determine who will be the evaluator.

Arrange for the evaluator to participate.

Discuss the purpose, targeted result, and process of the evaluation with the evaluator(s).

U8. El Modelo ASSURE para el Diseño Instruccional: Evaluación y Revisión.

6. Ask users/evaluators to declare as many concepts as they can think of related to a particular domain. Concepts that are mentioned close together are assumed to be associated closer in the user's mind.

7. Generate a comprehensive list of major concepts to be included in your instruction and write each concept down on index cards (one concept per card). If a user interface, with defined options, has already been created, write each option down, along with key concepts to be accessed through each option on separate index cards.

8. Ask users to sort cards into logical groups; cards/concepts that get sorted together should probably be in the same menu option.

An alternative technique for accomplishing the goal is to give users a questionnaire listing of all possible pairs of concepts and ask them to rate their similarity.

Outcome

Introducción al Diseño Instruccional

Data describing organization of novices’ and experts’ content area knowledge. This information provides insights on how to group and organize information presented in your instruction. This information is believed to be particularly useful in designing an overall menu structure and/or determining if any changes are necessary to the.

Icon Intuitiveness Test Icon intuitiveness tests are used to determine if the meaning of icons intended by designers is congruent with users’ interpretation of the ic ons. They can also be used to examine users’ interpretation of figures, navigation buttons, or other graphic elements. In addition, Icon intuitiveness tests can also help refine the overall organization of information contained in your program. Such tests are often based on the results of a card sorting study. Icons are developed for each major category of data (aka. Menu option) and users are asked what they think each icon symbolizes. They can also be asked to describe the types of information they think is accessible by selecting the icon. The audience, procedure, and outcome involved in this testing process are described: Audience A sample of 3-5 users who are representative of the target population along three main dimensions of experience (see Audience description provided for Card Sorting above). Content experts can also provide effective feedback.

U8. El Modelo ASSURE para el Diseño Instruccional: Evaluación y Revisión.

Procedure 1.

2.

3.

4.

5.

Define the purpose and procedure for the evaluation.

Develop necessary instruments.

Determine who will be the evaluator.

Arrange for the evaluator to participate.

Discuss the purpose, targeted result, and process of the evaluation with the evaluator.

6. Show icons to the evaluator(s) (one at a time on the computer screen or printed on separate pieces of paper) without labels.

7. Have them describe or write down what each icon symbolizes and the types of information they think is accessible if the icon is selected.

8. Ask her/him to answer any prepared evaluation/interview questions.

9. Take notes from their responses.

10. Thank the evaluator for participation.

11. Arrange for additional evaluations to verify recorded responses.

Introducción al Diseño Instruccional

Outcome Data describing users’ interpretation of icons, figures, navigation buttons, or other graphic elements. This provides useful information for improving the design of all graphic elements, as well as insights on how to group and organize information.

Card Distribution to Icons The purpose of the Card Distribution to Icons usability test is to determine if the changes made to the icons based on the data collected from the Icon Intuitiveness tests were sufficient. The primary question to be answered here is "can users associate the correct meanings and related information with the icons?" The audience, procedure, and outcome involved in this testing process are described: Audience A sample of 3-5 users who are representative of the target population along three main dimensions of experience (see Audience description provided for Card Sorting above). Content experts can also provide effective feedback.

Procedure 1.

2.

3.

4.

5.

6.

Define the purpose and procedure for the evaluation.

Develop necessary instruments.

Determine who will be the evaluator.

Arrange for the evaluator to participate.

Mock up a table as a large-scale version of the eventual web page design.

Print out large magnifications of the icons on a color printer and place them on the desk in a rough sketch of the layout designed for the web page.

7. Discuss the purpose, targeted result, and process of the evaluation with the evaluator.

U8. El Modelo ASSURE para el Diseño Instruccional: Evaluación y Revisión.

8. Ask evaluator(s) to review each icon one at a time without labels.

9. Have them describe or write down on index cards what each icon symbolizes and the types of information they think are accessible if the icon is selected.

10. Ask her/him to answer any prepared evaluation/interview questions.

11. Take notes from their responses.

12. Thank the evaluator for participation.

13. Arrange for additional evaluations to verify recorded responses.

14. Test the icons both with and without labels.

Outcome Data describing users’ interpretation of icons, figures, navigation buttons, or other graphic elements. This provides useful information for improving the design of all graphic elements, as well as insights on how to group and organize information.

Introducción al Diseño Instruccional

Thinking Aloud Page Walk Through The purpose of Think Aloud Page Walk Through test is to reduce usability problems by focusing on the intended meanings and aesthetics of icons, buttons, and graphics. It may be used in combination with, or separately from Card Sorting, icon Intuitiveness, and Card Distribution to Icon tests. Think aloud protocols may also be used to test the clarity and effectiveness of directions, learning tasks, and instructional sequences. In short, targeted users are think aloud (talk through their thoughts and thought processes) as they are presented with icons, directions, tasks, instructional sequences, or other program elements. The audience, procedure, and outcome involved in this testing process are described: Audience A sample of 3-5 users who are representative of the target population along three main dimensions of experience (see Audience description provided for Card Sorting). Content experts can also provide effective feedback.

Procedure 1. Define the purpose and procedure for the evaluation.

2. Develop necessary instruments.

3. Determine who will be the evaluator and range for the evaluator to participate.

4. Discuss the purpose, targeted result, and process of the evaluation with the evaluator.

5. Print out magnified color screen dumps of your interface. Use a paper version instead of a screen version to avoid the problem of users clicking on buttons.

6. Ask the test users to point to each button and think out aloud what information they think will be accessed through that button.

7. You may also present users with directions, learning tasks, and a sequence of instructional events, along with scenarios, and ask them to think aloud as they address each element.

U8. El Modelo ASSURE para el Diseño Instruccional: Evaluación y Revisión.

8. At the end of the page walk-through, ask the users to comment on the aesthetics of the icons, buttons, and/or other graphic elements and to list those they liked and disliked the most.

9. Ask her/him to answer any prepared evaluation/interview questions.

10. Take notes from their responses.

11. Thank the evaluator for participation.

12. Arrange for additional evaluations to verify recorded responses.

Outcome Data describing users thoughts and thought processes as they are presented with various program elements (e.g., icons, navigation buttons, directions, tasks, instructional sequences). This information may be particularly useful for revising elements to be better aligned with course developers’ intentions.

Introducción al Diseño Instruccional

Heuristic Evaluation Heuristic evaluations provide a relatively quick, cheap, and easy assessment of user interface design. Heuristic evaluation is one of the most popular of usability testing techniques. The goal is to find usability problems, such as inconsistent titles and labels, unintelligible jargon, and confusing layout, so that they can be attended to as part of the iterative design process. Heuristic evaluation involves having a small group of usability experts examine the interface and judge its compliance with recognized usability standards (or "heuristics"). The following is an example of usability standards, along with a rating scale.

Sample Usability Standards • • • • • • • •

Speaks users' language.

Is consistent (e.g., layout, formatting, typefaces, labeling, titles).

Minimizes users' memory load.

Builds flexible and efficient systems.

Designs aesthetic and minimalist systems.

Uses chunking.

Provides progressive levels of detail.

Gives navigational feedback.

Sample Usability Rating Scale 0 = I don't agree that this is a usability problem at all.

1 = Cosmetic problem only: need not be fixed unless extra time is available on project.

2 = Minor usability problem: fixing this should be given low priority.

3 = Major usability problem: important to fix, so should be given high priority.

4 = Usability catastrophe: imperative to fix this before product can be released.

U8. El Modelo ASSURE para el Diseño Instruccional: Evaluación y Revisión.

Audience Three to five recognized usability experts.

Procedure 1.

2.

3.

4.

5.

Define the purpose and procedure for the evaluation.

Develop necessary instruments.

Determine who will be the evaluator.

Arrange for the evaluator to participate.

Discuss the purpose, targeted result, and process of the evaluation with the evaluator.

6. Ask each evaluator to inspect interface alone.

7. Ask evaluator to go through the interface several times, inspecting various elements, and compare them with a list of recognized usability principles.

8. Ask evaluator to describe problems as specifically as possible.

9. Ask her/him to answer any prepared survey/interview questions.

10. Take notes from their responses.

11. Thank the evaluator for participation.

12. Arrange for additional evaluations to verify recorded responses.

Introducción al Diseño Instruccional

Outcome Data identifying and indicating the severity of usability problems if any. This information provides vital input on essential and preferred revisions.

Scenario-Based User Testing The purpose of user testing is to find remaining usability problems that were not picked up in the heuristic evaluation after a redesign of the interface. This method is irreplaceable since it provides direct information about how people use computers and what their exact problems are with the concrete interface being tested. Usually, heuristic evaluations are performed first to clean up the interface and remove as many "obvious" usability problems as possible. An effective strategy is to alternate between heuristic evaluation and user testing: • •

The heuristic evaluation pass can abolish a number of usability difficulties without the need to "waste users," who at times can be hard to find and schedule in large numbers.

These two categories of usability assessment procedures have been shown to find fairly distinct sets of usability problems; therefore, they complement each other rather than lead to repeated findings.

Scenario-based user testing involves presenting representative end-users with scenarios or specific tasks designed to cover the major functionality of the software program and to simulate expected real-life usage patterns. A scenario is a summarized portrait of:

U8. El Modelo ASSURE para el Diseño Instruccional: Evaluación y Revisión.

• • • • •

An individual user.

Using a specific set of computer resources (e.g., equipment, facilities).

To achieve a specific outcome .

Under specified circumstances.

Over a certain time interval (time dimension of what happens when).

A sample scenario used to test the usability of this course sets the stage for a novice computer user with no previous experience with WebCT and the subject matter.

Introducción al Diseño Instruccional

Scenario I You are a new graduate student taking your first Instructional Technology course on-line. Although you have some experience surfing the web, you consider yourself a novice computer user. You’ve heard good things about the ITDE program at NSU and have wanted to start your degree earlier but could not find the time until now. You think you may be able to complete your degree if most of the courses can be taken on-line so you decide to register for a course to see what it is like. You receive electronic mail a couple of days after registering and decide to start preparing for the course as recommended by the letter. You are at home and decide that you have about an hour tonight to begin exploring the Website.

According to Nielsen's (1996), before any testing, you must develop plans by answering the following question.

• • • • • • • • • • • • •

The goal of the test: what do you want to achieve?

Where and when will the test take place?

How long is each test session expected to take?

What computer support will be needed for the test?

What software needs to be ready for the test?

What should the state of the system be at the start of the test?

Who will serve as experimenters for the test?

Who are the test users going to be, and how are you going to get hold of them?

How many test users are needed?

What guidelines will be used to decide when the users are done?

What test tasks will the user be asked to perform?

What user aids will be made available to the test users?

What data is going to be collected and how will it be analyzed once it has been collected?

Audience A sample of 3-5 users who are representative of the target population along three main dimensions of experience (see Audience description provided for Card Sorting).

U8. El Modelo ASSURE para el Diseño Instruccional: Evaluación y Revisión.

Procedure Define the purpose and procedure for the evaluation.

Develop prototypes—vertical and/or horizontal.

Construct scenarios to test specific aspects of your program.

Develop necessary instruments.

Determine who will be the evaluator.

Arrange for the evaluator to participate.

Discuss the purpose, targeted result, and process of the evaluation with the evaluator, including the amount of time required to carry out the test(s).

8. Conduct test in relaxed atmosphere. Sit with the users as they navigate through the interface. Record observations and user comments.

9. Make sure that the first task is simple, so that users are ensured of an early successful experience. The entire examination should be conducted without disruptions.

10. Collect test and/or other results.

11. Ask evaluator(s) to answer any prepared survey/interview questions.

12. Take notes from their responses.

13. Thank the evaluator for participation.

14. Arrange for additional evaluations to verify recorded responses.

Introducción al Diseño Instruccional

1.

2.

3.

4.

5.

6.

7.

Outcome Data related to users ability to complete tasks under given scenarios. Results are then put in table form using such measures as whether the participants correctly accomplished the tasks, the time taken for each task, and the number of pages/screens accessed for each task. This information provides valuable input for necessary and preferred revisions.

Integrating Usability Tests It is believed that the usability engineering tests described in this chapter may be integrated with Dick, Carey, and Carey’s (2001), four phases to provide an effective and comprehensive formative evaluation strategy for technology based training and instruction. While Dick, Carey, and Carey provide methods for effectively evaluating the instructional and learning aspects of training and educational programs, the usability tests provide useful tools and techniques for evaluating user interfaces, menu structures and navigation mechanisms.

To integrate the two methods, I recommend that you select the usability tests that are appropriate for your given situation and schedule them into a general plan that uses Dick, Carey, and Carey’s four phases as a basic framework (see sample below). Furthermore, it is believed that the Scenario based user testing technique compliments can be readily adapted as a tool for focusing one-to-one, small group and field trial evaluations. The other usability testing techniques described in this chapter (i.e., card sorting, icon intuitiveness test, card distribution, thinking aloud, and heuristic evaluation) may be applied as an integral part of the four phase framework but use different instruments and procedures than described by Dick, Carey and Carey. Exact questions, procedures and materials can then be addressed as you develop your specific formative evaluation plans.

U8. El Modelo ASSURE para el Diseño Instruccional: Evaluación y Revisión.

Evaluation Expert Reviews One-toOne Evaluations

Introducción al Diseño Instruccional

Small Group Evaluations

Evaluator Human factors expert

Date

Target

Technique

3/10/99 Interface Heuristic

Questionnaire

5/20/99 Icons

Icon Icon test intuitiveness chart

Student (advanced computer skills)

6/20/99 Units 13

Scenario I

5 Students (range of skills)

7/7/99

5 target students

Observations forms Interview questions

Online Syllabus

Scenario II

10 Students 8/1/99 (volunteers)

Study Guide Interface

Survey Interview Questions

User Interface Field Test

Instruments

Scenario III

Survey Interview Questions

Units 112 Figure 10.4. Sample of General Formative Evaluation Plan Summary Chapter 10 reviewed key formative evaluation concepts posited by Dick, Carey, and Carey (2001) and extended their discussion by presenting a model depicting the relationships among key concepts, identifying limitations associated with the Dick, Carey, and Carey model and examining several usability engineering techniques.

Little time was spent reviewing method for reporting formative evaluation results and revising materials because they are covered very well in Chapter 11 of the course textbook. You are responsible for conducting, as well as reporting the results of your formative evaluations so be sure to read the related sections in the course textbook. However, you are not responsible for revising your materials. Table 10.1 summarizes a few things to remember when conducting tests of prototypes or other materials during the formative evaluation process.

U8. El Modelo ASSURE para el Diseño Instruccional: Evaluación y Revisión.

Table 10.1 Summary of formative evaluation procedures.

Before the Test • •

•

•

Introducción al Diseño Instruccional

•

•

•

Have everything ready before the user shows up.

Emphasize that it is the system that is being tested not the user.

Acknowledge that the software is new and untested, and may have problems.

Let users know that they can stop at any time.

Explain any recording, keystroke logging, or other monitoring that is used.

Tell the user that the test results will be kept completely confidential.

Make sure that you have answered all the user's questions before proceeding.

During the Test

After the Test

? Try to give the user an early successful experience. ? Hand out the test tasks one at a time. ? Users are human and are often nervous when tested. Keep a relaxed atmosphere in the test room, serve refreshments, and have breaks. ? Avoid disruptions, close the door and post a sign on it; disable telephones. ? Never indicate in any way that the user is making mistakes or is too slow. ? Minimize the number of observers at the test. ? If necessary, have the experimenter stop the test if it becomes too unpleasant.

? End by stating that the user has helped you find areas of improvement. ? Never report results in such a way that individual users can be identified. ? Only show videotapes outside the group with the use permission. ? The test results should be kept confidential, and reports from the test should be written in such a way that individual test users can not be identified.

Online Discussion Questions Having read the materials related to this topic in Dick, Carey, and Carey (2001), as well as the above text, please discuss your answers to the following questions with your fellow students in the WebCT Bulletin Board: 1. What should you do if you do not have sufficient resources to conduct a comprehensive formative evaluation of your instructional materials (e.g., expert reviews, one-to-one, small group and field tests)?

2. What types of tables should you use to analyze and report formative evaluation data? How do they differ?

U8. El Modelo ASSURE para el Diseño Instruccional: Evaluación y Revisión.

References Bordonaro, T., McAlpine, L., & Weston, C. (1995). A Model for Understanding Formative Evaluation in Instructional Design, Educational Technology Research and Development, 43(3) 29-28.

Nielsen, J. (1993). Usability engineering. Orlando, FL: Academic Press.

Smith, P. L., & Ragan, T. J. (1993). Instructional Design. New York, NY: Macmillan Publishing Company.

Related Online Resources Neilsen’s Top Ten Mistakes in Web Design [On-Line] Available: http://www.useit.com/alertbox/9605.htm

Introducción al Diseño Instruccional

Nielsen, J.(1997). Severity ratings for usability problems [On-line] Available: http://www.useit.com/papers/heuristic/severityrating.html Usability Testing as told by a Sun Computer Engineer [On-line] Available: http://www.useit.com/jakob/index.html Why Usability Testing? [on-line] Available: http://delphi.colorado.edu/~irm/about_irm/schedule/WWW_conf_jan_96/ljdpres/

Para citar este texto.

Hirumi, A. (2002). Formative Evaluation. Miami, F.L.: Nova Southeastern University