Program Evaluation Methods and Case Studies, 8E Emil J. Posavac Test BankProgram Evaluation Methods

Program Evaluation Methods and Case Studies, 8E BY Emil J. Posavac

Email: richard@qwconsultancy.com

TABLE OF CONTENTS Chapter 1 Program Evaluation : An Overview

1

Chapter 2 Planning an Evaluation

5

Chapter 3 Developing and Using a Theory of the Program

9

Chapter 4 Developing Measures of Implementation and Outcomes

14

Chapter 5 Ethics in Program Evaluation

19

Chapter 6 The Assessment of Need

25

Chapter 7 Monitoring the Implementation and the Operation of Programs

30

Chapter 8 Qualitative Evaluation Methods

34

Chapter 9 Outcome Evaluations with One Group

39

Chapter 10 Quasi-Experimental Approaches to Outcome Evaluation

43

Chapter 11 Using Experiments to Evaluate Programs

48

Chapter 12 Analyses of Costs and Outcomes

53

Chapter 13 Evaluation Reports: Interpreting and Communicating Findings

57

Chapter 14 How to Encourage Utilization

61

CHAPTER 1. PROGRAM EVALUATION: AN OVERVIEW

1. One of the major failings in the development and implementation of social service programs is * a. beginning the service without first demonstrating the effectiveness of the service. b. spending too much effort identifying the needs of the population. c. saddling the service program with too many requirements to demonstrate its effectiveness. d. isolating the service program from the political structure of the area needing the service.

2. The effectiveness of social services has often not been evaluated because a. the values of all such services are already well documented. * b. the desired outcomes are indeed hard to measure. c. it is usually sufficient to merely show interest in helping. d. government regulations forbid evaluations of human services.

3. Evaluators have often a. ignored existing measures of the outcomes of services. b. failed to contact the managers of programs. c. used large samples with low Type II errors. * d. overlooked possible negative side effects of programs.

4. Program evaluation gets confused with basic research when evaluators a. focus on the needs of the program sponsor. b. become advocates for the program participants. * c. ignore practical issues in order to examine theoretical questions. d. use variables suggested by the staff of service programs.

5. An evaluation of need refers to evaluations that a. are similar to performance appraisals. b. isolate the costs of providing various services. * c. are particularly useful in planning a new program. d. violate fundamental scientific standards.

6. Evaluations of process often focus on the degree to which a. a program is successful. b. a specific service is needed in a community. c. the costs required to run a program can be justified. * d. the program is implemented as planned.

..

1

7. Outcome evaluations provide the data to show whether * a. a service has achieved appropriate results. b. a target population will use a service. c. the evaluator has made every effort to work with the program director in an effective manner. d. resources were spent on programs that are most needed.

8. When an evaluation relates the cost of a program to its outcomes, this form of evaluation can be thought of as a. an evaluation of performance. * b. an evaluation of efficiency. c. a variant of basic research. d. overly scientific.

9. Formative evaluation is to summative evaluation as a. process is to outcome. * b. constructive criticism is to final grades. c. apples are to apple pies. d. cancellation is to initiation.

10. The purpose of a formative evaluation is to provide information to program sponsors a. making decisions about maintaining or terminating the program. b. to minimize the effects of arbitrary federal requirements for program evaluations. * c. making program improvements. d. for public relations needs.

11. It is sometimes hard to learn who is conducting program evaluations in an organization because a. few organizations really need to have their services evaluated. b. formal evaluations are conducted by most managers of service programs. * c. the job titles of evaluators are not standardized. d. when evaluations are completed, they usually look like basic research.

12. If the management of an organization plans to maintain or eliminate a service on the basis of its documented degree of effectiveness, then a * a. summative evaluation is needed. b. needs analysis should be conducted. c. formative evaluation is needed. d. person familiar with the program should conduct an evaluation.

13. The strongest advantage that an external evaluator has over an internal evaluator is a. the trust he/she has built up over years. b. more well-developed analytic skills. * c. the potential to be more objective. d. the influence to see that the findings are used by the program staff.

..

2

14. The four major forms of program evaluation are a. internal, external, conclusion, and construct. b. performance, institutional, accountability, and program review. * c. need, process, outcome, and efficiency. d. community, governmental, professional, and special interest.

15. A process evaluation would probably NOT include which one of the following questions? a. Do the program’s actual participants represent the target population? b. How much staff-client contact really occurs? c. Do actual staff activities match the program plan? * d. Is the program cost-effective?

16. Internal program evaluators usually have which following advantage over consultant evaluators? a. better statistical and research competence b. better support staff c. a more objective approach to the program * d. more complete knowledge of the organization sponsoring the program 17. Program evaluation is frequently confused with a. billing and accounting procedures. * b. basic research and individual assessment. c. therapist’s progress notes. d. crisis management methodology.

18. The ultimate purposes of program evaluation include a. the collection of data on the outcome of services. b. an assessment of how program staff people spend their time so that ineffective staff members can be identified. c. making decisions about the proper groups a program should serve. * d. the improvement of implemented programs and the wise selection among possible programs.

19. Monitoring is a form of evaluation which focuses on a. providing definitive proof that a program is worth the investment. b. being sure that data are available to make refinements in a program. * c. providing frequent feedback to be sure a program stays on track. d. financial resources so that funds are not misspent.

20. Implementation of programs should be examined because a. fraud is rampant in social programs. b. irresponsible service providers often destroy programs. c. internal evaluators usually ignore this issue. * d. programs are less effective when not offered at full strength.

..

3

Essay Questions 21. In parallel columns, list the advantages and disadvantages of internal and external evaluators.

22. Outline the essential questions that evaluators seek to answer.

23. To what degree is it true that people seek to evaluate their own efforts?

24. Why is it more complicated to evaluate organized programs than the effort of an individual?

25. Make a list of college (or community) services that you use which you believe could be improved if evaluated carefully. Pick one and explain why it is in particular need of evaluation.

..

4

CHAPTER 2. PLANNING AN EVALUATION

1. The first step in planning an evaluation is to a. suggest outcome measurement instruments to the staff. * b. identify the stakeholders of the evaluation. c. try to figure out how the evaluation could be done as an experiment. d. caution the program director that the evaluation will be tough.

2. Evaluators must be very careful to be sure that they understand the actual purpose of a request for an evaluation because * a. many stakeholders talk about evaluation in terms that sound like summative evaluation when they really want to improve the program. b. most program staff members are too willing to disrupt the service program to accommodate the evaluation. c. the demand for evaluation is so widespread that program staff are eager to participate. d. stakeholders are eager to apply all recommendations from an evaluator.

3. Program evaluation, like other types of applied social science research, differs from basic research in that a. validity is irrelevant in applied research although reliability is critical to both. b. reliable measurement is irrelevant in applied research. * c. there are definite time limits for completing applied research projects. d. program participants cannot refuse to provide data.

4. A skill that is more important in program evaluation compared to basic social science research is * a. estimating accurately how much time each phase of the project will require. b. being able to plan valid research designs. c. selecting measures of behavior that are reliable. d. molding the project to answer questions relevant to social theories.

5. “Evaluability assessment” refers to the a. ability of the program managers to fund an evaluator. b. knowing how to carry out statistical analyses. c. reliability of the measures of program outcome that are chosen. * d. likelihood that a valid evaluation can be completed.

6. A major failing of many evaluations is that the evaluator never learned a. what the outcome measures were. * b. why the program activities were expected to lead to the desired outcomes. c. who was to receive the report and use the findings. d. how the analysis was done or what the statistical findings were.

..

5

7. Understanding the conceptual foundation of a program provides an evaluator with * a. hints about how the program services are expected to lead to the outcomes that the staff and director hope to achieve. b. some indication of how much impact the evaluation will actually have on the organization sponsoring the program. c. some indication that the evaluation will really have an effect on the people who need the services of the program being evaluated. d. the expected conflicts between the evaluator and the management that may well impact the quality of the project.

8. An evaluation of high quality usually __________ than one of lower quality. a. is based on data from a greater number of program participants b. was designed with input from fewer stakeholder groups * c. is based on data from a more representative sample of program participants d. employs less costly surveys and questionnaires 9. The data collection is a. best turned over to the program staff since they are on the scene and can easily handle the activity. b. to be handled by someone without direct knowledge of the setting of the program since it is usually a rather mechanical process. * c. likely to be corrupted even by well-meaning program staff members if they are permitted to control data collection. d. fairly simple since confidentiality is not an issue in applied social research projects.

10. A written proposal outlining the steps to be followed in carrying out a planned program evaluation * a. should always be prepared. b. can be omitted if the evaluation is conducted internally. c. should always bind the evaluator to exactly the procedures described. d. cannot realistically be prepared since so many unplanned events impinge upon program evaluations.

11. Resistance to an evaluation is likely to be greatest when the evaluation is a. a formative evaluation. b. a cost-effectiveness evaluation. c. an outcome evaluation. * d. a summative evaluation.

12. Small improvements in the achievements of program participants * a. can be very important if the program affects many people. b. are seldom of interest to serious program managers. c. cannot be justified in a politically-charged environment. d are often all that is expected by program staff members.

..

6

13. Many members of the staff of human service programs resist program evaluations in their facilities because a. evaluation methods cannot be sensitive to their concerns. * b. it may appear that they are losing control of their program. c. their jobs on are on the line. d. evaluators have a way of taking over.

14. One way for evaluators to mollify the critics of a program evaluation project is a. to demonstrate clear analytical skills. b. to mold the project design to avoid any points of concern. * c. to listen to the worries of the critics and relieve all concerns that are founded on misperceptions of program evaluation. d. for the evaluator to ally him/herself with the facility management, who will see to it that the correct procedures are followed.

15. Goal-free evaluation refers to evaluations that are conducted a. on programs with no goals. b. when program managers cannot describe their goals. c. when the program is designed to show that the government is “doing something” but no one expects improved outcomes. * d. when evaluators want to examine the program without their own expectations being affected by knowledge of the program’s goals.

16. One way to reduce fear and resistance to evaluation activities is to reassure staff that a. the larger community will be better off knowing when a program is ineffective. b. program participants (i.e. the clients) have a right to know how effective the program is. * c. documenting success would increase the organization’s commitment to the program. d. less that half of outcome evaluations result in the curtailment of a program.

17. The social science model for program evaluation served to a. get rid of a lot of ineffective services. b. demonstrate that a non-significant statistical test showed that the program was ineffective. c. introduce more creativity into program evaluation methods. * d. introduce additional rigor into program evaluation practices.

18. Using expert opinion as a form of program evaluation is especially useful when * a. the program is complex and there are few clear objective criteria of effectiveness. b. there are readily observable outcomes from a program. c. it is essential that the causal connection between the program and the outcome be definitely understood. d. new medications are being evaluated.

..

7

19. Finding discrepancies between client needs and services offered or between projected outcomes and achieved outcomes a. shows why a service ought not be supported. b. identifies ways to improve a program. * c. reveals the issues that must be faced in making program improvements. d. shows that evaluators have done their work.

20. Inappropriate reasons to evaluate a program include a. documenting program activities to satisfy a funding agency. b. demonstrating effectiveness prior to seeking a new source of funding. c. seeking more efficient ways to provide social and educational services. * d. commissioning an evaluation to deflect criticism and postponing a decision.

21. Considerable assistance can be obtained in planning a program evaluation from a. published evaluations. b. the Internet. c. informal conversations with other evaluators. * d. all of the above 22. An evaluation to focus on participants who have achieved the program goals a. is usually a good basis for a summative evaluation. b. is inherently dishonest. c. lies at the heart of the social science model. * d. is called the Success Case Method.

Essay Questions 23. Describe how one would discuss random assignment to treatment or non-treatment conditions with a group of people who applied for a new oversubscribed job-training program.

24. How would the difference between formative and summative evaluations affect the negotiations with program staff and other stakeholders in the planning of an evaluation?

25. Discuss the advantages of involving the relevant stakeholders in the planning of an evaluation.

26. If during the planning of an evaluation, an evaluator discovers that critical stakeholder groups differ in their views of central objectives of a program, what are the best courses of action for the evaluator?

27. What can evaluators do to encourage stakeholders to take additional responsibility for programs in which they are involved?

..

8

CHAPTER 3. DEVELOPING AND USING A THEORY OF THE PROGRAM 1. The selection of indexes of program effectiveness is usually difficult because a. program staff members refuse to specify what changes are to occur in their clients. * b. different stakeholders have different views about what the most important criteria of program success are. c. program effectiveness cannot be measured. d. it is unethical to measure program effectiveness when participants need the program’s services.

2. When there is disagreement about the choice of the criteria of program effectiveness, the best procedure for program evaluators would be to a. take over the choice of criteria. b. use the criteria preferred by the stakeholder who is funding the evaluation. c. use the criteria preferred by the program staff. * d. arrange discussions among stakeholders to find common concerns and to gain agreement to measure a variety of criteria.

3. The program goals are usually described to evaluators because * a. goals help evaluators to determine what variables to measure. b. evaluators confine their attention to the goals that program sponsors can describe. c. it is impossible to carry out any form of evaluation when an evaluator does have not an explicit statement of goals. d. program goals will show evaluators which negative side-effects should be studied.

4. The criteria used in a program evaluation may include an assessment of how completely the program has been implemented a. whenever evaluators doubt the honesty of the program staff. b. whenever severe side-effects are likely to lead the staff to reduce the strength of the program. * c. because practical difficulties can lead well-intentioned people to depart from program plans. d. because stakeholder groups usually view each other with hostility and suspicion.

5. In outcome evaluations, the achievement of intermediate goals is often examined because a. it shows that evaluators understand the program. b. staff members insist on measuring such criteria. c. success in achieving intermediate outcomes can often make up for failure to achieve outcome goals in summative evaluations. * d. this aids evaluators to learn whether program activities do indeed contribute to changes in the participants even when the achievement of the ultimate outcomes is minimal.

..

9

6. Evaluators can increase the practical value of their evaluations by a. gathering information on criteria that are expected to be unaffected by the program. b. keeping a narrow, precise focus for the evaluations. c. avoiding all contact with the program staff. * d. including observations of criteria which must apply to all programs, such as avoiding discriminatory behavior and treating clients with respect.

7. Social values a. must be ignored by all evaluators. b. do not enter into program planning. c. are not held in common by members of a pluralistic society so they do not enter into the choice of possible services that could be funded publicly. * d. are central in the choice and implementation of programs; however, empirical data cannot dictate which values are to be adopted.

8. An examination of the needs of program clients would be important in an evaluation of outcome because * a. a mismatch between the needs of clients and the services provided could be responsible for disappointing outcomes. b. a target population may have additional needs that the program was not designed to meet. c. stakeholders disagree on the needs that are most pressing. d. the staff members may be better trained to meet needs different from those the program was funded to deal with.

9. Evaluators who learn that a treatment group shows a higher average level on measures of desired outcomes than a comparison group at a p < .05 level have a. completed the work necessary to show that the program is worthwhile and should be maintained. b. made a serious error since statistical significance tests are never used in program evaluations. * c. not completed their work since field settings require an understanding of the extent of improvement. d. focused on a criterion that is value-oriented and, consequently, inappropriate for program evaluation.

10. In setting a specific mean level expected for a criterion of success among a treatment group in an outcome evaluation, the evaluation team a. depends solely on the program staff since the staff has the best understanding of the program. * b. works with the staff, but recognizes that program staff members often overestimate the degree clients will change even in an excellent program. c. should ignore the program staff members since their level of self-interest will so greatly affect what they suggest that their input is worse than useless. d. should use objective information available in the research literature. If such information is not available, the team should decline to conduct an evaluation. ..

10

11. Evaluators discuss the conceptual, or theoretical, basis for the program because a. theory is more interesting and complex than the service program being evaluated. b. most program staff members and agency administrators have thoroughly considered the conceptual basis of their service and are eager to have it used in an evaluation. c. they have been trained to do so, but such discussions add little to the quality of the program. * d. program theory helps evaluators to learn what might be important to observe and measure.

12. Theories that are relevant to the development and operation of a service program a. are the concern of program developers, not evaluators. b. add little to carrying out an effective program evaluation. * c. can assist evaluators to carry out more useful evaluations. d. are of no practical use, but can help get an evaluation published in a scientific journal.

13. The degree to which a program is accepted and valued by potential users * a. is a proper concern to program evaluators and stakeholders. b. is irrelevant since people seldom like what is good for them. c. should form the primary focus of an evaluation to the exclusion of other criteria especially when evaluation funds are limited. d. is the best guide to whether a program would be effective.

14. Concern over whether the program funds are spent on the activities described in the approved program plan * a. is part of evaluations of program implementation. b. indicates obsessive-compulsive thinking since the target population benefits regardless of what services are provided. c. should concern accountants; program evaluators focus on softer criteria. d. would not be of interest to governmental bodies responsible for supporting the program.

15. The objectivity of program evaluators is most likely to be jeopardized by a. meeting program staff in program settings. b. seeking information about similar programs. * c. depending on only one stakeholder group for information on the program. d. getting input from representatives of all major stakeholder groups.

16. Evaluation criteria should be chosen a. to ensure the evaluation has the most scientific credibility regardless of how long it would take to complete. * b. to reflect the practical limitations of budget and time available for an evaluation. c. by external evaluators with excellent reputations. d. by those who are responsible for funding the evaluation.

..

11

17. When agreement has been reached regarding the criteria that are to be used to indicate program success, * a. it is helpful that some means be used to document that agreement has been reached. b. evaluators know that work will progress smoothly the rest of the way. c. stakeholders seldom question the choice of criteria at a later time, even if the evaluation results are unfavorable. d. it is never appropriate to add or subtract any criteria during the evaluation.

18. The selection of criteria and standards by which to judge the effectiveness of a program a. is not all that important since good programs will look good and ineffective ones will look bad regardless of the criteria. b. can be put off until after the data are being analyzed as long as many different criteria have been measured during data gathering. c. is easy to carry out since all except the most disorganized program already have their criteria and standards clearly described in program materials. * d. can determine whether the evaluation turns out to be favorable or an unfavorable.

19. An evaluation of a reading program that focuses on the reading achievement of fourth graders (in units of grade level equivalents) is using an a. implementation goal. b. intermediate goal. * c. outcome goal. d. efficiency goal.

20. Outcome measures with low reliability (for example, .40), a. can never be used by program evaluators. * b. may be used as indicators of program success, but not as measures of individual success. c. are probably not especially central to the goals of the program. d. may be used as criteria, but only for the evaluators’ purposes; they should not be mentioned in reports.

..

12

Essay Questions 21. Illustrate how an ill-considered choice of evaluation criteria can distort program activities to the extent that the value of a program is reduced. You might think of a school lunch program, or a youth sports program.

22. Illustrate how an evaluator might mistakenly use criteria that can be measured with high reliability but that does not represent the program’s purpose very well.

23. Why are statistically significant differences between treatment and non-treatment groups not sufficient to show that a program is worth offering?

24. Illustrate how a program could produce favorable outcomes yet be seen as unfair to some stakeholders.

25. Why are negative side effects often ignored in program evaluations? Imagine the side effects of introducing a modern medical clinic into a rural village in an undeveloped nation.

..

13

CHAPTER 4. DEVELOPING MEASURES OF IMPLEMENTATION AND OUTCOMES

1. Desirable aspects of gathering data for program evaluation from records include the a. freedom to choose the most relevant outcome variables. b. uniformly high validity of archival data. c. thoroughness with which program records are routinely kept in many settings. * d. non-reactivity of the data selected from program records.

2. A major disadvantage of using program records in program evaluation is: * a. records were not designed for evaluation purposes. b. measurement processes will not affect the program participants. c. there can be no attrition of respondents who are willing to provide data. d. another evaluator can examine the records independently from the original evaluator.

3. Among the advantages of using program participants as a source of evaluation data is: * a. the information is relatively inexpensive. b. measurement is not reactive. c. their views about program quality are likely to be very valid. d. participants provide particularly reliable reactions.

4. When obtaining information from program participants, one should a. never use objective variables. b. depend on the program staff to choose the variables. c. question whether participants are truly stakeholders. * d. ask only about issues familiar to the participants.

5. Program participants are likely to be defensive about answering an evaluator’s questions whenever * a. they feel that they have something to lose by providing their comments. b. they feel that the program might be improved by their answers. c. the evaluator contacts them in person. d. they are particularly pleased with the program.

6. One of the major problems with using program participants as sources of evaluation data is their tendency to a. give primarily negative evaluations of service. b. be uninterested in program evaluation. * c. make indiscriminately favorable evaluations of services. d. be incapable of making any judgments about service providers.

..

14

7. Asking individuals familiar with the kinds of services provided to make direct observations of the program a. is seldom disruptive of program services. * b. could produce very unbiased data if the experts are involved in neither the program nor its competitors. c. is among the least expensive evaluation data available. d. should never be used because it could threaten the staff.

8. Failing to detect a true program outcome because a variable is affected by so many influences is a particular problem with * a. community-level indexes such as crime rate or Gross Domestic Product. b. program participant surveys of satisfaction with a program. c. data gathered from the program staff. d. questions addressed to relatives of the people being served.

9. The choice of the criteria of effectiveness is very sensitive in program evaluation because a. no one is ever satisfied with the criteria selected. b. program staff are always trying to select criteria that are too easy to achieve. * c. the choice of criteria can determine the outcome of the evaluation. d. most evaluations show that human services are highly effective.

10. Selecting one particular variable to be the criterion of program effectiveness will * a. probably corrupt it. b. complicate the data analysis. c. reflect the complexity of the program. d. be desired by most staff members.

11. Evaluations are more likely to be effective if the list of variables to be measured is a. lengthy and thorough. * b. focused on important variables. c. includes some personality characteristics. d. costly to obtain.

12. The criteria of effectiveness used in a program evaluation should a. be limited to the objectives of the program. b. be determined by the program staff. c. not focus on any negative outcomes of the services. * d. be related to the information needs of program stakeholders.

13. Negative side-effects of programs are important for evaluators to consider because * a. staff members frequently do not consider them. b. participants usually experience some strong negative side effects. c. some participants will be harmed by any program. d. program plans usually mention possible negative side effects.

..

15

14. The criteria selected for measuring program effectiveness should be sensitive to changes caused by the program. A good way to determine if a variable might be sensitive to change is to a. ask program participants for their opinions. b. depend on program staff for ideas. * c. see if the variable distinguishes between groups of people who differ in degree to which they achieved planned program outcomes. d. use well-constructed, standard personality or intelligence tests.

15. In evaluating the reliability of a measure to be used in program evaluation, evaluators * a. recognize that means of groups are more stable than the scores of individual assessments. b. realize that reliability is not very important in the area of program evaluation. c. must choose between having a valid measure or having a reliable measure; they can’t have both. d. know that surveys are probably more reliable than program effectiveness indexes based on program records.

16. The most widely used type of measurement procedure in program evaluation is most likely a. a community-level index. * b. the written survey completed by program participants. c. a rating of the program by a significant other. d. an observation made by an expert working with the evaluator.

17. The relative costs of using various methods of data collection in program evaluations a. cannot be considered because truth is more important than cost. b. are always a trivial factor in the whole study. c. seldom exceed the cost to duplicate surveys or forms. * d. should be weighed against the validity of the data sought.

18. Because behavioral change is usually the goal of most human service programs, behavioral observations a. should be the sole type of measure used in evaluations. * b. may produce very valid information. c. are relatively cheap to gather. d. might be valuable, but cannot be made ethically.

19. In selecting indexes or variables that are to reflect program success, it is useful to choose variables that are a. historical, instrumental, and well-known. * b. important, sensitive to change, and cost-effective. c. summative, formative, and sensitive. d. hidden, covert, and implicit.

..

16

20. Interviewing is usually done when * a. busy people have information about the program that few others have. b. there is not enough time to prepare a reliable survey. c. the costs of the evaluation must be kept to a bare minimum. d. many people have similar experiences with the program, making surveys redundant.

21. Including observations of actual behavior among the measures of program success a. makes the evaluation too expensive. b. creates incredible anxiety among the program staff members. * c. adds markedly to the credibility of the evaluation. d. is usually felt to be unnecessary or unethical.

22. If the choice is available, evaluators will choose to a. develop new measures rather than to use existing measures. b. use measures more similar to psychological tests than to classroom tests. c. measure stable traits rather than changeable states. * d. use previously developed measures rather than to develop measures of their own.

23. An important characteristic of self-report surveys is that a. they can be constructed in a short time. b. they can be used in any program setting. * c. considerable thought needs to be devoted to designing them. d. any criterion of program effectiveness can be measured in a survey.

24. What should you say if the following item were suggested for use in an evaluation of teaching: “My teacher was well-informed and well-meaning”? * a. The item mentions two aspects of a teacher. b. It is best to avoid the use of a personal pronoun. c. Teaching cannot be evaluated. d. This is a good item; let’s use it.

25. Using multiple measures of program achievement may not add to the validity of an evaluation if * a. the measures share the same biases. b. all measures are based on interviews. c. few measures have reliabilities exceeding .80. d. objective data sources such as expert observers or artifacts were not used.

26. When using official records gathered over a period of years, evaluators must be wary of a. the inter-rater reliability of records. b. program participants who refuse to cooperate with the evaluation. c. threat to insecure program managers. * d. changes in the definitions of the variables.

..

17

Essay Questions 27. Contrast the strengths and weaknesses of self-report surveys with observations of behavior.

28. Distinguish between the characteristics of tests used in classrooms versus tests to measure stable psychological characteristics.

29. Define reliability for hypothetical stakeholders who have not studied social service research methods or assessment principles.

30. Suggest some artifacts that might be used in measuring the success of a program to encourage a company’s employees to quit smoking.

..

18

CHAPTER 5. ETHICS IN PROGRAM EVALUATION

1. If an evaluator believes that a potential client has requested an evaluation that will not meet the agency’s real needs, the evaluator should a. simply do the best job she can. * b. attempt to redefine the form of evaluation to be done. c. tell the client what is to be done. d. seek to gain power by going to the client’s supervisor or board of directors.

2. Ethical issues in program evaluation seem more compelling than in basic research because a. evaluators are less skilled than basic researchers. b. program evaluations are often replicated, thus assuring that errors will be detected. * c. program evaluations are seldom replicated and the findings may be applied even if in error. d. service program staff members are notoriously unethical.

3. Program evaluators stop thinking about ethical issues a. once a project is planned in an ethical manner. b. after it is clear that the data have been gathered ethically. c. after a report of an ethically conducted evaluation has been written. * d. at their peril because ethical standards can be violated at any stage or even after the report is finished.

4. “Informed consent” refers to a. the contract used to employ an evaluator. * b. obtaining the agreement from people to supply data for an evaluation after they have been adequately informed about what is expected of them. c. the consent that program staff must give to the evaluator who has been requested to evaluate their program; without such consent, the evaluation cannot be done. d. participation of people in the service program.

5. Evaluators are especially concerned about the possibility of invalid conclusions because * a. evaluations are more likely to have an impact on people’s lives than basic research reports do. b. what they conclude cannot be applied. c. they work in applied settings in which mistakes cannot be tolerated. d. evaluation is a young field that has yet to prove its worth.

..

19

6. If after beginning an evaluation the evaluator comes to feel that the evaluation plan cannot be carried out in a way that will yield a valid evaluation, the best alternative is to a. do the best one can do, recognizing that no evaluation is perfect. * b. seek to renegotiate the evaluation plan so that it describes a plan that can be carried out well. c. provide a favorable evaluation to the stakeholder paying for the project. d. refuse to complete the project on the grounds that the findings will not be valid.

7. In conducting an evaluation of an innovative program, the most important responsibility of the evaluator is to a. conduct a valid evaluation. b. consider the generalizability of any positive findings. c. be sure that the most needy individuals get into the program. * d. be sure that the evaluation does not harm the participants.

8. Evaluators prefer to have informed consent from all participants in an evaluation. When a full description of the program groups (for example, innovative program and control groups) threatens to undermine the internal validity of the evaluation, a good way to decide if full disclosure is necessary is to a. ask the major stakeholders what is the best procedure. b. argue that a valid evaluation is more important than program participants having full information. * c. consider the cost to the participants of each group if they did not have such information. d. insist that full disclosure is always required.

9. Prior agreement from people who are to be sources of data in program evaluations can be called “informed consent” only when a. an Informed Consent form has been signed. b. the evaluator is present when the form is signed. * c. agreement has been given only after sufficient information has been provided to permit understanding the requirements of the evaluation. d. the sponsor of the program fully understands the evaluation and what is expected of program participants.

10. Problems that develop when an evaluator becomes an advocate for the program being evaluated include: * a. the loss of objectivity needed to conduct an evaluation. b. the temptation to design evaluations that are too complex. c. being forced by the staff to produce only favorable evaluations. d. the development of hostility between evaluator and program staff.

..

20

11. The term stakeholder refers to a. a tool used in barbecue restaurants. b. the evaluators of a program. * c. anyone affected by the program being evaluated. d. all people who are not affected by the program evaluation.

12. The evaluator needs to be concerned about serving the information needs of stakeholders because a. these groups determine whether an evaluator will be paid. * b. stakeholders who feel that they have been ignored will seldom use the evaluation. c. they must be mollified even if their input is not essential. d. the evaluator will learn little from them.

13. A problem with using measurement tools that do not validly measure the outcome of a program is that a. the evaluation is not adequately scientific. b. the stakeholders are less likely to understand the evaluation report. c. a biased evaluation favoring the program is likely. * d. the actual outcome may be obscured.

14. Using a general school achievement test when a measurement of a specific skill is needed a. can only lead to a favorable, but invalid report. b. will lead to very dissatisfied stakeholders. * c. is not likely to be sensitive to the effect of the program. d. will increase the likelihood that the report will be accepted.

15. Interviewing to gather data for an evaluation a. is usually used in low-budget jobs. b. indicates that the evaluator does not understand scientific measurement. c. is seldom necessary for conducting a valid evaluation. * d. requires considerable training and skill.

16. When considering evaluation designs, ethical evaluators seek to * a. match the research design and the informational needs of the stakeholders. b. design the most scientifically valid outcome evaluation. c. convince stakeholders that standardized tests are always better than other measurement procedures. d. trace causal relationships between the program elements and the outcome even when examining a new program.

..

21

17. The most scientifically sound evaluation design can fail to be useful if * a. the evaluator never checked to learn if the program elements were actually implemented. b. interviews are relied upon for the data. c. complex statistical analyses are not done. d. the evaluation results are not published in scientifically respectable journals.

18. It would be unusual for an evaluator to discover that a. there were conflicts in the values of different stakeholder groups. * b. there were no conflicts in the values of different stakeholder groups. c. the various stakeholder groups have difficulty articulating how the program is supposed to affect participants. d. the various stakeholder groups draw the same interpretation from the results of an evaluation.

19. A stakeholder group that is seldom consulted in planning an evaluation is the a. management of the facility sponsoring the program. b. staff of the program being evaluated. c. leadership of the program being evaluated. * d. program participants or clients.

20. Sources of different interpretations of the meaning of an evaluation that are largely beyond the control of an evaluator are disagreements over the * a. relative values placed on the outcomes of a program. b. ways variables should be measured. c. sources of evaluation data to be used. d. ways to measure the cost of the program.

21. Evaluation results can fail to detect the positive outcomes of an effective program when a. Type I errors are large. * b. the measures or observation procedures used are not reliable. c. the staff goes to extra effort during the data collection period. d. evaluators overlook negative side effects.

22. Evaluators are more concerned about Type II errors than are basic researchers because a. Type II errors can harm the participants in basic research. b. the probability of making a Type I error can be controlled. * c. concluding that an effective program does not work can deprive people of services. d. the probability of a Type II error is easy to calculate and explain to stakeholders.

23. A major criticism of the designs of published evaluations is that a. Type I errors are too large. b. too many participants are tested or observed. c. too much time is wasted on examining the implementation of the programs. * d. the designs were too weak to detect even moderately large effects.

..

22

24. Focusing solely on the stated goals of a program when designing an evaluation might mean that * a. undesirable side effects of the program might be overlooked. b. undesirable side effects of the program might become the focus of the evaluation. c. the program’s critics might have too much input into the evaluation. d. the program’s sponsors would object to the evaluation design.

25. Unexamined values held by an evaluator might distort the results of an evaluation because * a. the choice of variables to measure may reflect these values. b. the sponsors of the program might recognize these assumed values. c. evaluators usually adopt the values of less powerful groups in society. d. statistical methods do not incorporate values.

26. Detecting negative side effects of programs a. are the result of Type I errors. b. are more likely in programs involving children. * c. can lead to a better program if side effects are recognized and reduced. d. will earn the evaluator accolades from the program’s sponsors.

27. Statistical Type I and Type II errors * a. refer to false positive and false negative conclusions, repectively. b. need not be a concern of program evaluators. c. refer to false negative and false positive conclusions, respectively. d. are sometimes desirable if they lead to an evaluation conclusion that is favorable to the program.

28. Type II errors * a. can be harmful if they lead to the elimination of beneficial services. b. can be avoided by testing a few participants instead of all of the participants. c. lead to conclusions that a program is effective when it is not. d. are, fortunately, rare in program evaluation.

29. Practicing evaluators report that the most common ethical problems involve a. evaluators who do not make contact with stakeholders. b. the refusal of clients to reveal their honest opinions. c. stakeholders who are more interested in basic research than in practical information. * d. stakeholders who seek to influence the conclusions of evaluations.

..

23

Essay Questions 30. Contrast the interests of college students and faculty in creating plans for an evaluation of general education requirements in an arts and science college.

31. How should an evaluator respond to a program manager who requests access to the original survey forms that his employees completed after employees had been assured of confidentiality by the evaluator?

32. Why are Type II errors particularly likely in program evaluations?

33. Outline the most important ethical principles for program evaluators.

34. How might one present the idea of random assignment to a program group or to a control (wait-list) group? Assume that you are asked to prepare a paragraph or two for the staff members who are to obtain informed consent from the people seeking service from the agency sponsoring the evaluation?

..

24

CHAPTER 6. THE ASSESSMENT OF NEED

1. Before a program is implemented, the evaluator should a. not be consulted. b. support whatever decisions the stakeholders make. * c. ask about the conceptual foundations of the plans. d. measure outcomes carefully and reliably.

2. The conceptual model (also called the “impact model”) of a program refers to a. the statistical analysis to be used. b. the needs that program planners ascribe to the population. * c. the processes hypothesized to lead to the outcomes desired. d. a careful description of the program participants and the setting of the program.

3. Evaluators need to know the specific definitions of good program outcomes because a. representatives of the media may ask them. b. using specific objectives permits program sponsors to hide their ignorance. c. the staff members might communicate these expectations to the program participants. * d. using vague definitions makes it impossible to know if program goals were achieved.

4. Knowing the characteristics of the target population will make it easier to a. get stakeholders’ approval for program plans. * b. design a useful service program. c. reduce Type I errors. d. develop financial backing for the program.

5. The needs that professionals ascribe to a population * a. might not be what the population needs. b. will not be questioned publicly. c. should not be changed as a result of an evaluation. d. will always reflect the population’s actual needs.

6. If a target population does not acknowledge the needs that program planners ascribe to them, a. people will probably still pay for the service. b. funding agencies will never support the service. c. people should be educated about their needs because planners are never wrong. * d. people are unlikely to cooperate with the treatment.

7. The staff members of human service programs often __________ the extent of the need for their services. * a. overestimate b. underestimate c. accurately tabulate d. deny

..

25

8. A program might be ineffective because the * a. target population does not feel a need for the service provided. b. program treats the needs of the target population. c. program deals with needs that the population really has. d. program has a well thought-out impact model.

9. Program goals should stress a. only final outcome goals. b. only goals related to the functioning of the program. c. specific initial steps since anything can go wrong while offering a human service. * d. various types of goals, from the development of resources through final outcomes.

10. An impact model permits one to a. hide a weak program from the evaluator. b. avoid hard decisions about resource allocation. c. lower Type I but not Type II errors. * d. relate program activities to final outcomes.

11. Programs that are too weak to affect the problem they are planned to treat a. are seldom implemented. * b. should be revised in the planning stage. c. never have widespread stakeholder support. d. should be implemented carefully.

12. Evaluability assessment refers to careful a. analysis of stakeholder needs. * b. analysis of program plans. c. measurement of program results. d. measurement of program implementation.

13. Census data can be used in program planning by a. asking key informants to evaluate the census data. * b. contrasting one community with larger areas, such as a state. c. searching for a community’s strong points. d. showing errors in census procedures.

14. Case studies of individuals who were unable to obtain social services, such as health care or assistance in getting a job, a. have no place in scientific program evaluation. b. can show the extent of a need for a social service among the residents of a community. c. form the foundation of a quantitative assessment of need. * d. can be used to demonstrate that existing social agencies do not meet the needs of all residents.

..

26

15. Incidence of a problem is to prevalence of a problem as * a. number of cases of the common cold per year is to number of cases of the common cold on a given day. b. number of cases of the common cold on a given day is to number of cases of the common cold per year. c. number of cases of the common cold is to severity of the colds. d. frequency is to magnitude.

16. When surveying residents about the need for a human service, one must not confuse a. informants versus community residents. b. opinions versus attitudes. * c. verbal support for a service versus likely level of use. d. statistical significance versus the reliability of the results.

17. Process evaluation includes a. an assessment of interpersonal interaction during psychotherapy. b. an assessment of successful outcomes. c. a decision about program termination or continuation. * d. an assessment of how the program is being implemented and whether it is being implemented as originally planned.

18. Need was defined as ____________________ between what is and what should be. * a. a discrepancy b. an identity c. a similarity d. a confusion 19. The textbook definition of need would include a difference between what people have and what people a. desire. b. expect on the basis of social norms. * c. require to be in a satisfactory state. d. ideally could make good use of.

20. People offering a service often overestimate the degree to which people need the service because they are in a good position to learn about the people most in need. Remembering these people but ignoring others who do not need the service is an example of a. wishful thinking. b. biased selection among various available pieces of information. c. limited conclusion validity. * d. the decision heuristic, availability.

..

27

21. The consumer price index and FBI crime statistics are examples of * a. social indicators. b. equivalent dependent variables. c. social problems. d. availability biases.

22. Social indicators are quite valuable; however, they a. are nearly impossible to interpret to satisfy a client. b. invariably overestimate the need for services. c. seldom display any valid relationship to real social problems. * d. can be corrupted to serve political purposes.

23. The Focus Group approach to assessing needs utilizes a. representative, randomly selected, members of the public answering a survey. * b. selected community members discussing needs with a trained leader. c. selected community members answering a survey. d. randomly sampled census data focusing on a particular social problem.

24. Need assessment is particularly hard because a. random assignment to groups is ethically wrong. b. statistical analyses so helpful to other phases of evaluation do not apply to need assessment. * c. some people deny their needs and others do not recognize some of their needs. d. census figures are so much in error that they cannot be used.

25. The place to begin planning a social program is with * a. an assessment of the unmet needs of the community. b. an inventory of the resources available from the government and private agencies. c. the staff available. d. a list of services experts think should be provided.

26. Some social needs can be overestimated because * a. people tend to remember tragic, bizarre, and unusual events. b. proponents of social programs usually make balanced presentations of needs. c. quantitative information is easily recalled. d. common events experienced by many people are easy to identify.

..

28

Essay Questions 27. List the best key informants for a curriculum development planning committee in a public high school. Add their strengths and weaknesses as information sources to the list.

28. Develop three potentially useful questions that could be used in a survey to determine the needs of college students for some non-academic, campus service such as personal counseling, vocational counseling, entertainment, etc.

29. If people do not recognize a need, how would this finding affect the planning of a program?

30. Discuss an example of a program that did not contribute to meeting unmet needs.

31. Why is it important to assess needs with careful awareness of and sensitivity to the social practices of the people who are thought to have unmet needs?

..

29

CHAPTER 7. MONITORING THE IMPLEMENTATION AND THE OPERATION OF PROGRAMS

1. Program monitoring includes * a. an examination of a program’s activities. b. an assessment of the conceptual foundations of a program. c. a correlation of the service received and the outcome achieved. d. careful program planning.

2. Program implementation refers to a. careful program planning. b. the degree plans achieve the ends hoped for. * c. the degree plans are put into effect. d. the degree the target population accepts the program.

3. A study of the people actually served by a program often reveals that a. they really do not need the program. b. the staff members seldom understand their own programs. c. they do not appreciate the service provided. * d. the staff members hold some inaccurate views about the population served.

4. Program monitoring a. always involves complex data gathering and analysis. * b. can involve relatively simple observations or a complicated system. c. seldom produces useful information. d. is quite useful, but is seldom done.

5. Program monitoring is carried out in order to a. contribute to the basic research literature in psychology, sociology, or education. b. do the most thorough job of evaluation possible. c. describe what the programs might achieve if thoroughly implemented. * d. describe the essential elements of the program as implemented.

6. When reporting information to stakeholders, it is crucial to remember that people a. resist all new ideas. * b. often incorporate new information without recognizing that some of their beliefs had been incorrect. c. tend to ignore any comments that are critical of the program. d. eagerly seek information that can be used to improve programs.

7. Social service agency records can be used in program evaluation; however, a. agencies do not need program monitoring. b. most service providers refuse to allow evaluators to use these records. * c. many agencies do not keep well-organized records. d. few sets of records are of any use in program evaluation. ..

30

8. When using agency records, the ____________ of the program participants must be ___________. * a. privacy; protected b. individuality; retained c. privacy; sacrificed d. identity; revealed 9. The information used to describe the program participants will be a. about the same regardless of the type of program being described. b. restricted to impersonal information. c. based on the most easily obtained information. * d. determined by the type of program being described.

10. Management information systems are implemented to a. provide information to individual service providers to use with specific clients/students/patients. b. reveal whether service is effective or ineffective. c. root out ineffective staff members or build a case for firing them. * d. track the delivery of services to various groups of clients/students/patients.

11. Management information systems will provide summarized feedback to service providers that is * a. unavailable from other sources. b. useful in adjusting the service to individuals. c. usually very threatening. d. of use only to managers.

12. Feedback on job performance given to individual staff members should usually include * a. specific information about the individual receiving the feedback but only group-level summary information about peers on the staff. b. clearly labeled information on the work of each individual staff member in the immediate work area. c. only summary information on all staff members. d. general performance on the facility but no information on the staff’s performance.

13. Although an individual service provider may not want a supervisor to compare his/her performance to that of other service providers, a. individual staff members should have no say in the matter. b. such comparisons are made so that specific people can be selected for termination. c. they can be assured that such comparisons have never been used in an unethical manner. * d. it is best to learn about problems while they can still be corrected.

..

31

14. A management information system provides informational feedback; however, a. such feedback should be withheld from the staff. * b. actions to effect improvements should be developed. c. the threat of such feedback makes it impossible to use. d. staff members should not even know that such information is being gathered.

15. One problem in developing an information system is a. the staff is usually too eager to work on the system. b. comparing the work of individual staff members is unethical. c. privacy of program participants is necessarily violated. * d. the information might duplicate information already gathered.

16. When designing a management information system, evaluators are particularly concerned to satisfy the information needs of a. the stakeholder group paying the evaluator. * b. all the crucial stakeholders. c. the program participants. d. residents of the community.

17. By working with only one or two departments of an agency a. an evaluator is likely to learn useful information about the information needs of the whole organization. b. an evaluator will be able to design an information system that will meet the needs of all potential users. * c. an evaluator may design an information system seen as irrelevant by many people in the organization. d. an evaluator can efficiently learn about the organization.

18. In setting up an administrative monitoring system it is essential a. that the most sophisticated program be used. b. that the variables be measured without error. c. that the selection of variables to measure does not motivate people to change their behaviors. * d. that the selection of variables represent the behaviors that are actually important to the purpose of the organization.

19. The reasons for developing a management information system include * a. the efficient preparation of required reports. b. satisfying the manager’s curiosity. c. keeping up with good facilities that have information systems. d. having good case studies available to release to newspapers whenever a piece of damaging publicity appears.

..

32

20. Without a management information system, it is likely that staff members will hold some mistaken beliefs about the types of clients or the problems clients have because some clients and problems are more easily remembered than others. When this happens, mistaken beliefs are often held. Judgment theorists call this tendency a. the standard error of the mean. b. the standard error of the estimate. c. hindsight bias. * d. availability bias.

21. The most appropriate type of computer program to use in developing a management information system is a. a sophisticated statistical program. * b. a relational database program. c. a word processing program. d. a newly prepared program written especially for each facility.

Essay Questions 22. Describe the information that a high school principal should have in order to assist in scheduling classes for his school. Focus on the number of various classes needed; ignore state curricular requirements.

23. What is some of the information that a district manager of a fast food chain might use to keep track of the restaurants in her district?

24. Why should evaluators teach program managers and staff members about random variation in indexes of program operation? Think of indexes such as number of clients prematurely leaving a therapist in a given month, student ratings of a specific course in a semester, or percentage of a surgeon’s patients developing infections in a four-week period.

25. Develop some monitoring indexes that would require so much time to complete that staff members would probably refuse to keep the records regularly, but would be tempted to just fill them out from memory at some convenient time.

26. What are the characteristics of monitoring indexes that would encourage staff members to keep and report accurate, timely information on work activities? How do such indexes differ from indexes that might prompt haphazard reporting practices?

..

33

CHAPTER 8. QUALITATIVE EVALUATION METHODS

1. Qualitative evaluation procedures are particularly appropriate when a. the stakeholders can define clear, objective outcome goals. b. there are ample opportunities to randomly assign program participants to groups. * c. stakeholders cannot wait until the program is over to evaluate the program. d. when there are many groups of participants getting identical services (e.g. medicines).

2. Qualitative evaluation methods are especially sensitive to * a. the unique features of the organizational setting of the program to be evaluated. b. the test/retest reliability of the measures used. c. the advantages of matching groups using standardized tests. d. statistical analyses using large samples.

3. When a program has vague or diffuse goals, evaluators a. may want to refuse to conduct an evaluation. * b. may decide to do a qualitative evaluation. c. should seek to pin the staff down to something specific even if it doesn’t reflect the richness of the program. d. suspect that the staff members really don’t know what they are trying to achieve.

4. If the entity being evaluated is a single unit with complex, multifaceted goals, * a. it is hard to imagine how to do an experiment. b. evaluators should insist that the program be simplified. c. there is no valid way of conducting an evaluation. d. a qualitative evaluation should be ruled out.

5. When a carefully done evaluation takes a long time to complete, a. stakeholders become more and more eager to get the results. * b. some stakeholders may no longer be interested in the issue. c. it is probably best to refrain from giving progress reports because political forces will distort them. d. it is probably correct to conclude that it was done poorly.

6. A crucial difference between qualitative and quantitative evaluations is the central importance of _______ in qualitative evaluations. a. complex statistical analyses b. objective assessment c. automated data collection * d. the observations of the data gatherer

..

34

7. Qualitative evaluators believe that a. the goals of programs must be specifically laid out when the evaluation is planned. * b. the evaluator should react to initial findings redirecting the focus of the evaluation as necessary. c. all interpretations of subjectively evaluated data are equally plausible. d. the evaluation design should never be altered regardless of what the initial data reveal.

8. The text likens the understanding derived from qualitative evaluations to the kind of qualitative understanding developed in * a. mystery stories and court decisions. b. descriptions of sports contests. c. the results of political races. d. biological research.

9. When qualitative data gatherers are present in a program setting but do not have a role in the interaction, they are called ______ observers. a. participant b. pseudo-participant * c. non-participant d. pseudo-staff members 10. The settings in which non-participant observers are most likely to fit into easily are those that are a. relatively private. b. laden with conflict. c. highly complex. * d. relatively public.

11. The settings in which the use of participant observers is most likely to be necessary to make direct observations are those that are * a. relatively private. b. do not involve a defensive program staff. c. highly complex. d. relatively public.

12. Physical artifacts of programs are called a. silent witnesses. * b. traces. c. accretions. d. deposits.

..

35

13. When interviewing for a qualitative evaluation, it is best to ask questions that a. demand a “yes/no” answer similar to questions in court cross-examinations. b. can serve to make the best argument for the success of the program. * c. encourage the respondent to elaborate his/her own views about the topic of the question. d. are readily coded numerically and subsequently analyzed statistically.

14. The first phase of a qualitative evaluation requires that an evaluator a. draw up mock tables to be used to present the data to be gathered. b. develop a set of questions with a definite order. c. share openly his/her interpretations with the program staff. * d. make unrestricted observations.

15. Qualitative evaluators share their interpretations of program activities with the staff and other stakeholders as a a. good public relations practice. * b. check on subjectivity and to correct misunderstandings. c. way to verify that observers do have access to all program activities. d. method of gaining insights into statistical analyses.

16. Qualitative evaluators defend their style of conducting evaluations by pointing out that evaluators who use only quantitative methods a. often are influenced by subjective expectations in the choice of issues to address and variables to measure. b. seldom have complete control over threats to internal validity. c. have never carried out a useful evaluation. * d. both a and b 17. Reliability of the interpretation of qualitative observations can be examined by a. calculating split-half reliability coefficients. b. repeating a set of program observations. * c. having a trained person look through the evaluator’s detailed field notes and draw independent interpretations. d. replicating the program in other settings with different program staff members.

18. Some proponents of qualitative methods argue that one of the purposes of evaluation is to empower people by * a. being sure that the most disadvantaged groups in society participate in evaluations. b. expanding basic research related to the type of services offered by the program being evaluated. c. being sure the evaluations match the canons of sound research design. d. being sure that the best program is available to people.

..

36

19. Stakeholder concerns and the empowerment of less powerful stakeholders is central for the _________ evaluator. a. quantitative b. external * c. qualitative d. internal 20. Even the most seasoned evaluators favoring quantitative methods recognize that evaluation design cannot plan for a. the successful achievement of program goals. b. expected negative side-effects. c. failure to meet program goals. * d. unexpected negative side effects.

21. Good probes to use in a qualitative interview encourage an interviewee to * a. elaborate on the first answer using his/her own words. b. defend the initial answers given. c. provide specific information that an evaluator can use to support the conclusions that the evaluator wanted to draw in the first place. d. weigh his/her words carefully just in case confidentiality is violated.

22. The Success Case Method focuses on those participants who were most successful in order to a. provide a favorable evaluation. b. win over cynical stakeholders. c. shame unsuccessful participants into better behaviors. * d. identify behaviors and program experiences that are associated with success.

23. The Appreciative Inquiry strategy begins by learning what staff members believe a. needs the most improvement. b. are the most controversial aspects of a program. c. are the directions that they most want the program to begin. * d. are the aspects of the program that work particularly well.

..

37

Essay Questions 24. Describe the characteristics of program settings in which qualitative evaluations are probably more appropriate than quantitative evaluations.

25. Compare the potential for threat to program staff members from a qualitative evaluation versus program monitoring standpoint.

26. Describe the advantages of combining features of qualitative evaluations and program monitoring in terms of the credibility and interpretability of the evaluation and its potential usefulness to stakeholders.

27. Which steps (that is, need assessment, implementation, outcome, cost analyses) in a program evaluation are most appropriately examined in a qualitative fashion?

28. Why do some evaluators criticize qualitative evaluation methods?

..

38

CHAPTER 9. OUTCOME EVALUATIONS WITH ONE GROUP

1. Pretest-posttest designs permit the evaluator to learn whether a. the program caused a change in the outcome variable. * b. the participant improved during the program. c. improvement was great enough to reject threats to internal validity. d. Type I errors could account for a change.

2. A posttest-only evaluation design is often useful when a. the stakeholders do not have advanced training in research methods. * b. there is a definite, objective criterion of successful outcome. c. there is little time or resources for an evaluation. d. program success is hard to define.

3. Internal validity refers to whether a. a statistically significant change occurred. b. Type II errors occurred. * c. a research design provides sufficient control that one can learn whether the program caused a change in an outcome. d. the findings can be generalized to other similar groups.

4. Evaluation designs should be selected so that * a. the stakeholders’ questions can be answered. b. basic knowledge in the field can be gained. c. the evaluation will merit publication. d. the expectations of the major stakeholders will be supported.

5. The use of small samples can lead unsophisticated evaluators to conclude falsely that a. Type I errors were held to only 5%. b. findings could be applied to additional populations. * c. a program had no impact. d. statistically significant results were observed.

6. Conclusion validity refers to * a. statistical significance. b. high internal validity. c. high external validity. d. adequate clinical power.

7. The question of how much improvement in a treatment group is enough * a. cannot be settled empirically. b. should be determined after examining the results. c. is based on internal validity considerations. d. is an irrelevant issue.

..

39

8. Correlating the amount of service received with the condition of the participants after receiving services from a program * a. can sometimes suggest that a service is not effective. b. cannot be useful if internal validity is low. c. simply requires correlating a change score with a measure of service. d. is more valid in isolating causes than a true experiment.

9. The difference between the mean of the outcome variable after a program and the mean of the variable measured before the program divided by the standard deviation of the pre-program scores is an index of * a. effect size. b. correlation. c. statistical significance. d. generalization.

10. The threat to internal validity (called “maturation”) refers to * a. predictable changes in people that can be expected solely due to the passage of time. b. events happening in the community that will change the behavior of the program participants. c. the improved skill of a program staff as they become more experienced. d. differences between people who want a program and those who do not want it.

11. The threat to internal validity called “history” refers to a. predictable changes in people that can be expected solely due to the passage of time. * b. events happening in the community that will change the behavior of the program participants. c. the improved skill of the staff as they come to understand the history of the program. d. improvement in the skills of the evaluator in measuring outcome variables.

12. Local history is a form of threat to internal validity referring to a. predictable changes in people that can be expected solely due to the passage of time. b. events happening in the community that will change the behavior of the program participants. * c. events that are unique to a particular program group. d. the changing nature of an evolving program.

13. The fact that people who choose to, or who are chosen to be, in a program are different from those who are not in a program creates the threat to internal validity called a. history. b. regression to the mean. * c. selection. d. testing.

..

40

14. Another name for the threat to internal validity called “attrition” is a. history. * b. participant loss. c. instrumentation. d. regression to the mean.

15. If an evaluator judges the rate of success of a program on the basis of those who complete the program, she is likely to be failing to deal with the threat to internal validity called * a. attrition. b. regression to the mean. c. testing. d. history.

16. When initially extreme scores are followed by less extreme values (even when hypothesized), one wonders whether _____________ can account for the change. a. maturation b. selection * c. regression to the mean d. testing 17. Regression to the mean is a threat to the internal validity of the pretest/posttest design whenever a. a long time transpires between the two observation periods. b. program participants get older between the two observations. * c. people are chosen for a program because initial observations showed that they needed the program more than others did. d. evaluators gain experience making the observations for the pretest and the posttest.

18. The threat to internal validity called “testing” * a. refers to changes in behavior due to the observation techniques. b. refers to the use of measurement procedures themselves. c. refers to school-based program evaluations. d. cannot be a threat to pretest/posttest designs.

19. The threat to internal validity called “instrumentation” refers to a. failures in the electronic equipment sometimes used to collect data. b. problems that are beyond the evaluator’s control. * c. changes in the meaning of the scores of the measures used. d. that which cannot be a threat to pretest/posttest designs.

20. Repeatedly dividing the sample on the basis of demographic descriptors to look for differences in the means of outcome variables for different subgroups of the sample * a. raises the chances of a Type I error. b. raises the chances of hindsight error. c. lowers the internal validity of a study. d. increases the level of external validity. ..

41

21. The pretest/posttest design using only the program’s clients can help introduce the staff to the merits of program evaluation because a. this is a very sophisticated research design. b. the threats to internal validity are all controlled except for instrumentation and the interaction of selection by maturation. * c. a plan to compare the program group with people in other programs might be rejected because the staff feels too threatened. d. it is easier to take the needs of all stakeholders into account using this design.

22. Retrospective pretests were developed to be sensitive to a. hostile participants in the evaluation. b. expert opinion about what went wrong in the implementation of the program. c. how people believe that their parents raised them. * d. changes created by the program that may have changed how participants understand themselves.

Essay Questions 23. Explain which threats to internal validity need to be considered in a comparison of the mean scores on a test administered on the first and last days of this course.

24. Contrast the effects of attrition depending on whether the most successful participants leave prematurely (because they feel they no longer need the program) or the less successful participants leave prematurely (because they feel that they are not making progress).

25. Illustrate how advances in medical technology can make it appear as though the incidence of a disease has changed over time.

26. Consider a program to assist new parents to develop parenting skills. Why might their views of their own preprogram skill levels change as a result of their own development?

27. Develop two hypothetical situations in which threats to internal validity would serve to create an apparent positive program effect and two in which an apparent negative effect would be created.

..

42

CHAPTER 10. QUASI-EXPERIMENTAL APPROACHES TO OUTCOME EVALUATION

1. To show that a program had an effect on an outcome variable, it is necessary to a. inform all stakeholders beforehand. b. dismiss Type II errors. c. explain why some people did not benefit from the program. * d. eliminate explanations of change not related to the program.

2. The ways in which evaluators expand on the pretest/posttest design in order to control for threats to internal validity include the following EXCEPT for * a. increasing the size of the sample to reduce Type I and Type II errors. b. observing the participants at additional times before and after the intervention. c. observing additional people who have not received the program. d. using a variety of variables, some expected to be affected by the program, some not.

3. The primary reason to increase the number of observation periods is to * a. examine non-program effects related to history and maturation. b. reduce the chances of a Type I error. c. increase external validity. d. simplify the statistical analyses necessary.

4. A time series evaluation design leads to a. an unavoidable loss of statistical power and an increase in Type II errors. * b. requires repeated observations of the same entity at many points in time. c. includes the development of measures of alternative outcome variables. d. is usually needed for programs administered to small groups who are part of a large organization.

5. In the graph below, all preintervention observations are less favorable than the post interventions observations; a. clearly the intervention had an impact. b. regression to the mean is the most likely reason for the change. * c. the possible effects of maturation or history should be considered carefully. d. partial correlations should be calculated before drawing any conclusions.

..

43

6. A pattern like the one in the graph below could occur because * a. staff members’ enthusiasm led to a short term improvement in outcomes. b. program participants do not accept the intervention. c. maturation effects may be strong. d. the evaluator became more accurate in assessing success during the preintervention observations (i.e., instrumentation problems).

7. A particularly strong time series design is available when a. an outcome variable is based on self-report. * b. the intervention can be removed and reinstated later. c. the samples are small. d. local history has an effect on the outcome variable.

8. Smoothing a graph of observations made over a time series a. is best done visually by passing a smooth line through the data points. b. is essentially dishonest and unethical since means or medians are substituted for the real data. * c. can be done by substituting the median of 3 adjacent data points for the original points. d. can make any pattern of data look like an effect of a program.

9. When a group similar to the program group is available, a good quasi-experimental evaluation design is a a. time series design. * b. non-equivalent, control group design. c. partial correlational analysis. d. pretest/posttest design.

10. The major weakness of the nonequivalent, control group design concerns a. instrumentation effects. b. testing effects. c. low statistical power. * d. selection effects.

..

44

11. If the program participants in a non-equivalent control group design are chosen for a program because they are extremely needy, it is likely that a * a. regression effect will be a viable alternate interpretation. b. instrumentation effect will be a viable alternate interpretation. c. attrition effect will be a viable alternate interpretation. d. testing effect will be a viable alternate interpretation.

12. If one selects the poorest readers from an elementary school class using a standard reading test and then places them in an ineffective reading program, retesting will likely show an improvement in scores due to a. attrition. b. regression to the mean. * c. regression to the mean and maturation. d. instrumentation and history.

13. An uncontrolled threat to internal validity could a. make a program appear effective. b. make a program appear ineffective. * c. make a program appear either effective or ineffective. d. raise the chance of a Type I error.

14. The regression discontinuity design is a. a specific form of a time series design. b. meant to make use of the threat of regression to the mean. * c. a specific form of a non-equivalent control group design. d. impossible to use in evaluation settings.

15. The regression discontinuity design can be analyzed using a. a t test. b. medians of three adjacent points. c. change scores. * d. multiple regression.

16. A most crucial concern in selecting additional dependent variables to compare to an outcome variable of interest is to be sure that these additional variables will * a. not be affected by the program. b. be affected by the program. c. increase the external validity of the evaluation design. d. not reduce the probability of a Type I error.

17. An especially strong quasi-experimental design combines the features of a. posttest only design with regression discontinuity. b. time series with a posttest only design. * c. a time series design with a non-equivalent control group design. d. regression discontinuity with regression artifacts.

..

45

18. Using two time series with interventions delivered at different times a. is especially affected by maturation influences. b. is not affected by history. c. can be used with only one group. * d. includes a replication.

19. Comparison groups are best chosen in order to control for * a. specific threats to internal validity relevant to a particular evaluation setting. b. all possible threats to internal validity. c. unauthorized tampering with the administration of the outcome variables by worried staff members. d. sample size variations.

20. Work or school absenteeism is a particularly good outcome variable because a. it cannot be faked. b. only good reasons would keep employees or students at home. * c. it is an important variable to businesses and schools and is routinely recorded. d. no threats to internal validity can influence absenteeism.

21. A time series design used when an intervention has occurred during a short time period (e.g. one week out of ten observation periods spread out over ten weeks) is called a(n) a. pretest/posttest time series. * b. interrupted time series. c. intervention time series. d. regression time series.

22. Nonequivalent control group designs a. nearly always ensure high internal validity. b. are best used if a selection-by-maturation interaction needs to be controlled. c. are seldom used in program evaluation. * d. utilize at least two groups which are not randomly formed.

..

46

Essay Questions 23. What would be some appropriate comparison groups for a quasi-experimental evaluation of a change in a probation law? A change in the requirements for a college major? A review course for the Graduate Record Examination?

24. Suppose that a hospital made an effort to improve the meals served to patients. How could a control construct design be used to learn if patients liked the meals better than before the change?

25. The selective control design refers to combining various quasi-experimental designs to control the threats to internal validity that are thought to be the most viable to a particular evaluation. Suppose that a college instituted a program to help retain freshmen so that a greater proportion re-enrolled for the sophomore year than in the past. The obvious comparison is between the re-enrollment rates of the current freshmen and the previous year’s freshmen. What threats to internal validity weaken the conclusions that one could draw from such a comparison?

Expand on this basic design, seeking to control at least some of the threats to the internal validity of the design.

26. Suppose that the North Dakota state legislature raised the high school requirements for mathematics in an effort to assure their citizens had a better chance to gain admission to more selective colleges. Further suppose that the legislature of South Dakota adopted the identical requirement two years later. Explain which quasi-experimental design you would use to evaluate the outcomes of the changes in standards. What negative side effects might be observed among high school students?

..

47

CHAPTER 11. USING EXPERIMENTS TO EVALUATE PROGRAMS

1. Although some quasi-experimental designs are fairly effective in separating the effect of the program from viable alternative interpretations, a. they are not feasible in agency settings. b. they are more disruptive of agency procedures than are experiments. * c. a lingering doubt may remain about internal validity. d. staff will greatly object to quasi-experimental methods.

2. Whenever preexisting, intact groups are used for an evaluation, a. one can treat the groups as if they were equivalent. b. one can be sure that the participants were randomly assigned to the groups. c. the evaluation is readily interpretable. * d. the groups usually differ on many variables.

3. When an evaluator recognizes a potential threat to the internal validity of an evaluation, * a. it may be impossible to control for it short of randomization. b. correlational designs are needed. c. using just one group would permit the threat to be ignored. d. a pretest/posttest design can be used.

4. A randomized experiment a. cannot be a good evaluation design. * b. relieves the evaluator of worrying about most of the threats to internal validity. c. has high external validity. d. lowers Type I error compared to a quasi-experimental design.

5. Pretests are recommended even when the evaluation design consists of randomly formed groups because a. correlations between amount of service and change scores improve interpretability. b. programs are not implemented as designed. * c. pretests will permit some interpretations even if randomization procedures are compromised. d. non-equivalent control groups need to be equated using a pretest.

6. The use of pretests in an experimental evaluation may introduce the internal validity threat of a. regression to the mean. b. history. c. maturation. * d. testing.

..

48

7. Service providers sometimes object to random assignment to groups because they try to match particular social services to particular individuals, a. however, it is impossible to match services to program participants. * b. however, there is considerable variation in the services provided to people with similar needs even in the absence of an evaluation. c. therefore, evaluators often permit program staff to form the comparison groups. d. moreover, services are usually planned to meet each participant’s unique needs.

8. The use of pretests with an experimental evaluation * a. permits one to control for random individual pretest differences, thereby lowering Type II error. b. gives the appearance of credibility to the evaluation but is of no practical use. c. is a violation of experimental design. d. will not be understood by stakeholders.

9. A good approach to interpretation when a program fails to achieve its objectives is a. to hold tight to previously stated objectives and call a spade, a spade. b. to recognize that the objectives may have been too ambitious and to focus on some phase of the program that worked. * c. to treat well-conceived programs as experiments and search for what can be learned to improve the next version of the program. d. to point out that repeated failures probably indicates low staff motivation.

10. When determining the costs of a valid, experimental evaluation of a new medical treatment, innovators often overlook the costs of * a. making a Type II error. b. data collection. c. monitoring the program and control groups participants. d. not setting the probability of a Type I error accurately.

11. An evaluation that _____________ is a waste of resources. a. does not focus on causes * b. cannot be interpreted c. uses correlation coefficients d. does not support the program 12. Usually a new program cannot be implemented in all parts of an organization simultaneously.

This pattern a. is unfortunate for evaluation. * b. can be utilized by an evaluator to create control and experimental groups. c. creates threats to internal validity that are impossible to control. d. violates ethical standards since this is unfair to those who receive a program later.

..

49

13. One can expect the most objections to a proposal for an experimental evaluation of a program when a. a new program is introduced. b. resources are scarce. c. there is controversy about the effectiveness of the program. * d. the program has been in place and accepted for years.

14. When an ambitious program evaluation of a public policy (such as welfare reform) fails to find that the new policy had a positive impact, a. the effort in implementing and evaluating the program was a waste of time and money. b. one should assume that the evaluator did a poor job. * c. one can expect that the quality of the debate over the policy will be more enlightened. d. the policy will be changed within months of the evaluation report.

15. Once stakeholders agree that random assignment to groups will be permitted, it is crucial that a. the evaluator periodically monitor the outcome variables. b. stakeholders be permitted to make the actual group assignments. * c. different rates of attrition among groups not be permitted to introduce bias. d. regression to the mean be carefully examined.

16. Diffusion or imitation of the program by people in a non-treatment control group tends to * a. make an effective program look less effective than it is. b. make an ineffective program look more effective than it is. c. make a good program appear to have a negative impact. d. nullify the effects of attrition from control groups.

17. Attempts to compensate the control group tends to * a. make an effective program look less effective than it is. b. make an ineffective program look more effective than it is. c. make a good program appear to have a negative impact. d. nullify the effects of regression to the mean.

18. When control groups feel rivalry with the experimental group, it is possible that a. attrition will increase among the members of the experimental group. * b. an effective program will look less effective than it is. c. an ineffective program will look more effective than it is. d. the performance of the experimental group will improve.

19. If a control group experiences resentful demoralization because its members realize that they are not receiving desired program benefits, it is possible that a. attrition will increase among the members of the experimental group. b. an effective program will look less effective than it is. * c. the program will look more effective than it is. d. the performance of the control groups will improve.

..

50

20. When an outcome evaluation is designed as an experimental evaluation, a. it is unnecessary to worry about implementation since a poorly implemented program will probably yield a poor outcome. b. one can relax since the data can be clearly interpreted. c. the data will “speak for themselves” when the analysis is done. * d. it is still essential to examine the degree of program implementation.

21. One of the many differences between quasi-experiments and true experiments is that a. threats to external validity do not have to be considered in true experiments. b. outcomes observed reflect the degree of experimental control. * c. a true experiment does not require a pretest. d. results from quasi-experiments are usually more interpretable than results from true experiments.

22. One objection made about true experiments that is not valid is that a. the control group may miss out on a valuable program. b. random assignment creates some artificiality in the program setting. c. they are more expensive than pretest/posttest designs. * d. evaluators are only interested in data, not people’s lives.

23. When services or medical treatments are offered before being carefully evaluated, a. little harm is done because if a treatment is ineffective, people will quickly notice and discard it. * b. much harm can be done because some ineffective treatments have come to be considered standard treatments. c. theoretically some harm could be done, but people seldom implement a harmful program. d. much harm will occur because nearly all non-evaluated programs are harmful.

24. Interpretations of outcomes of experimental designs are much more useful when a. stakeholders focus on statistical significance. b. t tests are used. * c. the amount of change a program could cause has been discussed before examining the findings. d. correlations are calculated for program and control groups separately.

..

51

Essay Questions 25. Describe the organizational conditions that favor conducting an experimental evaluation.

26. Describe the threats to internal validity that can hinder the interpretability of experimental evaluations.

27. Describe the reasons that program staff often give when they object to conducting an evaluation using an experiment involving random assignment to groups.

28. How might an evaluator seek to reduce attrition among the members of control groups?

29. Distinguish the internal validity threat of history from the internal validity threat of local history.

..

52

CHAPTER 12. ANALYSES OF COSTS AND OUTCOMES

1. An advantage that accountants have over program evaluators is that a. their crucial unit of analysis, a dollar, is widely disputed. * b. their crucial unit of analysis, a dollar, is widely accepted. c. accountants are better educated. d. business activity contains fewer ethical conflicts than human service work.

2. Variable costs are to fixed costs as * a. electric bills are to monthly rent charges. b. monthly rent charges are to electric bills. c. costs incurred are to future costs. d. liability insurance costs are to salaries.

3. Sunk costs are those that a. change with the size of the program. b. occur as the program continues. * c. have already been expended. d. can be recovered if the program is cancelled.

4. Costs that have already been expended and cannot be recovered are called _______ costs. a. opportunity costs b. variable c. excessive * d. sunk 5. Indirect costs are a. usually well understood by the staff. b. those that are incurred in providing services for specific clients. c. largely a fiction developed by accountants for bookkeeping conveniences. * d. those that cannot be associated with specific individual clients.

6. Inaccurate estimates of program costs a. seldom hurt anyone. * b. make it very hard to plan rationally. c. are usually pessimistically high. d. are never made by program planners working for the Federal Government.

7. The outcomes of a cost-effective program a. exceed those of other programs. b. are less costly than those of other programs. * c. are achieved at a lower cost per unit of outcome than other programs seeking to affect the same variables. d. are achieved at a greater cost per unit of outcome than other programs seeking to affect the same variables. ..

53

8. When outcomes of a program can be expressed in dollars, then * a. a cost-benefit study can be done. b. a cost-effectiveness study cannot be done. c. one can be fairly sure that the program is effective. d. the program has been trivialized.

9. An evaluator doing a cost-benefit study may ignore certain benefits when a. it has been shown that costs exceed benefits. b. some of the costs have been ignored as well. c. one is not interested in seeking to publish an article reporting the findings. * d. it has been shown that some benefits exceed program costs.

10. Cost-effectiveness analyses are performed when one a. can express the outcome in dollars. * b. cannot express outcomes in dollars, but can measure the outcomes of alternative programs that are designed to affect the same outcomes. c. quasi-experimental designs are not possible. d. cannot find parallel outcomes from two or more programs.

11. Cost-effectiveness analyses _________ be used in decisions when outcomes of possible programs are expressed in different units, i.e. lives saved versus jobs created. a. can b. must * c. cannot d. should not 12. Cost-utility analyses refer to methods of relating program costs to a. objective judgments of the value of certain outcomes. b. the costs of negative side effects. c. losses due to inadequate program implementation. * d. subjective judgments of the value of certain outcomes.

13. In judging the usefulness of a cost-benefit evaluation, one must be wary of a. values assumed in the choice of outcome variables. b. the validity of the evaluations from which program benefits have been estimated. c. the assumptions used in assigning dollar values to the outcomes. * d. all of the above 14. The value of income one forgoes by attending college instead of getting a job right after high school is called a. an indirect cost. b. a sunk cost. * c. an opportunity cost. d. a recurring cost.

..

54

15. Since most human service programs are expected to yield benefits in the years after the program participants leave the program, a. cost-benefit studies are conceptually impossible. b. costs cannot be related to effectiveness. * c. the values of future benefits need to be estimated. d. sunk costs take on considerable importance.

16. If you have never had an automobile accident and, thus, never have collected from your auto insurance policy, you should a. see the insurance as having been a waste of money. * b. recognize that the peace of mind from having insurance was a benefit. c. drive carefully because you have been living a charmed life. d. drive carefully because an accident is overdue.

17. Evaluators using cost-effectiveness and cost-benefit analyses have been criticized for * a. placing a value on priceless human lives. b. valuing all people equally. c. failing to conduct internally valid research. d. avoiding more standard research designs.

18. An evaluator might place a dollar value on a human life; however, a. no one takes it seriously. b. this shows disrespect for the people studied. * c. this is simply an attempt to show how much money society, in fact, devotes to avoiding the premature death of a person, such as an employee. d. such information never has any use in making selections among programs.

19. A problem with using cost-benefit analysis is that a. while the value of benefits is easy to calculate, the costs of the program are difficult to estimate. * b. many benefits, especially in human service settings, are hard to place a dollar value on. c. the dollar unit is not that important in human service settings. d. it does not add much information to program evaluation reports.

20. Suppose that an administrator had to select neither or either program A or B and that the cost-benefit ratios were Program A = 1:2.3 and Program B = 1:1.7. What should the administrator do? * a. Select Program A. b. Select Program B. c. Select neither A nor B since 1 is less than 2.3 and 1.7.

..

55

21. If the cost of a program is $1,000 and its benefits are estimated to be $900, what is the costbenefit ratio? a. 1000 to 900 b. 900 to 1000 * c. 1 to 0.9 d. 0.9 to 1 22. Estimated costs for planned medical services for children and the elderly a. should be compared to earnings potential. b. provide a basis for cost-benefit analyses. * c. can be used in cost-utility analyses. d. should not be publicized or used in planning.

23. When considering the costs and benefits of programs, different stakeholder groups are especially likely to draw different conclusions from the analyses if * a. the people who collect benefits are different from the ones who bear the costs. b. the people who benefit are also the ones who pay the costs. c. opportunity costs are placed in units of dollars. d. sunk costs are correctly considered.

Essay Questions 24. Support the position that cost analyses must be part of comprehensive program evaluations.

25. University faculty members often dispute the need to cover indirect costs for their research activities. They sometimes comment to the effect that “The buildings are here already, my lab will be heated because the classrooms will be heated anyway; therefore, indirect costs of my research is just a way of raising money.” Marshall the reasons why indirect costs must be considered in assessing the costs of research.

26. Why is a $100 benefit available in five years from now more valuable than a $100 benefit available ten years from now? (Assume that both are absolutely assured.)

27. The core issue of cost-benefit analyses (or cost-effectiveness analyses) is the unit of analyses used—dollars of benefit per dollar of cost (or units of outcome per dollar of cost). Illustrate the importance of the issue of units of analysis in examining program impacts. How might one err in examining benefits or measuring effectiveness? (Hint: think about choosing outcome variables to measure.)

28. Some critics of cost-benefit analyses point to the monetary value set on saving a human life, say, $8 million dollars, and ask: “Does that mean that if I pay $8 million dollars to someone (or to his heirs), I can kill them?” What misunderstanding of cost-benefit analyses does this comment reflect?

..

56

CHAPTER 13. EVALUATION REPORTS: INTERPRETING AND COMMUNICATING FINDINGS

1. The introduction to an evaluation report usually contains a. a summary of the findings. b. the statistical analyses. * c. a description of the setting of the program. d. an analysis of the conflicts among the stakeholders groups interested in the evaluation.

2. The introduction should deal with the reasons for conducting an evaluation because a. the evaluator needs to justify his/her work. * b. the kind of evaluation done and the implications drawn will follow from the reasons for conducting an evaluation. c. few readers will be clear on the statistical analyses. d. formative evaluations, although few in number, need to be clearly labeled.

3. It is important to be sure that readers of an evaluation report recognize what is NOT dealt with in an evaluation * a. because many readers hold limited definitions of what a program evaluation is. b. because readers may be very willing to apply whatever they find in an evaluation report. c. and why stakeholders should seldom have much input into the study. d. even though the concept of program evaluation has become widely understood in most organizations.

4. Including a description of the program participants observed during program evaluations helps readers to a. find ways to discredit the evaluation. b. depersonalize the participants. c. rationalize program failures so that funding is not withdrawn. * d. judge the degree the participants are the ones planned for.

5. Methods of selecting a sample are described in reports because * a. internal validity may be threatened by some methods of finding a sample. b. testing effects can confuse some participants. c. correlational analyses are always more informative when the sample is homogeneous. d. Type I errors can be affected by an improper approach to selecting a sample.

6. An evaluation report could be more difficult to write than an article on basic research because a. there are often more observations in an evaluation. b. the readers of an evaluation report will have more statistical sophistication. * c. the readership of an evaluation report may be more heterogeneous. d. the readership of an evaluation report will be more skilled in detecting threats to internal validity.

..

57

7. In presenting evaluation results a. it is sufficient to present statistical significance tests since the crucial issue is whether the results are reliable. b. it is essential to compare group differences to the standard deviations of the variables. c. the data will speak for themselves so detailed presentations need not be made. * d. the meaning of group differences can only be interpreted in the light of the importance of the variable observed.

8. Small improvements in important variables—death rate, for example—are * a. worth finding and stressing in reports. b. simply small effects and not worth stressing. c. just as important as any other statistically significant finding. d. to be judged on the basis of the t test or size of the correlation coefficient.

9. Small improvements in outcome variables a. should be omitted from evaluation reports in order to focus attention on large effects. * b. can be crucial if the variables are important or affect many people. c. should not be overly stressed. d. are important only if the results are statistically significant.

10. The “adequacy” of a program refers to whether a. the program produces statistically significant changes in outcome variables. b. financial supporters of the program are satisfied with the program. * c. the program is sufficiently ambitious to serve the actual needs of the target population. d. the program has been implemented as specified in the program’s plans.

11. A written evaluation report includes a. an abstract much like those used with research reports. b. elaborate footnotes detailing the precise statistical tests used. c. detailed statistical results in the body of the report so that readers can evaluate the analysis. * d. a summary that provides readers with an overview of the evaluation and recommendations.

12. The use of various versions of evaluation reports is prompted by the need to a. hide some outcomes from certain stakeholders. * b. write for a variety of stakeholders. c. convince all stakeholders of the value of the program. d. thoroughly document program failure to avoid rationalization by the program staff.

13. Mixing quantitative summaries of observations and qualitative descriptions of the program * a. will illuminate the meaning of the numbers and the descriptions. b. is likely to confuse readers. c. cannot be condoned by methodologically sound evaluators. d. inevitably opens the door to sloppy evaluations.

..

58

14. Large sample sizes tempt evaluators and writers of popular reports of evaluations (e.g. newspaper reporters) to a. dismiss the work as overly expensive. * b. ignore the representativeness of the sample. c. question whether the sample represents the people served. d. conduct overly simple analyses.

15. Internal evaluators often experience the problem of being unable a. to carry out correct statistical analyses. * b. to observe samples large enough to detect even moderately-sized effects. c. to present their finding to the right stakeholders. d. to understand the program in the context of the larger organization.

16. In presenting a written report, a conflict about who has the right to see the report first may be reduced by a. releasing it to all stakeholders simultaneously. * b. making a distinction between a draft report and the final report. c. allowing those who pay for the evaluation to have the report first. d. making those who want to see the report request it specifically.

17. A strong argument for letting the program manager and staff see a near-to-completed draft before other stakeholders do is to * a. learn about mistakes in factual matters. b. respect friendships that may have developed in the course of the evaluation. c. permit them to develop rationalizations about program failures. d. encourage recommendations to be put into effect before others learn about problems.

18. Evaluation data, unlike basic research data, belong to a. the evaluator. b. the program participants who provided data. c. the program staff members. * d. whomever financed the project.

19. Disputes about release of evaluation findings are best a. resolved by the funding stakeholder. * b. avoided by making explicit agreements about what can be done with the data before the evaluation is conducted. c. handled by the evaluator’s best judgment since the evaluator conducted the evaluation. d. placed in the hands of the manager and staff of the program that was evaluated.

20. Program participants a. have no right to evaluation data. b. may have evaluation data if they are personally paying for the service. * c. could be sent a summary of the evaluation, which includes no confidential information. d. should be approved on a case-by-case basis before receiving any program evaluation results. ..

59

21. Evaluators recognize that an oral presentation of findings * a. is essential in assuring that stakeholders understand the findings. b. is of marginal importance in communicating the findings of the evaluation. c. can replace a written report. d. permits the evaluator to alter the conclusions depending on the reaction of the stakeholders.

22. Plans for communicating the findings of evaluations * a. should be developed while the evaluation is being planned. b. can be dealt with once the findings are available. c. should be kept confidential during the analysis of the observations. d. should be left in the hands of the stakeholder funding the evaluation.

23. In forming plans to communicate evaluation findings, evaluators a. should follow a standard procedure with all program evaluation projects. b. remember that a confidential approach to communication is likely to create interest in the report and greater acceptance. * c. are most successful if they are flexible and develop a communication plan that matches the specific needs of the setting of the project. d. stress the scientific nature of program evaluation even if that means that some potential users cannot understand the findings.

24. When preparing evaluation reports, it is important to reflect on a. the gullibility of most stakeholders involved in social services. * b. common misunderstandings of statistical analyses. c. the likelihood that evaluations usually find large program effects. d. the need to maintain good interpersonal relationships even if it means shading the findings to match the manager’s views.

Essay Questions 25. Distinguish between the purposes of a written report and those of an oral presentation of the findings of a program evaluation.

26. Contrast the different emphases that distinguish an evaluation report from an article in a research journal.

27. Why is it important to send progress reports to stakeholders while the evaluation is underway?

28. Contrast the issues that would be the focus of (a) program evaluation report available to program managers, versus (b) a report prepared for recipients of program services.

29. Describe some ways that recommendations could be offered that increase the probability of their being totally rejected by the stakeholders. ..

60

CHAPTER 14. HOW TO ENCOURAGE UTILIZATION

1. If program staff members are worried about the findings of an evaluation, * a. they will resist adopting ideas for change contained in an evaluation report. b. they are likely to grasp at any suggestion. c. they will be eager to innovate. d. one can assume that the staff has pride in its work.

2. When evaluators sense a fear of evaluation, they would be wise to a. avoid conducting the evaluation since all reports will be distorted by the timid staff. * b. encourage the management to state clearly that program improvement and innovation are wanted, even if some innovations may turn out to be ineffective. c. suggest in strong terms that the staff should get its act together. d. point out to the staff that fear of evaluation is a symptom of a staff which feels there are problems with its own program.

3. If unsuccessful program innovations are not identified, * a. others are more likely to implement ineffective programs. b. no one will forget them. c. a Type I error has occurred. d. implementation problems may have caused the failure.

4. Evaluators are more likely to see evaluations used if a. reports are simple, clearly labeling failures as failures. b. suggestions are only tangentially related to the program. * c. staff feel respected. d. staff are concerned about program termination if goals are not met.

5. The conclusions of evaluation reports should seldom a. deal with failure. b. give support for a program. * c. give blanket approval or disapproval of a program. d. be based on specific data.

6. Sometimes an innovative program will fail to do better than currently used approaches, * a. but the new approach may still have certain positive features (e.g. lower cost) that suggest it be adopted anyway. b. but the possibilities of this are extremely small. c. and the evaluator can expect to see recommendations carried out with little hesitation. d. but for the sake of the morale of the staff, this failure should be ignored or explained away.

..

61

7. When an evaluator fails to document successful achievement of outcome goals, but the innovation seems to makes sense anyway, evaluators * a. can speak of the innovation having “common sense plausibility” and urge that further evaluations be conducted. b. can question the integrity of those who firmly supported it. c. should immediately raise the possibility of a Type I error. d. should actively try to persuade program supporters to change their attitudes.

8. If only the more costly elements of a program are eliminated because of negative findings, a. one should question the value of the evaluation. b. the staff probably has excessive power. c. the evaluator did not do a thorough job of communicating the negative nature of the findings. * d. the evaluation served a useful purpose, especially if additional study is to be done.

9. Evaluators need not be overly discouraged if the implications of an evaluation are not taken into consideration immediately because a. organizational politics are strong. b. you can’t win ‘em all. c. evaluators should not be concerned about utilization anyway. * d. evaluation findings are only one form of information that managers must use.

10. The finding of “no differences” between a comparison group and a treatment group could * a. be due to poor evaluation methodology or inadequate analyses. b. be due to sabotage on the part of the program staff. c. create an arrogant attitude among the program staff. d. be what most innovators of new programs want to see.

11. Implementing the results of evaluation studies is encouraged when a. the staff holds the evaluator in fearful awe. b. the written results are permitted to “speak for themselves.” * c. the evaluation feeds back into program planning. d. evaluators keep a proper “hands off” attitude toward the program.

12. Given the limited development of the science of program evaluation, weaknesses in every study could lead to all results being questioned. It is indefensible to use this fact to * a. explain away all program weaknesses identified by an evaluation. b. call for further evaluation. c. work closely with stakeholders to interpret the results. d. suggest improvement in further evaluations.

..

62

13. Acceptance of valid, unfavorable feedback is enhanced when managers can view such feedback a. like a baseball score—its only a game. * b. like medical tests—if the results show that the patient is ill, the doctor does not reject the test. c. with regard to the source of the feedback—students or employees do not have the big picture. d. like a horse race—there can be only one winner, but the losers may well still be fast horses.

14. Evaluation findings do not provide definitive conclusions about the value of a type of program because a. evaluators do not use valid research designs. b. evaluators often fudge reports to favor a program. * c. conditions at other locations are often different from the setting in which an evaluation was conducted. d. reports are written which are sensitive to stakeholder information needs.

15. Strongly polarized attitudes toward a policy, such as limitations on handgun ownership, will result in people who a. are able to find common ground with each other to draw reasoned conclusions from an evaluation of the policy. b. are eager for the evaluation findings. c. trust the evaluator seeking to remain neutral on the issue. * d. are likely to select those findings of the evaluation that support their views and reject opposing ideas.

16. Evaluation conclusions are not likely to be applied and indeed should not be applied when * a. a fixation on outcome evaluation led to the use of methods that could not have produced interpretable findings. b. evaluations focus on the extent of program implementation. c. the costs of implementing the findings are carefully described in the evaluation report. d. stakeholders have had input into the recommendations.

17. Examining reports of completed evaluations in a systematic, quantitative manner a. is useful only to keep the evaluator from making gross errors in his/her work. b. is probably a waste of time since each evaluation is unique. * c. can provide the program planners with invaluable ideas that have been effective. d. is not possible from a statistical point of view.

..

63

18. Program managers often look for a simple, definitive conclusion from an evaluation (e.g. good/bad), but many evaluators prefer to deal with the findings as working hypotheses on which to base a. funding decisions. b. research articles. c. staff salary increases. * d. program improvement efforts.

19. The most important contribution that an evaluator can make to a program is to * a. encourage the development of a learning culture among the program staff and organization management. b. get staff to care about the participants of the program. c. perform an extremely rigorous evaluation that would draw attention to the organization sponsoring the program. d. be sure that the staff respects the credentials of the evaluator.

20. It is valid to label a program as ineffective when a. a statistical comparison between program and comparison groups is non-significant. * b. statistical comparisons between randomly-assigned program participants and a control group are non-significant and statistical power is 95%. c. the program lacks common sense plausibility but the statistical test is significant (i.e. p < .05). d. statistical comparisons between randomly formed program and control groups are nonsignificant and statistical power is less than 5%.

21. Internal evaluators have an advantage over external evaluators in that internal evaluators a. are often friends of the program staff or administrators. b. do not have a long-term commitment to the agency. * c. can repeatedly remind program managers of the evaluation and its findings. d. have a perspective gained by working in many different agencies.

Essay Questions 22. What are the steps that evaluators can take on their own to reduce the fear of evaluation?

23. What are the steps that evaluators can take to encourage others to take to reduce the fear of evaluation?

24. Although some anxiety about evaluation is normal, a high level of evaluation apprehension can lead to ineffective reactions to evaluations that are not totally favorable. Describe some of the fears and show how they lead to problems between evaluators and staff members.

25. Describe how “the evaluator’s perspective” differs from that of the staff and program managers.

..

64