Visit to download the full and correct content document: https://ebookmass.com/product/program-evaluation-an-introduction-to-an-evidence-b ased-approach-6th-edition/
More products digital (pdf, epub, mobi) instant download maybe you interests ...
Children’s Speech: An Evidence-Based Approach to Assessment and Intervention
https://ebookmass.com/product/childrens-speech-an-evidence-basedapproach-to-assessment-and-intervention/
Evidence-Based Statistics: An Introduction to the Evidential Approach - From Likelihood Principle to Statistical Practice Peter M. B. Cahusac
https://ebookmass.com/product/evidence-based-statistics-anintroduction-to-the-evidential-approach-from-likelihoodprinciple-to-statistical-practice-peter-m-b-cahusac/
Protocols for High-Risk Pregnancies: An Evidence-Based Approach 6th Edition – Ebook PDF Version
https://ebookmass.com/product/protocols-for-high-riskpregnancies-an-evidence-based-approach-6th-edition-ebook-pdfversion/
Program Evaluation for Social Workers: Foundations of Evidence-Based Programs Richard M. Grinnell Jr
https://ebookmass.com/product/program-evaluation-for-socialworkers-foundations-of-evidence-based-programs-richard-mgrinnell-jr/
Forest Ecology: An Evidence-Based Approach Dan Binkley
https://ebookmass.com/product/forest-ecology-an-evidence-basedapproach-dan-binkley/
Musculoskeletal physical examination : an evidencebased approach Second Edition. Edition Malanga
https://ebookmass.com/product/musculoskeletal-physicalexamination-an-evidence-based-approach-second-edition-editionmalanga/
Symptom to Diagnosis An Evidence Based Guide, Fourth Edition
https://ebookmass.com/product/symptom-to-diagnosis-an-evidencebased-guide-fourth-edition/
Program Evaluation for Social Workers: Foundations of Evidence Based Programs 7th Edition, (Ebook PDF)
https://ebookmass.com/product/program-evaluation-for-socialworkers-foundations-of-evidence-based-programs-7th-edition-ebookpdf/
Orthopaedic and Trauma Nursing: An Evidence-based Approach to Musculoskeletal Care 2nd Edition Sonya Clarke
https://ebookmass.com/product/orthopaedic-and-trauma-nursing-anevidence-based-approach-to-musculoskeletal-care-2nd-editionsonya-clarke/
Chapter3NeedsAssessment63
TypesofNeedsAssessment65
HowDoesaNeedsAssessmentStart?68
WhatAretheStepsInvolvedinConductingaNeeds Assessment?70
HowDoISelectaNeedsAssessmentApproach?71 WhatDoINeedToKnowAboutSurveys?78
SpecialConsiderations:ConvergingtheData81 ThinkingCreativelyAboutNeedsAssessment83 WhatIsaCommunityReadinessStudy?84
Chapter4QualitativeandMixedMethods inEvaluation93
Introduction93
WhatIs “QualitativeEvaluation”?94 WhatIs “MixedMethodsEvaluation”?94
WhenIsQualitativeEvaluationUseful?WhenIsMixed MethodsEvaluationtheRightChoice?95
QualitativeMethodsandEvaluationofPractitioners InSitu98
QualitativeMethodsCommonlyUsedinEvaluation Research99
QualitativeEvaluation99 DesigningaMixedMethodsEvaluation100 ManagingandOrganizingQualitativeData104 DataAnalysis105 MixedMethodsDataAnalysisandIntegration107
QualityControl108
WritingtheReport110
DisseminationandAdvocacy111 ExamplesofQualitativeEvaluations113
Chapter5WhatAreFormativeandProcess Evaluation?121
HowDoIDevelopaLogicModel?122 ARealisticScenario125
FormativeEvaluation125
ConductingaFormativeEvaluation126
WhatIsProcessEvaluation?129
ProcessEvaluation:ProgramDescriptionFunction131
ProcessEvaluation:ProgramMonitoring136
BecomingaProgramMonitor136
MissionStatements,Goals,andObjectives138
WritingProgramObjectives141 WhatShouldBeMonitored?143
QualityAssurance146
TotalQualityManagement148
ChapterRecap151
Chapter6SingleSystemResearchDesigns155
WhatAreSingleSystemResearchDesigns?155
SelectingOutcomeMeasures158 AssessingMeasuresOverTime160 NotationandGeneralPrinciples161
NeedsAssessments163
FormativeEvaluations165
QualityAssuranceStudies167
SummativeEvaluationDesigns169
SummativeExperimentalDesigns173
ExternalValidity180
InferentialStatistics181
HowtoPrepareGraphs183
EthicsofSingleSystemResearchDesigns185 HowtoCriticallyReviewaSSRD186
ChapterRecap187
Chapter7ClientSatisfaction193
TheImportanceofMonitoringConsumerSatisfaction193 TheArgumentsforClientSatisfactionStudies195 TheProblemwithClientSatisfactionStudies196
ASamplingofRecentClientSatisfactionStudies199 HowDoConsistentlyHighRatingsGetExplained?200 RecommendationsforClientSatisfactionStudies202
Chapter8Sampling211
WhatAreNonprobabilitySamplingDesigns?212 WhatIsProbability(Scientific)Sampling?216 ConsiderationsinSelectingaSample219 HowBigShouldtheSampleBe?221
ChapterRecap224
Chapter9GroupResearchDesigns227
WhatAreGroupResearchDesigns?227 StartinganOutcomeEvaluation228 OutcomeEvaluationDesigns231
GeneralPrinciplesofGroupResearchDesigns234 Pre-ExperimentalResearchDesigns235 Quasi-ExperimentalResearchDesigns246 SomeThreatstoInternalValidity252 ProtectionAgainstAlternativeExplanations257 ExperimentalDesigns258 EfficacyandEffectivenessStudies266 WhatAboutNegativeOutcomeStudies?268 ANoteAbouttheTermExperiment271 SomeNewerDevelopmentsinExperimentalProgram Evaluation272
ChapterRecap274
Chapter10Cost-EffectivenessandCostAnalysis283
WhyConsiderCostasanEvaluativeCriterion?283 ExampleofaCost-EffectivenessEvaluation286 HowDoIBeginaCost-EffectivenessStudy?287 WhosePointofView?290
Cost-BenefitAnalysis292
ChapterRecap296
Chapter11MeasurementToolsandStrategies301
WhyIsGoodMeasurementsoImportanttoEvaluation?301 WhatShouldWeMeasure?303
Reliability306
WhatDoINeedtoKnowAboutReliability?307 WhatDoINeedtoKnowAboutValidity?312 HowDoIFindanInstrumentforMyEvaluation Project?315
HowDoesOneConstructa “Good” Evaluation Instrument?WhatDoINeedtoKnowAbout QuestionnaireDesign?317 WhatAreSomeoftheCommonsErrorsMadein DevelopingQuestionnaires?320 WhatDoINeedtoKnowAboutLevelsofMeasurementin DesigningInstruments?327
Chapter12SelectingtheBestEvaluationMeasurefor YourProject331
ChecklistforSelectingEvaluationInstruments332 ClinicalAnxietyScale334 EvaluationSelf-EfficacyScale336 TheCommunityAttitudesTowardSexOffenders(CATSO) Scale338
IntimateViolenceResponsibilityScale(IVRS)340 TheSchoolSupportScale343 CulturalSensitivity346
Chapter13PragmaticIssues351 WhatIsTreatmentFidelityandWhyIsItsoImportant? 351 FidelityNightmares356 ProgramDriftandtheTransferofPrograms359 HowMightPoliticalIssuesAffectaProgramEvaluation? 360 The “Threat” ofEvaluation362 GuidelinesforEvaluationinPoliticallyCharged Arenas363 InternalandExternalEvaluators367
CulturallySensitiveEvaluationPractice370 FinalThoughts374
Chapter14WritingEvaluationProposals,Reports, andJournalArticles379
ComponentsoftheEvaluationProposalandReport380 ConsiderationsinPlanningandWritingEvaluationReports andManuscripts391
CommonMistakesMadebyStudentsinWriting EvaluationReports392
ChecklistforWritingandAssessingEvaluation Reports395
TheUtilizationofEvaluationReports397 WritingforProfessionalPublication398 Index403
Preface
Atextbookmustprovide,firstandforemost,informationtoassistthereaderinbetterunderstandingthetopic.Theinformationmustbecredibleandeasilyaccessed anddigested.Readersofthiseditionwillfindnumerousexamplesfromtheliterature toillustratekeypointsandconcepts.Youwilldiscoveranewemphasisonevidencebasedpracticeandwecontinuallystresstheimportanceoflocatingevidenceon theeffectivenessofprogramsandpolicies.Webelievethatprogramevaluators andgraduatestudentslearninghowtoevaluatemustbeacquaintedwithsalient andcontemporaryevaluationsofprogramssimilartothosetheyareinterested in.Thiscanprovideusefulinformationoneffectiveandineffectiveprogramsand practices,aswellasexamplesofthetypesofdesignsusedtoevaluatesimilarprogramswiththelatterprovidingguidanceusefulforcraftingone’sownprogram evaluation.
Textbooksthathavegonethroughmultipleeditionscontinuetoimproveasa resultofreviewers’ commentsandreaders’ feedback,andthisoneisnoexception. Lookingbackovertheeffortsassociatedwiththesixthedition,wehavebuiltupon thesolidfoundationofpreviouseditions,butaddednewmaterialaswell.
Sometimesitishardtoknowhowmuchto “redo” andhowmuchtoleave alone.Salientandclassicstudiesareimportant,buttherearealwaysnewideas andrecentstudiestohighlight.Somecontentmustalwayschangetokeepatext current.However,educatorschoosebookstheyhaveusedinthepastbecause theyknowandlikethatcontent.Ifabookchangestoomuch,thenitmaynot alwayshavethesamecomfortablefitaswhenitwaspreviouslychosen.Itissomewhatofachallenge,then,toknowwhattokeepandwhattotoss.Andthatis why,dearreader,youareinvitedtoletusknowwhatworksandwhatdoesn’t.If thereissomeaspectofprogramevaluationthatwedon’texplainverywell,some chapterthatdoesn’tworkforyou,thensendanemailandtelluswhatyouthink.
Forthoseofyouwhoarejustdiscoveringthisbook, ProgramEvaluation isdesignedtobeaprimarytextbookforgraduatestudentsinsocialwork,sociology,psychology,publicadministration,counseling,education,nursing,andother relateddisciplineswhereitisimportanttoknowifinterventions,betheyindividual,family,group,orcommunityfocused,arehavingthedesiredimpact.We hopethatwehavewrittenclearlyenoughthatpractitioners,administrators,and
otherpersonswhohavejustacquiredtheresponsibilityforoverseeingorevaluating specificprogramsorpoliciesmayfindittobeavaluableresourceinplanningevaluationorneedassessmentprojects.
Ourfocushasbeenoncommunicatingtheessentials thatis,thebasictools andknowledgenecessarytoconceptualizeaprogramevaluationandcarryoutthe tasksassociatedwithexaminingandappraisingprogramperformance.Thisbookis designedtohelpstudentsandpractitionersunderstand and contributetoevidencebasedpractice.Itisvitallyimportantthatweprofessionalsandprofessionals-intrainingcontinuallyexaminetheeffectivenessofourprogramsandtheirimpact uponourclients.Thisisthewayimprovementcomesabout withbenefitsfor bothclientsandsociety.Wehopeinsomesmallwaythatourbookisabletoprepareandassistyouforthisimportantresponsibility.
DavidRoyse (droyse@uky.edu)
BruceThyer (bthyer@fsu.edu)
DeborahPadgett (deborah.padgett@nyu.edu)
CHAPTER 1 Introduction
Whyisitimportantthatpractitionersbefamiliarwith, andinsomecircumstances,beinvolvedin,conducting programevaluationstudies?
Itisthroughprogramevaluationsthatservicescanbecrediblyshown tobehelpful,ineffective,orharmful.Itisthehallmarkofagenuine professionthatconcernbegiventotheevidence-baseofitsservices. Allofthehumanservicefieldssupporttheimportantofprogramevaluationandcommendstheirmemberstotakeintoaccountcurrent credibleoutcomestudieswhenarrivingatpracticeandpolicy decisions.
WhatskillswillIneed?
Allhumanserviceprofessionalsneedtobeabletoread,understand andcriticallyevaluateprogramevaluationstudies,anddecide whethertoapplysuchfindingstotheirownsettings.Somepractitioners willfinditnecessarytohelpinthedesign,conductandinterpretationof programevaluations,assuggestedbycontemporaryethicalcodes.
THEIMPORTANCEOFPROGRAMEVALUATION
Welcometothefieldofprogramevaluation,thataspectofprofessionaltraining aimedathelpingyoutointegrateresearchandpracticeskills,usingtheformerto enhancethelatter.Werecognizethatrelativelyfewpractitionersinthehumanservicesofsocialwork,psychology,counseling,education,psychiatry,publichealth, publicadministration,andnursingwillhavecareersprimarilydevotedtoscientific research.However,everyprofessionalinthesefieldswillbeconcernedwithproviding servicestoclientsatsomelevel(individuals,families,couples,smallgroups,organizations,orcommunities),andeveryhumanservicespractitionerwillbeintenselyinterestedinlearningwhethertheservicesprovidedreallydohelptheclientstheyare
intendedtoserve.Thus,evenifresearchmaynotbeyourmajorprofessionalactivity, learningaboutgenuinelyeffectiveprograms,andevaluatingservicesandprograms usingscientificallycredibleresearchtoolsisaprofessionalskillthatyouwillfind valuable.Andapartfrompossiblybeinginvolvedinthedesignandconductofprogramevaluationsyourself,youwillreadandcriticallyanalyzeevaluationstudiesconductedbyothersandpublishedinvariousdisciplinaryandinterdisciplinaryjournals, books,andwebsites.Yourabilitytojudgethevalueofpublishedandunpublished programevaluationsisavaluableskilltoacquire.Otherwise,howwillyoubeable todecidewhattypesofservicesareworthprovidingtoclients,ateitherindividualor programmaticlevels?
Inthehumanservicesfields,scientificresearchcanbebroadlyclassifiedinto havingthreemain(ofteninterrelated)purposes:toobjectivelydescribethings;to empiricallyevaluatetheeffectivenessofservices;ortovalidlyexplainthings. Descriptiveresearchcanbeundertakentobetterunderstandthecharacteristicsor needsofclientsofaparticularagency.Evaluativeresearchhelpsdeterminewhether theseneedsarebeingmetorclients’ goalsattained,whileexplanatorystudiesaimat uncoveringthecausesofpsychosocialproblemsortheprocessesbywhichinterventionswork.Thislatterendeavormaycontributetowhatisknownrespectivelyas etiologicalorinterventivetheory.
Programevaluationcanbeseenasasubsetofthoseactivitieslabeledresearch, whichitselfhasbeensimplydefinedas “systematicproceduresusedinseekingfacts orprinciples” (Barker,2014,p.365).AmoreexpansivedefinitionofresearchisdisplayedinTable1.1,providedbytheAmerican EducationalResearchAssociation.Itis worthemphasizingthepointsfoundinthisdefinition.Researchmustberigorous,systematicandseeksobjectiveinformation.Agenuinelyscientificstudycannotbeconductedcasually,withoutcarefulplanning,andmustincludeadedicationtofindout whatactuallyis,asopposedtoattemptingtoproveone’spreexistingexpectations.It mustbelogical,andaverywiderangeofmethodscanbeemployed,methodscovered inthebalanceofthisbook.Theterm instruments inevaluationresearch,usuallyrefers topencilandpaperscalescompletedbyclientsorotherstakeholdersinaprogram (e.g.,caregivers,supervisors,teachers,etc.).Sometimes programevaluationinvolves thedirectmeasurementofobservablebehavior,inwhichcasesoundstudiesinclude appropriatemethodstoassessthereliabilityandvalidityoftheseobservationalmethods.Lessfrequently,programevaluationsmayincludetheuseofsomesortofphysical instrumentation,suchastheuseofbiofeedbackdevices,bloodpressureorheartrate monitors.Andoccasionallyprogramevaluationsassessclients’ physiologicalvariables suchasbloodsugar,cholesterollevels,ormeasuresof “stress.”
Manyprogramevaluationsinvolvetheanalysisofoutcomesusingvarious formsofdescriptiveandinferentialstatistics,andsuchmethodsmustmeetthe underlyingassumptionsofthesetestsandbecarriedoutcorrectly.Whengivena choiceofdifferentlegitimateapproaches,itiswisesttousethemoreconservative methodsofanalysis,asthisreducesthelikelihoodoffindingreliablebutpragmaticallyweakeffectsorresults.Researchreportsmustbeclearlywritten,soclearthat intheory,ifnotinfact,anindependentwell-trainedinvestigatorcouldreplicatethe essentialfeaturesofagivenprogramevaluation.Bydoingso,subsequentlyfinding similarpositiveresultsenhancestheconfidencethattheoriginalstudy’sconclusions werelegitimate.
Table 1.1 DefinitionofScientificallyBasedResearch*
I.Theterm principlesofscientificresearch meanstheuseofrigorous,systematicand objectivemethodologiestoobtainreliableandvalidknowledge.Specificallysuch researchrequires:
A.developmentofalogical,evidence-basedchainofreasoning;
B.methodsappropriatetothequestionsposed;
C.observationalorexperimentaldesignsandinstrumentsthatprovidereliableand generalizablefindings;
D.dataandanalysisadequatetosupportfindings;
E.explicationofproceduresandresultsclearlyandindetail,includingspecification ofthepopulationtowhichthefindingscanbegeneralized;
F.adherencetoprofessionalnormsofpeer-review;
G.disseminationoffindingstocontributetoscientificknowledge;and
H.accesstodataforreanalysis,replication,andtheopportunitytobuildon findings.
II.Theexaminationofcausalquestionsrequiresexperimentaldesignsusingrandom assessmentorquasi-experimentalorotherdesignsthatsubstantiallyreduceplausible competingexplanationsfortheobtainedresults.Theseinclude,butarenotlimitedto, longitudinaldesigns,casecontrolmethods,statisticalmatching,ortimeseriesanalysis.Thisstandardappliesespeciallytostudiesevaluatingtheimpactsofpoliciesand programsoneducationaloutcomes.
III.Theterm scientificallybasedresearch includesbasicresearch,appliedresearch,and evaluationresearchinwhichtherationale,design,andinterpretationsaredeveloped inaccordancewiththescientificprincipleslaidoutearlier …
*ObtainedfromtheAmericanEducationalResearchFoundationwebsite,foundat: http://www.aera.net/AboutAERA/Key Programs/EducationResearchandResearchPolicy/AERAOffersDefinitionofScientificallyBasedRes/tabid/10877/Default.aspx
Adefiningcharacteristicofscientificresearchisthatitispublishedincredible outlets,whichmakeuseofpeer-review.Thisinvolvessubmittingthepapertoa journalorotheroutlet,andhavingitscrutinizedbytwoormorecompetentpersons,whichprovidecriticalanalysisandarecommendationtorevise,reject,or acceptthepaper.Thisisusuallydoneblindly,withthereviewersnotknowingthe identitiesoraffiliationsoftheauthors,andtheauthorsnotknowingtheidentities ofthereviewers.Thisblindreviewisonewaytotryandreducebias,sothat reviewersarenotinfluencedbyanauthor’sexistingreputation,ortheprestigeof theauthors’ affiliations(e.g.,HarvardversusMidwestA&M).Alsothereviewers canprovidemorehonestfeedbackiftheyareassuredtheauthorswillnotknow whotheyare(andperhaps “retaliate” inthefuture).Onlyaftersuchrigorousvettingdoscientificpapersmakeitintotheprofessionalliterature,anditisthislayer ofpeer-review,whichlendsadditionalcredibilitytothesestudies.Professionaljournals(onlineorprint)remainamajoroutletforthedisseminationofscientific research,althoughincreasinglywebsitesareusedforthispurpose.
Inmanyinstances,programevaluationslookonlyatoutcomes,whichisavery legitimateundertaking.Otherapproachesattempttodetermineclient’sperspectives,
suchastheirsatisfactionwithservices,orhowrespectfullytheyweretreatedby agencystaff.Moreambitiously,someevaluationstudiesattempttodetermine whetherornotagiveninterventionactually caused anyobservedoutcomesordifferencesbetweengroups.Answeringcausalquestionscanbeanexceedinglydifficult undertakingandusuallyrequiresrelativelysophisticatedresearchdesigns.This bookwillcovertherangesofmethodsandresearchquestionsofinteresttoprogramevaluators.
Evaluationresearch refersto “systematicinvestigationtodeterminethesuccess ofaspecificprogram” (Barker,2014,p.148). Programevaluations areapractical endeavor,notanacademicexercise,andarenotprimarilyanattempttobuildtheoryornecessarilytodevelopsocialscienceknowledge(althoughitiswonderful whenthathappens).Tripodi(1987)hasnotedthat “themissionofprogramevaluationinsocialworkistoprovideinformationthatcanbeusedtoimprovesocial programs” (p.366).Similardefinitionscanbeprovidedacrossthehumanservice professions.Forexample,in TheDictionaryofPsychology,Corsini(2002,p.766) definesprogramevaluationas “… aprocesswhosepurposeitisto:(a)contribute todecisionsoninstalling,certifying,andmodifyingprograms;(b)obtainevidence tosupportoropposeaprogram;and(c)contributetobasicknowledge.” Program evaluationisdesignedtomakeadifferenceinthelivesofstakeholders,whichcan includeclients,serviceproviders,supervisor,administratorsandpolicy-makersat alllevels.Onecanevaluatetheeffectsofaprogramatthelevelofanindividual, oramongmanypeoplewhoreceivedagivenprogram,atsimilarlyconstitutedprogramsappliedinvarioussettings,andatthelevelofstatewide,nation,oreven internationalpolicies(e.g.,humantrafficking).
Curiously,althoughformanydecadesthesingularlycrucialimportanceof humanservicesprofessionalsdesigningandconductingprogramevaluationshas beenwidelyrecognized,programevaluationresearchremainsrarelyundertaken, relativetotheamountofliteraturepublished.ConsiderthisfactinlightofthequotationscontainedinBox1.1.
Thuswehavetheveryoddsituationthatalthoughmanyauthoritiesconsider programevaluationtobeperhapsthemostvaluabletypeofresearchcontribution onecanmake,suchstudiesseemtobeveryrarelyundertakenandpublished. Instead,mostempiricalstudieshaveafocusondescriptiveandexplanatory researchwork(Holosko,2010;LeCroy&Williams,2013;Rosen,Proctor& Staudt,1999),which,atbest,mayhavepotential implications forpracticeinthe humanservices,butnotthepotentialfordirect applications thatawell-craftedoutcomestudywouldpossess.
Whyisitnecessarytoevaluatebothnewandestablishedservices?Because therearealwaysalternative,andsometimesbetter,waystosolveproblems.Abusive alcoholuseisaseriousproblemonmanycollegecampusesanduniversitiesspenda gooddealofmoneyattemptingtopreventandreducealcoholabuseanditsassociatedproblems.AtreatmentknownasMotivationalInterviewing(MI)hasbeen widelyusedasatreatmentforalcoholabusers,andasapreventiveintervention oncollegecampuses.RecentlyWilke,Mennicke,Howell,andMagnuson(2014) evaluatedaMIpreventionprogramprovidedtomembersofsororitiesandfraternitiesatalargeuniversity.Arelativelystrongevaluationdesignwasused,whichrandomlyassignedfourGreekhousestoreceivetheMIprogram plus thestandard
BOX 1.1 OpinionsontheImportanceofEvaluationResearch
Iappealtoyou .Measure,evaluate,estimate,appraiseyourresults,insomeform,in anytermsthatrestuponsomethingbeyondfaith,assertion,and “illustrativecase.” Stateyourobjectivesandhowfaryouhavereachedthem….Outofsuchevaluations willcome,Ibelieve,betterservicetotheclient.(Cabot,1931)
Thethirdtypeofresearch,evaluativestudiesofwelfareprogramsandtheactivitiesof practitioners,arethemostimportantofall.(Angell,1954,p.169)
[S]ocialworkisnotasciencewhoseaimistoderiveknowledge;itisatechnology whoseaimistoapplyknowledgeforthepurposeofcontrol.Therefore,onthe researchcontinuumsocialworkresearchfallsnearertotheappliedend,becauseof itspurposeofpracticalknowledge.(Greenwood,1957,p.315)
Evaluationandclientfeedbackarenotonlynecessaryforeffectiveservicedelivery,but areanethicalrequirementoftheprofession.Systematicmethodsmustbedevelopedto assesswhethersocialworkersarehelping,harming,ordoingnothingforthepeople theyserve.(Rosenberg&Brody,1974,p.349)
Socialworkhasnomoreimportantuseofresearchmethodsthanassessmentofthe consequencesofpracticeandpolicychoices .[S]mallscale,agency-basedstudiesare worthwhileiftheysucceedinplacinginterestineffectivenessatthecenterofagency practiceandwhentheycreateacriticalalliancebetweenpractitionersandresearchers. (Mullen,1995,pp.282–283)
Studiesareneededontheeffectivenessofpsychosocialintervention,includinginterventionspreviouslytestedunderidealcontrolledconditions,inreal-worldhealthcare systems.(Ell,1996,p.589)
Researchonactualserviceinterventionsisthecriticalelementinconnectingresearch totheknowledgebaseusedbyprofessionalpractitioners .[T]heissuenowisoneof developinginvestigationsofsocialworkinterventioninitiatives,studiesthatgo beyonddescriptionsandexplanatoryresearch.(Austin,1998,pp.17,43)
Weneedtoestablisharesearchagendaforsocialwork….Andinterventionstudies mustbehighinprioritytosuchanagenda.(Rosen,Proctor,&Staudt,1999,p.9).
existingcampus-widealcoholabusepreventionefforts,andfourtoreceive just the standarduniversityservices.Measuresweretakenofstudentdrinkingandof alcohol-relatedconsequences,coveringthe30daysbeforetheprogramwasimplemented,andagain30daysafterward.Itwashypothesizedthatstudentsreceiving MIplususualcampusserviceswouldreducetheirdrinking,andexperiencefewer negativeconsequencesfromtheirdrinking,muchmorethanthestudentsreceiving usualservicesalone.Sadly,thisisnotwhatwasfound.Therewerenodifferences attributabletotheMIinterventionondrinkingoritsnegativeconsequences.There werenosignificantdifferencesbetweenthetwogroupsatthepretest,noratthe posttest,whenitwashypothesizedthatthepositiveeffectsofMIwouldbeevident. Now,ifyouwereauniversityadministratorchargedwithlocatingandimplementinganewcampus-widealcoholabusepreventionandinterventionprogramaimed atcollegestudents,wouldyouwanttoknowaboutthisarticle?Wouldtheresults
haveanybearingonyourchoiceofanewprogramtoadopt?Perhapsso,especially iftheMIlackedanycounter-balancingstudies,whichhadfoundthatit was effectiveatthepreventionandtreatmentofalcoholabuseoncampus.
Mentalhealthconcernsafflictasizeablenumberofindividuals.Aminorityof thesepeopleseekorreceiveprofessionaltreatment,inpartduetothelackof availablelicensedcliniciansavailabletoprovideempiricallysupportedtreatments.Domentalhealthclientsreallyrequiretreatmentfromdoctoral-levelpsychotherapists,orcouldlessintensivelytr ainedcliniciansalsoprovideeffective care?Thatwasthequestioninvesti gatedbyStanleyetal.(2014).Over200 olderadultswithGeneralizedAnxietyD isorder(GAD)wererandomlyassigned tothreeconditions:Treatment byadoctoral-levelpsychotherapists;treatmentby bachelors-leveltrainedtherapists;ort ousuallyavailablecare.MeasuresofGAD symptomsweretakenafterassignmenttotreatmentconditionsbutbeforetreatmentbegan,andatpretreatmentallgroupswereessentiallyequivalentlydisturbedandsimilarondemographicandothermeasures.Thebachelors-level therapistswereespeciallytrainedina treatmentknowntobehelpfulforpeople withGAD,somethingcalledcognitivebehavioraltherapy(CBT).ThesameCBT interventionwasprovidedbythedoctorallytrainedclinicians.Aftersixmonths, theclientswerereassessed.Itwasfound thatthedoctoralandbachelor-level therapistsobtainedsimilar lypositiveresultswiththeirclients,andbothtreatmentsoutcomeswerebetterthanamongt hosereceivingusualc are.Theseresults suggestthatmentalhealthagenciesmay beabletoprovideeffectiveservices deliveredvialower-cost(e.g.,bachelorslevel)clinicianscomparedtodoctorallevelpsychotherapists.Ifyouwerea managerofacommunitymentalhealth agencywantingtomaximizeyourbudget ,wouldtheresultsofthisstudybeof interesttoyou?
Policy-makersofallpoliticalpersuasionswishtoknowtheimpactofsocialpolicies.OnemajorpolicyintheUnitedStatesiscalledMedicaid,ahealthinsurance programintendedforrelativelypoorpeople.Statesarebeingencouragedtoexpand theirMedicaidprogramsandthefederalgovernmenthasbeenprovidingfinancial incentivestoinducethemtodoso.OneargumentmadeforMedicaidexpansionis thatitwillimprovepeoples’ health,andhavesomefinancialadvantagessuchas reducinghighlyexpensiveemergencyroom(ER)visitsforroutinecare.Poorpeople lackinghealthinsuranceoftenmakeinappropriateuseofERsbecausetheycan receivetreatmenttherewithoutnecessarilyhavingtopayforit.ProvidingMedicaid coveragetothepoor,ithasbeenargued,willgivethemtheinsurancecoverage neededtoobtainroutinecarefromaprimarycareprovider,suchasafamilymedicaldoctorornursepractitioner.DoesMedicaidactuallyreduceERuse?Thatwas thequestionposedbyTaubman,Allen,Wright,Baicker,andFinkelstein(2014)in theirstudyoftheOregonMedicaidexperiment.
In2008,Oregonwasprovidedadditionalfederalmoneytosupportexpanding theirMedicaidprogram.Notenoughmoneywasprovidedtocovereveryoneonthe Medicaidwaitinglist(about90,000people),soOregonusedanunbiasedlottery systemtorandomlyselectwhowouldgetMedicaid(about30,000),andwho wouldremainonthewaitinglist(about60,000).ERvisitstoPortland,Oregonhospitalswerereviewedover18monthsafterthisexpansionofMedicaid.Itwas
hypothesized(andexpectedbymoststakeholdersinvolved)thatERusewouldbe muchloweramongthepatientswhoreceivedaccesstoMedicaid.Therealitywas thatERusagewasabout40percent higher amongMedicaidrecipients,compared topeoplewhodidnotgetMedicaid.Theirconclusion?
WefindthatexpandingMedicaidcoverageincreasedemergencydepartmentuseacross abroadrangeofvisittypes,includingvisitsthatmaybemostreadilytreatableinother outpatientsettings.ThesefindingsspeaktoonecostofexpandingMedicaid … and maythusbeausefulinputforinformeddecision-makingbalancingthecostsandbenefitsofexpandingMedicaid.(Taubmanetal.,2014,p.4)
Nowimagineyouareagovernororlegislatorchargedwithtryingtodecide whetherornottoexpandMedicaidinyourstate.Asthisbookisbeingwritten,the Taubmanetal.(2014)studyisbeingusedinsomeconservativestatestojustify not expandingMedicaid,onthegroundsofincreasedcost.Thisillustratesthecomplextangleofmakingpoliticalandpolicydecisions.WecanbequitesurethatTaubmanetal.didnotundertaketheirstudyinordertoprovideammunitionfor politicianstodenyaccesstohealthinsurancetothepoor.Theywantedtosimply findoutwhattheresultswere,andtheseresultswouldhaveplayedacorresponding roleamongmoreprogressivepoliticianstojustifyexpandingMedicaid,had theresultsdemonstratedlowerERuse.Nooneisarguingthatempiricalresearch findingsbethesolecriterionforpolicy-making,butcurrentprogramandpolicyevaluations,especiallyverywell-doneones,shouldcertainlybeapartofthedecisionmakingprocess.
Therearemanyexamplesofevaluationswithpositiveresultsthatcanbeused todevelopeffectivepoliciesandprogramsbyhumanservicepractitionersseeking guidanceastowhattypesofinterventionsworkwell,orworkbetterthanothers. ResearchersintheUnitedKingdomandinSwedenundertookasystematicreview ofprogramevaluationsofhomerepairprogramsandtheirimpactsonresidents’ health.Thefocusofhomerepairsinthisreportinvolvedenhancinginsulation, improvinghomeheatingsystems,double-glazingwindows,anddraftabatement (chalking).Over39suchstudieswerefoundofvaryingquality,someofwhich wereofveryhighqualityindeed.Brieflystated,thermalenhancementstohomes wereassociatedwithdecreasedrespiratoryillnessesandenhancedgeneralhealth. Absenteeismfromworkandschoolwasreduced,andthermallyenhancedhomes hadmore “usablespace,” whichimprovedpsychosocialfunctioninginfamilies (e.g.,theydidnotneedtoclusterinoneroomwithaspaceheater)(seeThomson, Thomas,Sellstrom&Petticrew,2013).
Asanotherexample,agroupofScandinavianresearcherscompletedasystematic reviewofhigh-qualityevaluationstudiesexaminingtheeffectsofwelfare-to-work programs.Their122-pageanalysiswasgroundedinareviewof46programsinvolvingover412,000participantsworldwide,althoughmostsuchstudieswereconducted inNorthAmerica.Thesewelfare-to-workprograms “… resultsinaslightincreasein thelikelihoodthattheunemployedpersonwillfindemployment,thatearningswill increaseandthatwelfarepaymentswillbereduced.Theyalsoreducethelikelihood thattheunemployedpersonwillremainonlong-termwelfare” (see http://www.camp bellcollaboration.org/lib/project/18/).
Ifyouareworkinginthefieldofhelpingthepoorfindjobs,youwouldbe well-advisedtoobtainthissystematicreviewandtobecomefamiliarwiththose approachesshowntobemoreeffectivethanothers.Anothersystematicreviewon theCampbellCollaborationwebsitefoundthatcognitivebehavioraltreatmentis helpfultochildrensufferingfrompost-traumaticstressdisorder.Anotherreview supportedthepositiveeffectsofafter-schoolparentinvolvementatimprovingchildren ’sacademicperformance.ThewebsitesoftheCampbellCollaboration(www .campbellcollaboration.org)andtheCochraneCollaborations(www.cochrane .org)containnumeroussystematicreviews,whichrepresentsomeofthemost credibleresearchprojectseverundertakeninawidearrayofdomainsinthehumanservices,includingsocialwelfare,criminaljustice,education,health,andmedicine.These areexceptionalresourcesforprogramevaluatorstobefamiliarwith.
Webelievethatpreviouslypublishedandcredibleevaluationstudiesshouldbe amajorsourceofinputintothedesignofanagency’sprograms.Ourhopeis thatthroughreadingthisbook,youwillacquiregreaterskillsinlocatingandcriticallyevaluatingsuchstudies,andindesigningandconductingempiricalevaluationsofyourownpracticeandoftheoutcomesoftheagencywhereyoumaybe employed.
Whenweimproveourprogramsandinterventionsbymakingthemmoreeffectiveandefficient,allthoseinvolvedwithortouchedbythehumanservicedelivery systemareaffected.Consumersandtheirfamiliesmayrecoverfasterwhenwediscoverthatoneapproachworksbetterthananother.Armedwithinformationfrom theprogramevaluation,workersandmanagerscanbettertreatandadvocatefor theirclients possiblymakingtheirownjobsmoreenjoyableandlessfrustrating. Ultimately,eventaxpayersbenefit.Butletusbackupabitanddiscusswhatconstitutesaprogram.
WHATISAPROGRAM?
Aprogramisanorganizedcollectionofactivitiesdesignedtoreachcertainobjectives.Let’sconsiderthetwomainelementsofthisdefinitionindepth.Organized activities programs arenotarandomsetofactionsbutaseriesofplanned actionsdesignedtosolvesomeproblem.Ifthereisnoproblem,thenthereisno needforprogrammaticintervention.So,programsareinterventionsorservices thatareexpectedtohavesomekindofanimpactontheprogramparticipants. Barkerdefinesaprogramasa “Relativelypermanentorganizationandprocedure designedtomeetongoingclientneeds(asopposedtoa ‘project,’ whichismoreflexibleandshortterminscope” (2014,p.338).Thisbookincludesevaluationsaimed atarangeofvenues long-termprograms,time-limitedprojects,one-timegroup researchdesigns,andoftheeffectsofprograms(includingindividualservices)on particularclients.Wecoverbothquantitative,qualitative,andmixedmethod approachestoprogramevaluation,aswellasotherimportantformsofevaluations suchasneedsassessments,clientsatisfactionstudies,formativeandprocessevaluations,andcosteffectivenessandcostanalysisinvestigations.Eachisimportantand encompassedwithinthebroadscopeofevaluationwork.Tobegin,wewilltakea lookatwhataresomeofthefeaturesofgoodprograms.
CHARACTERISTICSOF “GOOD” PROGRAMS
Programstendtohavecertaincharacteristicsthathelpusidentifythem.Firstofall, programstendtorequire staffing.Aresidentialdrugtreatmentfacility,forinstance, isgoingtoneedalotofstaff.Itmayevenhaveaseparatestaffwhorunanaftercareoroutpatientdrugtreatmentprogram.Thepersonnelofbothprogramsmay occasionallybeaskedtospeaktohighschoolstudentsandgroupsinthecommunityaspartofthefacility’sdrugeducationprogram.Staffmayhavetheirtimeallocatedamongseveralprogramsordedicatedtoonlyone.
Second,programsusuallyhavetheirown budgets.Becauseemployingstaff requiresfinancialresources,programssometimescanbeidentifiedbytheirbudgets. However,somefineprogramshaveminimalbudgetsbecauseofheavyrelianceon volunteers. Stablefunding isimportanttothesuccessofmostprograms.Morale andperformancefallwhenemployeesdonotgetpaidonaregularbasis,orwhen theyareaskedtoputasidenormaldutiesandengageinlastminutefund-raisingor grantwritingtogettheprogramthroughseveralmoremonths.Programsstarted with “softmoney” (grantsornonrecurringfunds)oftenexperiencehighratesof staffturnoveruntiltheprogramssecuresomecontinuityinfunding.
Anothercharacteristicofprogramsisthattheyhavetheirown identity.In short,theyarevisibleorrecognizablebythepublic.BigBrothersBigSisters (BBBS)isanexampleofanorganizationwithanationalreputationforasingleprogram.OthersyoumayrecognizeincludeHabitatforHumanity,AlcoholicsAnonymous,ortheNationalAlliancefortheMentallyIll.Insomecommunities,a programmayberecognizedbythelocationwhereithasbeenhousedforanumber ofyears,orbyitsuniqueslogan,sign,letterhead,spokesperson,orpublicservice announcements.
Whenanorganizationhasmultipleprograms,differencesaresometimesfoundin philosophies,policiesorprocedures,andmissions,andperhapseveninthewaytheir correspondingstaffsdressandhowtheyaccountfortheirtime.Suchcontrastsmake iteasytodifferentiateoneprogramfromanother.Withinanagency,oneoutpatient counselingprogrammayhavethe servicephilosophy that “nooneisturnedaway,” whileanotheroutpatientcounselingprogrammayhaveadifferentphilosophy providingserviceonlyforthosewhomeetcertaineligibilityguidelines,suchashaving privateinsuranceorbeingabletoaffordtopay.Aservicephilosophymayalso clearlycommunicatehowtheclienteleistobetreated,forexample, “Werespect thedignityandworthofallthoseweserveincaringfortheirphysical,spiritual, psychological,andsocialwell-being ” or “ Thecustomerisalwaysright.”
Unfortunatelyforprogramevaluators,programscanbevagueandhardtodistinguishanddefine.Aformergovernoroncemadeapublicannouncementthathe wasunveiling “anewprogram” toputstatesocialworkersinpublicschools.The program,hesaid,shouldhelppreventdropoutsandpoorachievementamongstudentswhofacedseriouspersonalandfamilyproblems.However,thenewspaper accountsaidtheprogramwouldrequirenoadditionalstafforfunds.Inessence, somesocialservicesemployeeswouldbeplacedinschoolsthatcouldsupplythem withofficespaceandphone.
Didthegovernor’sannouncementcreateaprogram?Notinthisinstance.It nevergotofftheground.Whynot?Ithadnoname,nostaff,nofunds,noslogan,
novisibility.Mostschoolsdidnothavesurplusofficespace.Further,thegovernor madenosuggestionofanynewactivitiesorwaysoftacklingtheproblemschildren andtheirfamiliesfaced.
Ontheotherhand,startingabereavementsupportgroupinanelementary school,evenifvolunteerscontributetheleadershipandthegrouphasnobudget tospeakof,couldbeconsideredaprogramifithasanongoingpresenceandapresumedimpactthatcouldbemeasured.Forevaluationpurposes,speakingtoan assemblyofhighschoolstudentsonceortwiceayearaboutdrugsandalcoholism mightalsobeconsideredaprogram.
Inthebestofallpossibleworlds,everyhumanservicesprogramwouldbesolidly establishedonthebasisofscientificallycredibleevidencethathadbeenpreviously publishedinpeer-reviewedprofessionaljournals.Thatis,beforethepractitioners jumpedintoasocialproblemandstarted “helping,” someonedidaserioussearch andappraisaloftherelevantevaluationstudiesthattestedtheusefulnessofvarious methodsofpotentialhelping.Ifacarefulsearchoftheliteratureandcriticalreviewof theexistingoutcomestudiesfoundthatoneormoremodelsofinterventionhadcredibleevidenceofeffectiveness,andtheseapproacheswere “teachable” totheexisting serviceproviders,costeffective,andethical,contemporarystandardsofethicalpracticewouldsuggestthattheserviceprogrambefocusedaroundtheseempiricallysupportedservicesasopposedtointerventionslackingasufficientfoundationin empiricalresearch.
Now,thisposesadilemmaforpractitionersandadministrators,namely, “What ifnoempiricallybasedservicesareknowntoexistforaparticularproblem?” Inthat case,onewouldbejustifiedinprimarilyrelyingonthemoretraditionalsourcesof practiceknowledge,namely, theory, practicewisdom, commonsense , tradition, and authority .But(andthisisabigbut),practitionersshouldonlyclaimthat noempiricallybasedinterventionsexistafterhavingmadeathoroughandupto-datesearchoftherelevantpracticeresearchliterature.Fortunately,researchbasedinterventionsarenowwellestablishedforamajorityoftheseriousconditionsdescribedinthe DiagnosticandStatisticalManualofMentalDisorders (DSM;AmericanPsychiatricAssociation,2013),andincreasinglyforconditions thatdonotlendthemselvestotheDSMsystem problemssuchasunemployment,domesticviolence,childabuseand neglect,andtroubledyouth.Webelieve thateveryhumanservicesagencyshouldkeepabreastofthesedevelopmentsby subscribingtorelevantjournals(e.g., JournalofConsultingandClinicalPsychology , ArchivesofGeneralPsychiatry, ResearchonSocialWorkPractice )and acquiringthelatesteditionofprofessionalbooksthatsummarizethelateststudiesonresearch-supportedpractices.
Humanservicesprofessionaltryingtolocatecredibleinformationontheevidentiarystatusofvariousprograms,policies,andserviceshaveawidearrayof websitestheycanturnto.Table1.2listsafewofthese.
Nowofcoursebecauseaprogramhasbeenshowntobeeffectiveinthepast, perhapsinvolvingclients,therapists,andagencysettingsdifferentfromyourown, thereisnoguaranteethatsuchaprogram,ifimplementedinyouragencyorstate, willyieldsimilarlypositiveresults.Inphilosophythisisknownastheproblemof induction.Ifyousawonenewtypeofbirdanditiswhite,andanother,also white,andanother,youmaybetemptedtoconcludethatallthesebirdsarewhite.
Table 1.2 SelectedWebsitesContainingInformationonResearchSupportedPsychotherapiesandPsychosocialPrograms andPolicies
TheCochraneCollaboration(focusesonhealthcare,broadlydefined) www.cochrane.org
TheCampbellCollaboration(focusesonsocialwelfare,criminaljustice,education,and socialdevelopment) www.campbellcollaboration.org
Research-SupportedPsychologicalTreatments(focusesonmentalhealthdisorders) http://www.div12.org/PsychologicalTreatments/index.html
SAMHSA’sNationalRegistryonEvidence-BasedPracticesandPrograms http://www.nrepp.samhsa.gov/
CaliforniaEvidence-BasedClearingHouseforChildWelfare http://www.cebc4cw.org/
NationalInstituteforHealthandCareExcellence(focusedonHealthandMentalHealth) http://www.nice.org.uk/GuidanceMenu/Conditions-and-diseases AgencyforHealthcareResearchandQuality Evidence-BasedReports(focuseson Health)
http://www.ahrq.gov/research/findings/evidence-based-reports/overview/index.html
TherapyAdvisor Consumerorientedwebsiteonresearchsupportedpsychotherapies http://therapyadvisor.com/defaultSub.aspx
CoalitionforEvidence-BasedPolicy SocialProgramsThatWork http://evidencebasedprograms.org/
WhatWorksClearinghouse FocusesonEducationalInterventions http://evidencebasedprograms.org/
Thatiscommonsense.Butthepossibilityremainsthatthenextbirdofthesame typemaybeadifferentcolor.Inprogramevaluation,ifaprogramhasbeenevaluatedelsewhereandfoundtobehelpfulwithclientswithaparticularproblem,there isnoguaranteethatitwillhelpsimilarclientswiththatprobleminyouragency. Butifithasbeenrepeatedlyshowntobehelpfulwithdifferenttypesofclients,in differingagencies,withvariousserviceproviders,andsuchstudieshavebeenconductedbyindependentinvestigators,andthereislittletonocounterfactualevidence tobefound(nostudieswithnegativeresults),thentherationaleforchoosingthat interventionorprogramincreases.
Practitionersandpolicy-makers must decideonwhatservicestoprovide.Choosingfromamongprogramswithastrongerresearch-baseenhancesthelikelihoodthat yourclientswillbebenefitted.Itdoesnotguaranteesuccess,butitisasurerguide thanrelyingonalternativesourcesofinformationsuchassomeoneinauthoritytellingyouwhattodo,orrelyingsolelyonclinicalintuition,theory,orprayer.Ourperspectiveinthisbookisbyallmeansmakeuseofmultiplesourcesofinformationin decidinguponwhatservicestoprovide,butforheaven’ssakemakesureyouconsult
therecent,credible,scientificliteraturetofindoutwhatisalreadyknownabouteffective(andequallyimportantly,ineffectiveorharmful)services.
Thequestionmaylegitimatelyarise, “Howmuchevidenceisenoughevidence inorderforagiveninterventiontobeconsideredtohaveanadequateempirical foundation?” The TaskForceonPromotionandDisseminationofPsychological Procedures (Chamblessetal.,1996,p.16)ofDivision12(ClinicalPsychology)of theAmericanPsychologicalAssociationhaspromulgatedonesetofminimalrecommendationstoaddressthisquestion.Inorderforapsychosocialinterventiontobe consideredforinclusionintheirlistofempiricallysupportedtreatments,ithadto meetthefollowingcriteria:
TheTreatmentMustBeSupportedBy
1. Atleasttwogoodbetween-groupdesignexperimentsdemonstratingefficacy inoneormoreofthefollowingways:
a. Superiortopillorpsychologicalplaceboortoanothertreatment
b. Equivalenttoanalreadyestablishedtreatmentinexperimentswith adequatestatisticalpower,or
2. Alargeseriesofsinglecasedesigns(N 4 9)demonstratingefficacy.These experimentsmusthave:
a. Usedgoodexperimentaldesigns,and
b. Comparedtheinterventiontoanothertreatment,asin1a
Amongtheothercriteriatobeappliedarethatexperimentsmustbeconducted usingtreatmentmanuals(thisenhancestheabilitytoreplicateinterventions),the characteristicsoftheclientsmustbeclearlyspecified,andeffectsmusthavebeendemonstratedbyatleasttwodifferentinvestigatorsortreatmentteams.Althoughthese standardsmayseemabitstringenttostudentsandpractitionersunaccustomedtorigorousresearch,theyarenotunwarranted;andtheyserveasaninitialstartingplaceto beginclassifyingparticularinterventionsasevidence-basedornot.Humanservices professionalscanemploythissetofstandardswhenselectingtypesoftreatmentprogramstoprovide.Overtime,itislikelythatthesestandardswillbeaugmentedby additionalcriteria(e.g., “Thetreatmenthasbeenevaluatedinreal-lifeclinicalsettings”)thatwillenhancetheirusefulness.Listsofpsychosocialtherapieswhichmeet thesestandardscanbefoundatthewebsiteon ResearchSupportedPsychological Treatments http://www.div12.org/PsychologicalTreatments/index.html.
Psychosocialtreatments,programs,orpolicies,whichenjoyasubstantialdegree ofempiricalsupportarevariouslyreferredtoasempiricallysupportedorresearchsupportedtreatments.Whattheyshould not bereferredtoasevidence-basedpractices.Thefollowingsectionwillexplainwhy.
EVIDENCE-BASEDPRACTICE
Thehumanservicesprofessionsareslowlymovinginthedirectionofevidencebasedpractice(EBP),whichhasbeendefinedas “theintegrationofthebest researchevidencewithourclinicalexpertiseandourpatient’suniquevaluesand circumstances” (Straus,Glazious,Richardson,&Haynes,2011,p.1).Although thisstatementmayseemlikecommonsense,thefactisthatatpresentnoclear
ethicalorlegalmandatesrequirethathumanservicesprofessionalsdeliverresearchsupportedinterventions,wherethesepracticesareknowntobeestablished.This couldchange,however,inthenearfuture.Clearly,itisthedirectioninwhichthe humanservicesprofessionsareheading,andisbeingstronglyencouragedby nationalgovernmentsintheUnitedStates,Canada,theUnitedKingdom,and othercountries,andadoptedbyindividualstatesintheUnitedStates.ThedefinitionofEBPquotedearliercomesfromthefourtheditionoftheseminalbook Evidence-BasedMedicine:HowtoPracticeandTeachIt. AlthoughEBPoriginated inmedicine,itssimpletenetshavebeenwidelyadoptedwithinmanydisciplines, includingpsychology,nursing,publichealth,counseling,andpsychiatry.EBPoutlinesabasicfive-stepprocesspractitionerscanundertaketohelpthemandtheirclientstodecidewhattodo.Thesestepsare:
1. Converttheneedforinformationintoananswerablequestion.
2. Trackdownthebestevidencewithwhichtoanswerthatquestion.
3. Criticallyappraisethatevidenceforitsvalidity,impact,andapplicabilityto one’ssituation.
4. Integratethiscriticalappraisalwithone’sclinicalexpertise,andwiththe patient’suniquevalues,circumstances,andpreferences.
5. Evaluateone’seffectivenessandefficiencyincarryingouttheprecedingfour steps.
TheEBPmodelassumesthatonefollowsappropriateprofessionalethicalstandards.Whileoriginallyintendedtoguidephysicians,thegenericnatureofthese fivestepswerequicklyadoptedbyotherfields,andtherearenowlargeliteratures dealingwiththeEBPpracticemodelinvarioushealthandhumanservicesfields. NotethatEBPis not amedicalmodel.Thereisnoassertionthatclientproblems havetheiroriginsintheclient’sbiology,noassertionthatinterventionsmustbe medical,noranycontentionthattheclinicianmustbeaphysician(thethreecentral principlesofthemedicalmodel).Yes,EBPoriginatedinmedicine,butitisnomore amedicalmodelthantheinferentialstatisticsdevelopedinagricultureintheearly partofthe20thcenturyimplyingthatallthesocialandbehavioralscientistswho usedthemlimitstheiruseonlytofarmers!
Step2,trackingdowntheavailableevidencecanbeconductedbyanindividual practitionerorprogrammanagerhim-orherself,searchingtheliterature,readingall ofit,critiquingit,oronecanlocateusefulsummariescalled systematicreviews (SRs) devotedtoyouranswerablequestion,andtakeadvantageofcompetentpeople alreadyhavingdonethispreliminarylegwork.IfyoucanfindapertinentSR,thisis agreatsavingsoftimeandeffortforyou,andTable1.2listssomesourceswhereSRs maybefound,aswellascitingandlistingresearch-supportedinterventions.However aSRdoesnottellonewhattodo,whatservicetoprovide,ornottoprovide.ASR simplysummarizestheexistingevidence,andpermitsthereadertomakeuseofitor not.Forexample,ifaclientneededabloodtransfusionandherphysicianinformed herofthis,asaresearch-supportedtreatmentforhercondition,theclientmayrefuse, perhapsonreligiousgrounds.AccordingtothemodelofEBP,thepractitioner mustrespecttheclient’svalues,andnotsimplyapplyresearch-supportedtreatments withouttakingintoaccounttheclient’sviews.Orperhapsaclientwasseriously depressed,andarecentSRindicatedCBTwasawell-supportedpsychotherapybut
iftheclienthadaseriousintellectualdisability,thepractitionermayopt not toprovide CBT,giventhejudgmentthattheclientwasunabletoundertaketheself-reflectionand homeworkrequiredforthismodel.Instead,alternativetherapiesmightbeconsidered anddiscussedwithclientand/orresponsibleparty,interventionsuchasbehavioranalysis,orantidepressantmedication.InEBPthereis no assertionthat researchrules ! EBPdoesassertthatresearch mustbeconsidered,alongwithclientpreferences andvalues,professionalethics,availableresourcesandavailableskills.Thisis why,properlyconsidered,therearenosuchthingsasevidence-based practices (nouns).Thereisthefivestepdecision-makingmodelofEBP,whichismuch moresophisticatedthansimplylocatingresearch-supportedtreatmentsanddecidingtomakeuseofthemalongwithjudici ousconsiderationoftheotherequally crucialfactorsfoundintheEBPmodel(Thyer&Myers,2011;Thyer&Pignotti, 2011).Itisreallyquiteimportantforprogramevaluatorswhowishtobe informedaboutEBPtoreadtheoriginalsourcesofinformationaboutthis model,suchasStrausetal.(2011)andpreviouseditions,andnotonthirdor fourth-handrestatements,whichoftenpr esentquitedistortedpicturesofthis model.SeeThyer(2013)foranexampleofcorrectingmisrepresentationsof EBP,whichhavebeenpresentedinthesocialworkliterature.
TheRoleofTheoryinProgramEvaluation
Inthebestofallpossibleworlds,everyprogramwouldalsobebasedonasound theoreticalmodel.Thatis,beforethepractitionersjumpedintoasocialproblem andstartedhelping,theywoulddevelopamodelthatwouldhaveexaminedthe problem howandwhyitoriginatedandwhatwouldworkbesttoremedythesituation.Apsychosocialorothertheorycanbeanorganizingprincipleforeachprogram,onethatprovidesaconsistencyofeffortbysuggestingastandardapproach derivedfromsomewell-articulatedandcomprehensivesocialorbehavioralscience theory.Suchatheorymayserveasaguideinconceptualizingthecausesofproblemsandinproposedmechanismsofactionforinterventions.
Taketheproblemofhyperactivebehavior(HB).IfoneapproachestheunderstandingandtreatmentofchildrenwithHBfromabiologicalorientation,one wouldfocusonpossibleunderlyingdisturbancesinbrainchemistrytoexplainHB andontheuseofmedicationstoregulatethesepresumedchemicalirregularities.If onesubscribedtoalearningtheoryorientation,onemightexaminethepossiblerole ofanover-stimulatingenvironment,orofpeer/parental/teacherinadvertentreinforcementforHB,astentativecausesofHB,andofenvironmentalmanipulationandpositivebehaviormanagementprogramstoreinforceon-taskactivitiesasanintervention. Onetrainedinabiopsychosocialapproachmightemploybothapproaches.Thereare manytheoriesonthecausesofHBandtheoreticallybasedinterventions,whichcould beconsidered.
Anotherexamplemightbethatofdrinkingtoomuchalcohol(oftencalledalcoholism).Therearealsoawidearrayoftheoriesastotheetiologyofabusivedrinking, andofwaystohelppeopledrinklessortoabstainfromdrinking.Accordingtothe theoryofAlcoholicsAnonymous(AA),alcoholismiscausedbyabiologicallybased allergicreactiontoethanol,inthatthesmallestsiptriggersanirresistiblecraving todrinkmore,leadingtoalossofcontrol.Thisbiologicaltheoryleads,naturally,
toaninterventivetheory,namelytoabstaincompletelyfromexposuretoalcohol. Justasapersonwithapeanutallergyistoldtoavoidallconsumptionofpeanuts, thealcoholicisprovidedaprogramofeducation,interpersonalsupport,andspiritual developmentintendedtoreinforcetheirattemptstoabstaincompletely.Nooneis eversaidto “recover” fromalcoholismortobe “cured,” anymorethanoneislikely toceasehavingapeanutallergy.AccordingtotheAAtheory peoplearesimplyin variousstagesof “recovery.” AAisaprogramofrecoveryclearlyderivedfromboth anetiologicalandinterventivetheory,inthiscaseabiologicalone.However,there arecertainlyothertheoriesofabusivedrinkingbasedonlearningtheory,genetic,psychodynamic,andfamilialinfluences,andsoforth.Eachtheoryhasitsownderivative programs(e.g.,antabusemedicationtoproducenauseaupondrinking,anti-craving drugs,behaviormodification,thecommunity-reinforcementapproachtoabstinence, psychotherapy,rationalrecovery,familytherapy,meditation,etc.).
Consideradifferentproblem.Supposeyouarehiredtorunatreatmentprogram formenwhobatter.Dothesemenfitasingleprofile?Isoneinterventivestrategyall thatisneeded?Saunders(1992)arguesthattherearethreetheoreticallydistincttypes ofmenwhobatter:thosewhowereseverelyabusedaschildren;emotionallyvolatile menwithrigidsex-roleattitudes,whofearlosingtheirpartnersandaredepressed, suicidal,andangry;andfamily-onlyaggressorswhotendtohaverelativelyliberal attitudesaboutsexroles,thelowestrateofabuseinchildhood,themostmaritalsatisfaction,andwhoaregenerallynonassertive.Isthereapossibilitythatsomeinterventionsmayworkbetterwithonetypeofabuserthanwithanother?
Thefieldofinterpersonalviolence,specificallyspousalbattering,isdominated byfeministapproaches,whichattempttoexplainthisproblemintermsofthe exploitativeexertionofpowerencouragedbyunfairpatriarchalsocialinstitutions. Thishasledtothewidespreadadoptionanduseofthe “DuluthModel” ofinterventionandprevention(see http://www.theduluthmodel.org/about/index.html ). Alternativetheoriesofspousalbatteringtakeamoregender-neutralapproach,takingintoaccountfemale-on-femaleandmale-on-malebatteringinintimaterelations (Ferreira&Buttell,inpress).ThisisquitedifferentfromtheDuluthModel.Thus,it canbeseenhowthetheoryoneadoptsforagivenprogramcanstronglyinfluence howtheproblemisseentobecausedandhowinterventionmayoccur.
Theoreticalmodelscanbeimportanttounderstandinghowaprogramshould workandwhereoneshouldlookforindicationsthataprogramissuccessful(see Box1.2).Butalltoooftenwhatpassesfortheoryinmanysocialserviceagencies isablendofpastexperienceandtradition,andevenmythsorprofessionalfolklore.
BOX 1.2 Characteristicsof “Good” SocialServicePrograms
• Staffing
• Budgets
• Stablefunding
• Recognizedidentity
• Conceptualortheoreticalfoundation
• Aservicephilosophy
• Systematiceffortsatempirical evaluationofservices
• Evidence-basedresearchfoundation
Evaluatorswouldhavenoproblemwiththatiftheprogramwasoftensuccessfulin rehabilitating,helping,fixing,orcuringclients.Butwhenaprogramisnotsuccessfulmuchofthetime,thepossibilityexiststhateventhoughtheprogramwasimplementedasdesigned,theunderlyingtheoryisflawed.Suchasituationcallsfornew thinkingandanewapproachtotheproblem.Theoriesmaybeabletotellushow toaccomplishourgoals(Conrad&Miller,1987),butnotineverycase.
Programevaluationstudiesshouldgenerallybeconsideredasrelativelypoor testsofthevalidityoftheories.Take,forexample,aprogramwithatheoretically derivedinterventionthatisfoundtoproducepositiveresultswiththeclients they appeartobereallybenefittingfromtheprogram’sservices.Youmightbetempted toconcludethatthismeansthattheprogram’sunderlyingtheoryisthusprovento bevalid.Doingsowouldbemakingamistake.Aninterventionmayworkquite wellforreasonscompletelyunrelatedtoitsunderlyingtheoreticalrationale(see Lilienfeld,Ritschel,Lynn,Cautin,&Latzman,2014,foracomprehensiveaccountingofhowthiscanhappen).Forexample,theearly(circa1950s)behavioralpsychotherapycalledsystematicdesensitization(SD)wassaidtoworkbecauseofan underlyingtheoreticalprocesscalledreciprocalinhibition(RI).Carefultestsinthe late1960sand1970sfoundoutprettyconclusivelythatwhile,yes,SDwasmoderatelyhelpfulinovercomingcertainanxietydisordersitdid not workbecauseofany processesofRI.Morerecently,thepsychotherapycalledeyemovementdesensitizationandreprocessing(EMDR,circa1990s)wassaidtoworkbecauseofunderlying physiologicalchangesinthebraincreatedbyaskingtheclienttomovetheireyes backandforthinacertainpatternestablishedbythetherapist,whilethinkingof traumaticevents.Itturnsoutthattheseso-calledsaccadiceyemovementsare completelyirrelevanttothemodestimprovementsgarneredbyEMDR,whichare moreplausiblyattributedtomildexposuretoanxiety-evokingimageryandtoplacebofactors(seeLohr,Hooke,Gist,&Tolin,2003).Ifsomeoneimprovesafter enrollinginaprogrambasedonbehaviortherapy,psychotherapy,orsolutionfocusedtreatmentthatdoes not provethattheserespectivetheoriesaretrue.Generallyspeaking,thetheoreticalpropositionsthattherapiesandprogramsarebasedon require evidence oftheirvalidityindependentoftheirsuccessfuloutcomes.
Negativeoutcomesarealsoproblematicintermsofmakinginferencesabout thevalidityofaprogram’sunderlyingtheory.Sayyouconductaprogramevaluationofatheoreticallybasedserviceandtheoutcomeisnegative,forexample,the clientsdonotimprove,or,heavenforbid,getworse.Doesn’tthis “prove” thatthe theorytheprogramisbasedonisincorrect?Notsofast.Advocatesofthetheory mayclaimthattheoutcomeswerenegativebecausethetreatmentorservicewas notreallyimplementedcorrectly,orthatthetherapistswereinsufficientlytrained orsupervised,thedurationoftreatmentwastoobrief,theoutcomemeasureswere insufficientlysensitivetodetectthedelicatebutfar-reachingbeneficialeffectsof therapy,thatthefollow-upperiodwasnotlongenough,andsoon.Therecanbe amyriadofreasons(somegenuinerationalesandsomespuriousexcuses)whya programmaynotwork,andtheinvalidityofitsunderlyingtheoryisbutone.We donotencourageprogramevaluatorstomakeinferencesaboutthetruthorfalsity oftheoriesbasedontheprogram’soutcomes.Inveryrarecases,astringently designedandexecutedoutcomestudywithnegativeresults could beseenasreflectiveoftheunderlyingtheory’sinvalidity,butitismuchlesscredibletoclaimthata
positiveoutcomemeansthatthetheoryiscorrect.Programsmayworkformany reasonsapartfromthosehypothesizedbytheprogram’sdevelopersorconceptual grandparents.
Toacertainextent,programdesignersandimplementersareprotectedfrom manyoftheproblemsthataccompanymisguidedanderroneoustheorieswhen theybaseprogramsnotsolelyontheoriesalonebutalsoonmethodsofpractice thathavebeenempiricallysupportedbyresearchandevaluation.Thebestprograms,thoseonthe “cuttingedge,” willbebuiltonafirmfoundationofthelatest empiricalresearch.Theseprogramscanprovidealevelofconfidenceandknowledgeaboutwhattoexpectintermsofsuccessrates.Thesewouldbetheprograms wewouldchooseforourselvesifweorourfamiliesneededintervention.
Youshouldrealize,however,thatmany(perhapsmost)humanservicesprogramsare not basedonanyexplicittheoryofhumanbehaviororanyetiological socialorbehavioralsocialsciencetheoryexplaininghowparticularproblems arise,orevenanyparticularinterventivetheory.Such “atheoretical” programs maybebasedoncommonsense,authority,ortradition.Forexample,weknowof onegovernorwhoarrangedforthestatetoprovidethemotherofeverynewborn babyacompactdisc(CD)ofclassicalmusic.TheseCDsweredulydistributedto thousandsofnewmothers,costingthestatehundredsofthousandsofdollarsin taxpayermoney.Thisauthoritativeinitiativewasinstigatedbythegovernor’sreadingaboutsomethingcalledthe “Mozarteffect,” whereinlisteningtoaparticular Mozartsonatasupposedlyresultedincollegestudentsearninghighertestscores (nottrue).The(vain)hopewasthatmotherswhoweregivenCDsofclassical music(whichfailedtocontainarecordingoftheparticularMozartsonatainquestion!)wouldplaythesewithinhearingoftheirinfants,whowouldgrowuptobe smarter.Thishumanservicesprogramwasnotbasedonanyetiologicaltheory aboutintelligence,orevenaninterventivetheoryonhowlisteningtoclassical musicwassupposedtoimproveintelligence.Itwasbasedon authority,thegovernor’smandate.Unfortunately,subsequentstudiesfailedtodemonstratetheMozart effectintesttaking,andnonedemonstratedanyimprovementininfantintelligence.
Numerousexamplesofsuchatheoreticalprogramscanbefound,frompolicies removingthedriver’slicensesofhighschooldropoutstopolicieswithinchildprotectiveserviceprograms,homelessshelters,soupkitchens,andsoforth.Askthe practitionersprovidingsuchservicesabouttheirprogram’ssocialorbehavioraltheoreticalorientation,andoftenyouwillbemetwithpuzzledlooks.Tobesure,these programs canbe prospectivelydesignedinaccordwithsometheory,butinmany instancestheyarenot.Anditcanbemisleadingto retroactively attempttofitatheoreticalapproachtoexplainaparticularprogram,becausethesameprogramcould nodoubtbesimilarlyexplainedbyperhapsdozensofrivaltheories,leavingyouno closertoavalidunderstandingofhowtheprogrammaybeexertingitseffects,if any.Therelationshipbetweenformaltheoryandresearchonpracticeoutcomesis acomplexone,andThyer(1994;2001)addressesitmorecompletelythanispossiblehere.
Nottobeignoredarethepossible negative effectsoftheoryinhumanservices programs,especiallyastheyrelatetoprogramsandtheirevaluation(Thyer,2012). Manysocialprogramsarelikelybasedonfalsetheory,theoriesbasedoninvalid etiologicalandtherapeuticpropositionsorpseudoscience(Pignotti&Thyer,2009;
Thyer&Pignotti,2010).So-calledbootcampsforjuveniledelinquentsareone example.Herethetheory(looselyused)isthatyouthwhogetintotroublewith thelawhavesomesortofdefectintheir “character.” Certainpoliticians’ military experiencesledthemtobelievethataMarine-stylebootcampregimencouldsomehowcorrectthecharacterofmisbehavingyoungmen(andwomen),andthen,upon theirreleasefromthebootcamp,berestoredtotheirhomecommunitieshaving becomesomehow “stronger” incharacterandbetterpreparedtoresisttheinfluence ofevilpeers.Well,theevidenceisprettyclearthatbootcampsdo not workwellat preventingrecidivisminyouth(seesystematicreviewonthetopicavailableat www.campbellcollaboration.org andMacKenzie,Bierie,&Mitchell,2007).Those states,whichsupportbootcampsforjuvenilescanbeseenasprovidinga disservice tokids,squanderingtaxpayers’ funds,andemployinghundredsofstaffengagedin deliveringuselessservices,nottomentiondivertingyouthfrompotentiallymore effectiveprograms.Theeffortsexpendedbyevaluatorstoappraisetheoutcomes ofineffectiveprogramscanalsobeseenasdivertedfrommoreusefulactivities. Theseareallharmfuleffectsofgroundingsocialservicesininvalidtheories.
Programsvarygreatly.Somearesophisticatedandotherssimplistic evencomposedofasingleactivity.Inscaletheyrangefromasmallcooperativebabysitting programforyoungmotherstothefederalfoodstampprogramthattouchesmillionsoflives.Itisnotalwayseasytodeterminewhethercertainactivitiesshould beconsideredaprogramorpartofthecollectionofactivitiesthatcomprisealarger singleprogram.Eitherway,programevaluationcanbeundertakenaslongasa desiredobjectiveoroutcomecanbestated.Althoughinsomeagencies programs arereferredtoas services,inthisbookthetermswillbeusedinterchangeably.
PROGRAMEVALUATIONDEFINED
Programevaluationisappliedresearchusedaspartofthemanagerialprocess. Evaluationsareconductedtoaidthosewhomustmakeadministrativedecisions abouthumanservicesprograms.Unliketheoreticalresearch,wherescientistsengage inscienceforitsownsake,programevaluationsystematicallyexamineshuman servicesprogramsforpragmaticreasons.Decisionmakersmayneedtoknowifa programaccomplisheditsobjectives,ifitisworthfundingagainnextyear,orif alessexpensiveprogramcanaccomplishthesameresults.Welikethefollowing definitionofprogramevaluationprovidedbyGrinnellandUnrau(2008,p.553):
Aformofappraisal,usingvalidandreliableresearchmethods,thatexaminesthe processesoroutcomesofanorganizationthatexiststofulfillsomesocialneed.
Programevaluationislikebasicresearchinthatbothfollowalogical,orderly sequenceofinvestigation.Bothbeginwithaproblem,aquestion,orahypothesis. Normally,thereissomereviewofwhatisknownabouttheproblem,including prioreffortsandtheoreticalapproaches(thisisknownasreviewingtheliterature). Aresearchorevaluationdesign(ablueprinttoguidethedatacollectionefforts)is developed,anddataaregatheredandthenanalyzed.Whenthoughtofthisway, bothresearchandevaluationaresimilartothetask-centeredorproblem-solving processknowntomanyhumanservicesprofessionals.
Researchandevaluationdifferwithregardtotheexpecteduseorutilityofthe data.Theremaybenoanticipatedneedordemandfor “pure” research,whereasan assemblageofindividualsmayanxiouslyawaittheresultsofaprogramevaluation. Also,thegoalofresearchisoftentoproducegeneralizableknowledge,whileinformationfromaprogramevaluationmaybeapplicabletoonlyaspecificagency. However,bothareapproachedwithsomedegreeofrigor.Thinkofprogramevaluationasatool amanagementtoolthatyoucanusetomake(andtohelpothers make)betterdecisionsaboutsocialandhumanservicesprograms.Programevaluationhelpsustomakethebestuseofourresourcesaswelabortoimprovethequalityoflifeofourclients.
Programevaluationusuallyinvolvesmakingcomparisons.Infact,Schalockand Thornton(1988)havedefinedprogramevaluationas “structuredcomparison.” Few programscanbeevaluatedwithoutcomparingthemtosomething.Programsinone agencymaybecomparedtosimilarprogramsinotheragencies,topastorprior efforts,oragainstastatedobjective;butwithoutsomeformofcomparison,there canbenoevaluation.Amajorthrustofthisbookistohelpyoufindorcreate(to conceptualize)basesofcomparisonforyourownprogramevaluationefforts.
REASONSWHYPROGRAMSAREEVALUATED
Quiteoftensocialandhumanservicesprogramsareevaluatedbecauseofaneedto beaccountabletoasponsoringorfundingagency,orbecausecompetitionforscarce fundsrequiresthatonlyoneprogram(normally,themosteffectiveorefficientprogram)canbefunded.Programevaluationisneededwhenevernewinterventionsare beingtriedanditisnotknownwhethertheywillbeassuccessfulasformermethods, orwhenthereisaperceptionthataprogramcouldbeimproved thatitcould becomemoreproductiveorbetterinsomeway.Weevaluateonthoseoccasions whenitisimportanttohavesomeobjectiveassessmentorfeedbackaboutthe worthofoursocialandhumanservicesprograms.Thefollowingscenariosillustrate someoftheoccasionswhenprogramevaluationsareencountered.
Scenario1:TheRequiredEvaluation Youragencyisapplyingforfunding fromtheUnitedWayinyourcommunitytobeginanewprogramdesignedtoprovidecounselingtomenwhohavebeenprosecutedfordomesticviolence.Youhave beenaskedtopreparetheprogramproposal.Asyoureadtheinstructionsforpreparingtheproposal,younoticethatbesidesdescribingtheproject,listingitsobjectives,pointingoutitsuniqueness,andstatingtheamountoffundingthatwillbe required,theproposalalsorequiresaprojectevaluation.Attheendoftheproject year,datamustbepresentedtoshowthattheprojecthadasuccessfuloutcomeand animpactontheproblemofdomesticviolence.
Scenario2:CompetitionforScarceFunds Yourinnovativeprogramformen whobatterhasbeenoperatingforayear.Youhavebeenabletoobtainsomedata thatyouhopewillfavorablyinfluencethecommitteedecidingthecontinuationof fundingforyourprogram.Asyouprepareyourpresentation,youdiscoverthataseconddomesticviolenceprojectfromanotheragencywillalsobemakingarequestto befunded.Youfurtherlearnthatthereisonlyenoughmoneytofundoneprogram.
