Evaluation : Methods for Studying Programs and Policies

Preface ix Chapter 1 Setting the Scene 1(19) Domain of Evaluation 3(1) Definition of Evaluation 4(2) What Is Evaluated? An Excursus on Terminology 6(2) Outcome and Process Evaluation 8(2) Contributions of Evaluation 10(1) History of Evaluation 10(5) Comparison between Evaluation and Other Research 15(3) Summary 18(2) Chapter 2 Purposes of Evaluation 20(26) Who Wants Evaluation? 20(1) Overt and Covert Purposes 21(3) Conditions Unfavorable for Evaluation 24(1) Evaluation for Decision Making 25(1) Evaluation as Organizational Learning 26(2) Intended Uses for Evaluation 28(3) Formative and Summative Evaluation 31(1) Formative-Summative and Process-Outcome: Different Constructs 32(1) Compatibility of Purposes 33(1) How Evaluations Are Commissioned 34(3) Inside versus Outside Evaluation 37(2) Level in the Structure 39(2) Whose Use Shall Be Served? 41(3) Summary 44(2) Chapter 3 Understanding the Program 46(26) Why Program Knowledge Pays 46(2) Characterizing the Program 48(2) Getting to Know the Program 50(1) What Is the Program Trying to Achieve? 51(4) How the Program Works: Surfacing the Program's Theories of Change 55(2) Program Theory and Implementation Theory 57(1) Organizing the Evaluation to Investigate Program Theories 58(3) Building Program Theory 61(1) Using Theories of Change as a Guide to Evaluation 62(4) Comparing Program Theory to Actual Developments 66(1) Advantages of Explicating Theories of Change 67(2) Critiques of Program Theory 69(1) Summary 70(2) Chapter 4 Planning the Evaluation 72(25) The Right Time to Evaluate 73(1) Types of Evaluation Questions 74(2) Additional Decisions in the Planning Period 76(1) How to Decide Which Questions to Pursue 77(4) Specifying the Questions 81(1) Quantitative or Qualitative Study 82(5) Design of the Evaluation 87(2) One Study or a Fleet of Studies? 89(1) Designs for Different Types of Programs 90(1) Some Practical Plans 91(1) Ethical Issues 92(20) Summary 112 Chapter 5 Roles for the Evaluator 97(17) Types of Participative Roles 99(1) Reasons for Alternative Evaluator Roles 100(3) Implementing a Participatory Evaluation 103(2) Balance of Pros and Cons 105(3) Additional Limitations to Participatory Approaches 108(1) Ethical Issues 109(3) Summary 112(2) Chapter 6 Developing Measures 114(38) Measurement 115(2) Program Outcomes 117(6) Choices among Measures 123(3) Unanticipated Consequences 126(1) Interim Markers of Progress 127(3) Program Processes 130(2) Program Inputs, Resources, and Environment 132(4) Multiple Measures 136(1) How Many Measures Are Enough? 137(1) Measurement of Variables 137(2) Developing New Measures 139(5) Desirable Characteristics of Measures in Evaluation 144(6) Summary 150(2) Chapter 7 Collecting Data 152(28) Sources of Data 152(11) Sampling 163(3) Interviewing 166(2) Coding Responses 168(2) Existing Statistical Data 170(5) Ethical Issues in Collecting Data 175(3) Summary 178(2) Chapter 8 Design of the Evaluation 180(35) Designing the Process Evaluation 181(1) Designing the Outcome Evaluation 182(2) Important Concepts 184(4) Designs 188(1) Informal Designs 188(3) Formal Designs 191(22) Summary 213(2) Chapter 9 The Randomized Experiment 215(20) Random Assignment 215(6) Planned Variations 221(2) Procedures for Random Assignment 223(3) Coping with Difficulties That Arise 226(2) Conditions That Make Randomized Experiments Problematic 228(1) Criticisms of the Randomized Experiment 229(4) Summary 233(2) Chapter 10 Extensions of Good Design 235(17) Replication 235(1) Meta-Analysis 236(8) Cost-Benefit and Cost-Effectiveness Analysis 244(6) Summary 250(2) Chapter 11 Qualitative Methods 252(19) Design of Qualitative Evaluations 253(3) Collecting Data 256(7) Fieldwork and Analysis 263(2) Qualitative Evaluation and Program Theory 265(1) Ethical Issues 266(1) Combining Qualitative and Quantitative Approaches 267(2) Summary 269(2) Chapter 12 Analyzing and Interpreting the Data 271(23) Introduction 271(1) Analytic Tasks in Evaluation 272(11) General Strategies of Analysis 283(5) An Example of Program Theory as a Guide to Analysis 288(2) Books on Analytic Methods 290(2) Ethical Issues 292(1) Summary 292(2) Chapter 13 Writing the Report and Disseminating Results 294(26) Reporting 295(10) Dissemination 305(5) Utilization of Evaluation Results 310(10) Chapter 14 Evaluating with Integrity 320(7) Know the Program 321(1) Maintain High Technical Quality and Relevance 321(2) Use Balance and Judgment 323(1) Consider Use from the Start 324(1) Behave Ethically throughout the Study 325(1) Survive and Reap Rewards 325(2) Glossary 327(13) References 340(25) Index 365