Course Description
The worlds of Policy Analysis and Program Evaluation, while overlapping and sharing similarities in terms of general logical approach, are quite different in their level of focus, level of detail, and overall purpose. While the person put in charge of evaluating the success of a particular government program will be necessarily focused on whether or not predicted outcomes have been achieved or processes of delivery have been adequately established, the policy analyst will generally be more concerned with understanding the political and economic issues involved with the passage of a policy (which could affect many different programs) and how those issues will or will not make it possible to adjust the existing or create a new policy. Accordingly, this course will have two tracks, one focusing on the approaches and methods of Program Evaluation, the other on the approaches and methods of Policy Analysis.
Program Evaluation vs. Policy Analysis
While the course is attended by both MPA and PhD students, it is most heavily populated by MPA students. Therefore, the focus on Program Evaluation will be stressed to a greater extent. Fortunately, most of the information about approaches and methods of Program Evaluation is quite relevant and useful to the budding policy analyst and/or professor of public administration. In terms of Program Evaluation, this course will be reviewing the many approaches to conducting an evaluation (e.g. objectives-oriented vs. management-oriented) as well as the many designs that may be adopted (e.g. case studies vs. pretest-posttest vs. regression-discontinuity).
Tools of the Trade
What the Policy Analyst and the Program Evaluator do share is a need to understand the tools at their disposal for collecting the necessary information of their trade. Because the trade of government (either policy making or program implementation) is conducted by people and for people, the information collection tools used by the Policy Analyst and Program Evaluator are necessarily comprised of methods of soliciting information from people (from one person, a small group of people, or a large population of individuals). Therefore, one of the main foci of this course will be learning about the "tools of the trade" for eliciting information from populations, small groups, and individuals. Specifically, the students of this course will learn about Survey Design, Running a Focus Group, and Conducting Interviews (of multiple types).
Workshops and Practitioner Presentations
There is no better way to learn about a profession than to see someone in that profession in action! Second best is to hear them talk about it...so, to better demonstrate some of the concepts being discussed in the class, a number of "hands-on" workshops and practitioner presentations have been scheduled. Each workshop or presentation is being led by a practicing professional in the field of policy analysis and/or program evaluation. Two hands-on workshops are being presented on the topics of Designing Surveys and Leading Focus Groups. The survey design workshop is being directed by Dr. Steven Culver, the Asst. Director of Virginia Tech Office of Academic Assessment. The focus group workshop is being directed by research project manager Stephanie Baker of the Virginia Tech Transportation Institute. In addition, three practitioner presentations are being held. The first presentation is being given Mark Gribbin, a Senior Associate Legislative Analyst working for the Virginia State Joint Legislative Audit and Review Commission (JLARC) on the commission's evaluation approach. Also, Mark will also be hosting a JLARC recruiting event during his visit. The second presentation is being given by Mary Beth Dunkenberger, the Senior Program Director at The Virginia Tech Institute for Policy and Governance (IPG) on the evaluation work of the institute. The final presentation is being given by Owen Perkins, a new junior researcher at IPG on a cost effectiveness analysis he is currently undertaking.
MPA Student Requirements
Quizzes
MPA students will be responsible for taking 6 subject quizzes: Evaluation Approaches, Survey Design, Focus Groups, Interviewing Techniques, Evaluation Design, and Policy Analysis. Each quiz will be completed following discussion of the subject in class.
Quizzes count for 50% of your final grade.
Case Studies
Most of the approaches to evaluation, data collection tools, and evaluation design types will be demonstrated in use by a case study. Each MPA student will be required to present two (2) articles that illustrate the use of an approach, tool, or design under discussion. The presentation will be a 10 minute minimum PowerPoint presentation. A template for the format of the presentation will be on the web site. All case study articles will be available via the web site. Assignments will be made randomly. In addition, each student will create for distribution during the presentation a one-page "brief" explaining the approach, tool, or design discussed. These briefs will be converted in to pdf format and made available via the web site for use by class students in designing their project evaluations. At the end of the semester, the pdfs will be combined to form a single volume for download. The intent is that every student will be able leave the class with their own "workbook" of evaluation approaches, tools, and designs.
Case study presentations count for 20% of your final grade.
Evaluation/Research Mini-Project
Each student will be responsible for completing a small evaluation/research project by the end of the semester. In the fall semester, students in the Research Methods course were given a set of questions that the town manager of Blacksburg would like to have answered about different services in and around the town. The students then devised simple research plans to find answers for the town manager. For example, one question concerned what types of development ideas citizens would find appropriate for the old Blacksburg Middle School property. Working from a hypothesis that Blacksburg citizens would accept an alternative use for the property (other than leaving it as it is), a student proposed conducting a focus group to develop a survey to be distributed to an adequate sample size in and around Blacksburg. Some students devised quick approaches to other questions they had a personal interest in.
While these small research projects do not constitute on their own a program evaluation, they will give the MPA student the opportunity to apply some of the tools being discussed (e.g. stakeholder analysis, interviewing, focus groups, surveys, possibly applying a specific design). In addition, completion of the project will give the student the opportunity to interact with local government officials on real life topics. At the end of the semester these projects will be presented to the Blacksburg Town Manager and, possibly, other town officials.
If you did not take the research methods course in the fall you will need to select one of the existing topics or discuss with me a different question to answer. A list of the questions supplied by the town manager will be available on the web site.
The Mini-project presentation counts for 20% of your final grade.
Class Participation
Each student is expected to have familiarized themselves with the case studies being presented so that they can ask pertinent questions of the case study presenter. The instructor will track the number of pertinent questions asked of presenters per student. This number, in addition to general level of participation, will be used to calculate a final class participation score.
Class participation counts for 10% of your final grade.
PhD Student Requirements
Quizzes
PhD students will be responsible for taking 5 subject quizzes: Evaluation Approaches, Survey Design, Focus Groups, Interviewing Techniques, and Evaluation Design. Each quiz will be completed following discussion of the subject in class.
Quizzes count for 30% of your final grade.
Case Studies
Most of the approaches to evaluation, data collection tools, and evaluation design types will be demonstrated in use by a case study. Each PhD student will be required to present one (1) article that illustrates the use of an approach, tool, or design under discussion. The presentation will be a 10 minute minimum PowerPoint presentation. A template for the format of the presentation will be on the web site. All case study articles will be available via the web site. Assignments will be made randomly. In addition, each student will create for distribution during the presentation a one-page "brief" explaining the approach, tool, or design discussed. These briefs will be converted in to pdf format and made available via the web site for use by class students in designing their project evaluations. At the end of the semester, the pdfs will be combined to form a single volume for download. The intent is that every student will be able leave the class with their own "workbook" of evaluation approaches, tools, and designs.
The case study presentation counts for 10% of your final grade.
Teaching
PhD students graduating from CPAP most often end up working as professors of public administration or working in high level policy analyst positions at all levels of government. Therefore, there is a need by CPAP PhD graduates to be able to both understand policy decisions at a high conceptual level and be able to present those ideas in a clear, thoughtful, and interesting manner. These ideas could be conveyed to a classroom of public administration graduate students or to a room full of political staffers and, possibly, their political bosses. In either case, the public administration PhD student needs to learn to TEACH!
Accordingly, each PhD student in the class will be tasked with two teaching presentations. The first presentation will teach the rest of the class about one type of policy analysis framework (how they go about their work) and the second presentation will teach the rest of the class about one theory of knowledge (what is accepted as evidence/truth). Each presentation will use PowerPoint and will last approximately 30 minutes. Additional materials may be used and distributed as necessary. For the benefit of the presenters, the presentations will be evaluated by the MPA students according to criteria derived earlier in the class (these evaluations are for learning purposes only and will not be used by the instructor in final grading).
Upon completion of the Policy Analysis presentations, the PhD students will be responsible for devising a 10-12 question policy analysis quiz derived from information given in the presentations. The quiz will be administered to the MPA students via Blackboard for a grade.
The teaching presentations count for 25% of your final grade.
Policy Analysis Proposal
The final assignment for the PhD student will be a policy analysis proposal. The student will be responsible early in the semester for identifying a policy area in which they have an interest. After completing the policy presentations, the student will be responsible for developing a five-page minimum length proposal to investigate some aspect of the selected policy area. The proposal will discuss the political and economic context of the policy area, the stakeholders to be considered in the analysis, the framework and epistemological stance from which the analyst will operate, and the approach and methods that will be used to conduct the analysis (as there are too many approaches and methods to be covered in this one class, it is not necessary that you only stick to those discussed).
The policy analysis proposal counts for 25% of your final grade.
Class Participation
Each student is expected to have familiarized themselves with the case studies being presented so that they can ask pertinent questions of the case study presenter. The instructor will track the number of pertinent questions asked of presenters per student. This number, in addition to general level of participation, will be used to calculate a final class participation score.
Class participation counts for 10% of your final grade.
Textbooks |
|
|
|
| |
Program Evaluation: Alternative Approaches and Practical Guidelines, Third Edition, Fitzpatrick, Sanders, and Worthen | |||||
The Power of Survey Design: A User's Guide for Managing Surveys, Interpreting Results, and Influencing Respondents, Iarossi | |||||
Policy Analysis by Design, Bobrow, Dryzek PhD Students only | |||||
|
|
|
|
|
|
|
|
|
|
|
|
Grading |
|
|
|
|
|
| MPA |
|
|
| PhD |
| Quizzes (6): 50% |
| Quizzes (5): 30% | ||
| Approach/Tool Presentations (2): 20% | Approach/Tool Presentation (1): 10% | |||
| Research Presentation: 20% | Policy Analysis Presentations/Quiz (2): 25% | |||
| Class Participation: 10% |
| Policy Analysis Proposal: 25% | ||
|
|
|
|
| Class Participation: 10% |
|
|
|
|
|
|
|
|
|
|
|
|
16-Jan | Introduction |
|
|
| |
| Syllabus |
|
|
|
|
| Books |
|
|
|
|
| Indexes |
|
|
|
|
| Policy Analysis and Evaluation |
|
| ||
| Research vs. Evaluation |
|
| ||
| Frames of Reference in Policy Analysis |
| |||
| Introduction to Program Evaluation |
| |||
| The Research/Evaluation Project |
| |||
| Web Site |
|
|
|
|
|
|
|
|
|
|
23-Jan | Approaches to Evaluation and Analysis |
| |||
| Textbook - Fitzpatrick, Sanders, Worthen, Chs. 4-6 | ||||
|
|
|
|
|
|
| Lecture - Objectives-Oriented Evaluation |
| |||
| Student Presentations |
|
| ||
|
| Objectives Oriented |
|
| |
|
|
| When Hard Questions Are Asked: Evaluating Writing Centers | ||
|
| Logic Models |
|
| |
|
|
| A city initiative to improve the quality of life for urban youth: how evaluation contributed to effective social programming | ||
|
| Program Theory |
|
| |
|
|
| Using program theory models in evaluation of industrial modernization programs: three case studies | ||
|
|
|
|
|
|
| Lecture - Management-Oriented Evaluation |
| |||
| Student Presentations |
|
| ||
|
| CIPP - Context, Input, Process, and Product Evaluation | |||
|
|
| Imprisonment and Career Development: An Evaluation of a Guidance Programme for Job Finding | ||
|
| PERT - Program (or Project) Evaluation and Review Technique | |||
|
|
| Assessment and evaluation of contractor data against client goals using PERT approach | ||
|
|
|
|
|
|
| Lecture - Consumer-Oriented Evaluation |
| |||
| Student Presentations |
|
| ||
|
| Consumer Protection |
|
| |
|
|
| Evaluating technologies in reproductive health: case studies of a consumer protection approach | ||
|
|
|
|
|
|
30-Jan | Approaches to Evaluation and Analysis |
| |||
| Textbook - Fitzpatrick, Sanders, Worthen, Chs. 7-8 | ||||
|
|
|
|
|
|
| Lecture - Expertise-Oriented Evaluation |
| |||
| Student Presentations |
|
| ||
|
| Accreditation Standards |
| ||
|
|
| Flexner, Accreditation, and Evaluation | ||
|
| Adoption Panel |
|
| |
|
|
| Adoption Now: A Joint Initiative of New York's Courts and Child Welfare System | ||
|
|
|
|
|
|
| Lecture - Participant-Oriented Evaluation |
| |||
| Student Presentations |
|
| ||
|
| Stake's Countenance |
|
| |
|
|
| Stakes Countenance Model: Evaluating an Environmental Education Professional Development Course | ||
|
| Responsive Evaluation |
| ||
|
|
| A Qualitative Evaluation of a Project to Enhance Pupils' Emotional Literacy Through a Student Assistance Programme | ||
|
| Naturalistic Evaluation |
| ||
|
|
| An enhanced assessment and support team (EAST) for dementing elders - review of a Scottish regional initiative | ||
|
| Participatory Evaluation |
| ||
|
|
| Involving Children and Young People in Research on Domestic Violence and Housing | ||
|
| Utilization-Focused Evaluation |
| ||
|
|
| Utilization Focused Evaluation for Tourism | ||
|
| Empowerment Evaluation |
| ||
|
|
| Empowerment Evaluation Applied: Experiences, Analysis, and Recommendations from a Case Study | ||
|
|
| An Empowerment Evaluation Model for Sexual Assualt Programs: Empirical Evidence of Effectiveness | ||
|
|
|
|
|
|
| Lecture - Summary of Evaluation Approaches | ||||
|
|
|
|
|
|
| Approaches Quiz |
|
|
| |
|
|
|
|
|
|
6-Feb | Tools of the Trade: Surveys |
|
| ||
| Textbook - Iarossi, Chs. 1-3 |
|
| ||
|
|
|
|
|
|
| Lecture - Data Collection Methods: Survey Methodology | ||||
| Video - The Power of Survey Design, Giuseppe Iarossi | ||||
|
|
|
|
|
|
| Survey Workshop - "Survey Design" |
| |||
|
| Steven Culver, Virginia Tech Office of Academic Assessment | |||
|
|
|
|
|
|
| Research/Evaluation Topics Due |
| |||
| Policy Proposal Areas Due |
| |||
| Surveys Quiz |
|
|
| |
|
|
|
|
|
|
13-Feb | Tools of the Trade: Focus Groups |
| |||
| Textbook - A Manual for the Use of Focus Groups | ||||
|
|
|
|
|
|
| Video - "Focus Groups: Targeting the Market" | ||||
| Lecture - Focus Group Interviewing |
| |||
|
|
|
|
|
|
| Focus Group Workshop - "Small group process techniques that can be used to gather information in a focus group or other small group session" | ||||
|
| Stephanie Baker, Virginia Tech Transportation Institute | |||
|
|
|
|
|
|
| Focus Group Quiz |
|
|
| |
|
|
|
|
|
|
20-Feb | Tools of the Trade: Interviews |
| |||
| Readings |
|
|
|
|
| Using Structured Interview Techniques |
| |||
| Fast Tracking Research With Paired Interviews | ||||
| The use of email interviewing as a qualitative method of inquiry in educational research | ||||
| Conducting In-Depth Interviews: A Guide for Designing and Conducting In-Depth Interviews for Evaluation Input | ||||
| Asking the Right Questions in the Right Ways: Strategies for Ethnographic Interviewing | ||||
|
|
|
|
|
|
| Lecture - Conducting an Interview |
| |||
| Video - Gathering Facts from Interviews |
| |||
| Student Presentations |
|
| ||
|
| Structured Interviews |
|
| |
|
|
| The Structured Interview: An Alternative to the Assessment Center? | ||
|
| Paired Interviews |
|
| |
|
|
| Cannibis and Smoking Research: Interviewing Young People in Self-Selected Friendship Pairs | ||
|
| Telephone Interviews |
|
| |
|
|
| Restoring Rivers One Reach at a Time: Results from a Survey of U.S. River Restoration Practitioners | ||
|
| Email Interviews |
|
| |
|
|
| Development of a standard e-mail methodology: results of an experiment | ||
|
| In-depth Interviews |
|
| |
|
|
| Volunteerism and residential longevity in Saskatoon, Saskatchewan, Canada | ||
|
| Ethnographic Interviews |
| ||
|
|
| Ethnographic Interviews Guide Design of Ford Vehicles Website | ||
|
|
|
|
|
|
| Interviews Quiz |
|
|
| |
|
|
|
|
|
|
27-Feb | Practitioner Presentation |
|
| ||
|
| "JLARC Evaluation Process" |
| ||
|
| Mark Gribbin, Senior Associate Legislative Analyst | |||
| Mark will be hosting a recruiting session at Thomas Conner the evening of the 26th | ||||
|
|
|
|
|
|
5-Mar | Spring Break |
|
|
| |
|
|
|
|
|
|
12-Mar | Evaluation Designs |
|
|
| |
| Lecture - Designs for Causal and Descriptive Information | ||||
| Student Presentations |
|
| ||
|
| Case Study |
|
| |
|
|
| The Qualitative Method of Impact Analysis | ||
|
| Posttest-Only |
|
| |
|
|
| The Daily Show Effect Candidate Evaluations, Efficacy, and American Youth | ||
|
| Pretest-Posttest |
|
| |
|
|
| The Effectiveness of Art Therapy in Reducing Depression in Prison Populations | ||
|
| Nonequivalent Comparison Group | |||
|
|
| County Jail Suicides in a Midwestern State: Moving Beyond the Use of Profiles | ||
|
| Interrupted Time-Series |
| ||
|
|
| Getting the Deterrence Message Out The Project Safe Neighborhoods Public–Private Partnership | ||
|
| Regression-Discontinuity |
| ||
|
|
| An Evaluation of California's Inmate Classification System Using a Generalized Regression Discontinuity Design | ||
|
| Cross-Sectional |
|
| |
|
|
| Breast feeding and obesity: cross sectional study | ||
|
| Mixed Method |
|
| |
|
|
|
|
|
|
| Practitioner Presentations |
|
| ||
|
| "Program Evaluation Research at IPG - A Case Study", Mary Beth Dunkenberger, Senior Program Director | |||
|
| Cost-Design - "Cost-Effectiveness of Combined Adult and Child Day Care Services", Owen Perkins, Research Assistant, IPG | |||
|
|
|
|
|
|
| Evaluation Design Quiz |
|
| ||
|
|
|
|
|
|
19-Mar | Understanding the Context and Figuring Out Who Should Be Involved | ||||
| Readings - Stakeholder Analysis: A Tool for Network Management | ||||
|
| Contextual Assessment: Reading the Network | |||
| Textbook - Fitzpatrick, Sanders, Worthen, Chps. 14 and 16 | ||||
|
|
|
|
|
|
| Lecture - The Political and Economic Context | ||||
| Lecture - Stakeholder Analysis - Who Needs to be Included and Why? | ||||
| Lecture - Reporting and Using Evaluation Information | ||||
|
|
|
|
|
|
| PhD Presentation Practice Session | ||||
|
| ||||
26-Mar | Policy Analysis Frames of Reference (Approaches) | ||||
| Textbook - Policy Analysis by Design, Bobrow and Dryzek, Chps. 2-6 | ||||
| Lecture - A Framework for Comparative Scrutiny of Policy Frames and Introduction to Smoketown | ||||
| PhD Student Presentations |
|
| ||
|
| Welfare Economics |
| ||
|
| Public Choice |
| ||
|
| Social Structure |
|
| |
|
| Information Processing |
| ||
|
| Political Philosophy |
| ||
|
|
|
|
|
|
| Research/Evaluation Plan Draft Due |
| |||
|
|
|
|
|
|
2-Apr | Policy Analysis Theories of Knowledge (Epistomology) | ||||
| Textbook - Policy Analysis by Design, Bobrow and Dryzek, Chps. 8-11 | ||||
| Lecture - Theories of Knowledge |
|
| ||
| PhD Student Presentations |
|
| ||
|
| Positivism |
|
| |
|
| Piecemeal Social Engineering | |||
|
| Ecclectic, Forensic, and Relativism | |||
|
| Accomodationist and Critical Policy Analysis | |||
|
|
|
|
|
|
| Policy Analysis Quiz (MPA Only) |
| |||
|
|
|
|
|
|
9-Apr | Individual Meetings |
|
| ||
|
|
|
|
|
|
16-Apr | Interview and Data Collection Write-up Due | ||||
| Policy Analysis Proposal Rough Draft Due | ||||
|
|
|
|
|
|
23-Apr | No Class |
| |||
|
|
|
|
|
|
30-Apr | No Class |
| |||
|
|
|
|
|
|
7-May | Research/Evaluation Presentations |
| |||
| Policy Analysis Proposal Due |
| |||