MPH-503 Week 8 Policy Evaluation and Synthesis: How to Write It

What MPH-503 Week 8 asks for

Two years after a notification requirement took effect, a county's indicator has not moved. Three explanations compete and they lead in opposite directions. The policy may not work, in which case it should be replaced. It may work but never have been implemented, in which case the finding is about enforcement capacity rather than about the idea. Or the indicator may be measuring something the policy was never designed to change, in which case the evaluation is at fault and a good policy is about to be repealed on bad evidence. Distinguishing between those three is the analytic task, and papers that skip it deliver verdicts nobody should act on.

Design is what separates the three. A theory of change written before anything is measured tells you which intermediate steps should have moved first, so that a policy showing no change in enrollment, no change in submissions and no change in the outcome is failing at a different point from one where submissions rose and the outcome did not. Comparison is what allows attribution: a jurisdiction that did not adopt the policy, a period before adoption modelled as a trend rather than as a single point, or a group excluded by a threshold in the rule. A simple before-and-after comparison in one place credits the policy with everything else that happened that year, which in most real settings is a great deal.

Process evaluation deserves equal space with outcome evaluation and rarely gets it. Whether the regulated organizations knew about the requirement, whether the submission route functioned, whether enforcement occurred, and whether the affected population was reached are all measurable, and they are the questions that explain a null result. Unintended consequences belong in the same section, because policies produce effects outside their target, and in this territory the recurring ones are documented: requirements that push organizations toward the least burdensome interpretation, thresholds that create incentives to sit just below them, and reporting duties that shift attention toward what is counted.

The portfolio side of this stage is a construction problem. Six stages have produced a problem definition, a legal analysis, an economic case, an ethical justification, a political strategy and a communication product. Bound together untouched, they read as an anthology with inconsistent claims and six different framings. Rebuilt around one argument, with a common frame and a single set of figures, they read as the work of somebody who could hand this to a decision-maker.

The MPH-503 Week 8 method, step by step

Seven moves for an evaluation plan and a portfolio that holds together.

1. Draw the theory of change before choosing any measure

   Inputs, activities, outputs, intermediate outcomes, then the population outcome. Each arrow is an assumption you can test, and the chain tells you where to look first when nothing moves at the end.

2. Pick indicators that already exist where possible

   An indicator that requires new data collection frequently means no evaluation happens at all. Existing administrative, surveillance or licensure data with a known lag is worth more than a perfect measure nobody will fund.

3. Specify the counterfactual explicitly

   Comparison jurisdiction, pre-existing trend, or a group the rule excluded. Write the sentence that begins: without this policy, we would have expected. If you cannot finish it, you cannot attribute anything.

4. Build the process evaluation alongside the outcome evaluation

   Awareness, submission rates, enforcement actions, reach into the affected population. These measures tell you whether a null outcome means the policy failed or was never delivered.

5. Pre-specify what would count as success and as failure

   State the threshold and the window before the data exists. Advocacy has an incentive to reinterpret whatever happens as vindication, and pre-specification is the discipline that resists it.

6. Name two plausible unintended consequences and how you would detect them

   Each with a measure attached. A paper that anticipates a perverse incentive and proposes the indicator that would reveal it is doing the work that separates evaluation from monitoring.

7. Rebuild the portfolio around one claim and one set of numbers

   Reconcile every figure across the sections, hold a single frame throughout, cut what the argument does not need, and write connective passages so each part visibly earns its place.

A layout and word budget for the closing portfolio

Our frame for a culminating advocacy document with an evaluation plan, sized for roughly 1,600 to 2,000 words excluding any appended products. It is our own outline rather than anything the university issues, and your week's scoring guide outranks it wherever they disagree.

Evidence craft for evaluation and synthesis

Name the evaluation design in the field's own terms. Interrupted time series, difference in differences, a comparison jurisdiction design, or a qualitative process evaluation. Using the name signals that you know what your design can and cannot rule out, which is the point of naming it.

Report the data lag with the indicator. Many population measures are published a year or more after the period they describe, which means an evaluation window shorter than the lag cannot produce an answer in time for the decision it is meant to inform. Saying so is a practical insight decision-makers rarely hear.

Reconcile every number across the portfolio. A population figure in the problem statement that differs from the one in the economic section is the most common defect in stitched-together documents, and it undermines confidence in everything else on the page. One table of agreed figures, applied everywhere.

Keep the frame consistent from first page to last. If the case is framed as fairness in the brief and as fiscal prudence in the analysis, a reader notices, and in advocacy an inconsistent frame is read as an argument searching for a justification.

Handle reuse of your earlier submissions carefully. Academic programs generally expect work to be original to the assignment, and rules about resubmitting your own text vary, so check your program's policy and your section's instructions. Reusing the research while rewriting the prose around the unified claim is both safer and produces the better document.

Five mistakes that cost points in this week's territory

• Before and after in one place. Without a comparison or a modelled trend, the design credits the policy with every other thing that changed in the same period.
• Declaring failure without checking delivery. A null outcome with no process measures cannot distinguish a bad policy from an unimplemented one.
• Evaluation as an afterthought. A plan invented after the recommendation tends to select the measures the policy would move rather than the ones that matter.
• A stapled anthology. Sections carried over untouched from earlier stages, with inconsistent framing and mismatched figures, read as assembly rather than synthesis.
• No baseline. An indicator with no starting value and no window cannot show change, which makes the whole evaluation plan decorative.

Before you submit

• A theory of change appears before any indicator is chosen
• The counterfactual is stated as a sentence, not implied
• Process measures sit beside outcome measures
• Success and failure thresholds are pre-specified with a window
• Every figure is reconciled across all sections of the portfolio
• Every reference appears in the text and every in-text citation appears in the list

Questions students ask about this stage

Keep going