Purpose and Inputs
Use this rubric for a planning or completed-document review of a nonrandomized causal evaluation. Supply the question and intended effect, assignment and comparison records, exposure and measurement timing, data definitions/access/coverage, analytic rationale and available diagnostics, and applicable safeguards. No plan proves implementation; no polished report identifies an effect without defensible assumptions. Do not supply identifiable or protection-sensitive material.
Evidence and Applicability
Establish purpose, decision, intended claim, selected design, stage, applicable requirements and supplied evidence before rating. Judge a plan as a plan, not completed implementation. Distinguish observed defects, Not Evidenced, Conflicting Evidence and N/A with a reason. Cite exact sections or records. Missing records alone do not prove failure or require repeating a study. Do not invent data, results, assumptions as facts, citations, completed diagnostics, consent, approval or assigned owners. Label unconfirmed owners and timing as proposals. Choose the highest 1-5 anchor fully supported; consequential defects override favorable counts. If a required criterion is unavailable or conflicted, retain partial findings and an evidence state without a complete dimension score. Other-design branches are outside the selected assessment, not defects. If any active assessed criterion is N/A, not evidenced or unresolved, withhold totals and bands, percentages and rescaled equivalents. For the unadapted complete rubric, give a total only when all five dimensions and active criteria are applicable, sufficiently evidenced and conflict-free. Bands: Strong 22-25, Adequate 17-21, Needs Revision 11-16, Substantial Revision 5-10, as secondary documentation summaries. Critical inference, validity, ethics, safety or confidentiality blockers override any score. Scores never grant causal, operational or compliance clearance; designs/stages/adaptations are not automatically comparable.
CONDITIONAL DESIGN CHECKS (apply only the chosen family):
- DiD: justify untreated parallel trends, exposure/anticipation, comparison eligibility, composition and spillovers. Pretrends can inform but never prove the assumption. For staggered adoption or heterogeneous effects, justify cohort/time comparisons, target effects and aggregation rather than automatically interpreting a pooled two-way fixed-effects coefficient as the target effect.
- Sharp continuous-score, single-cutoff RD: define running score/cutoff and compliance; justify continuity and absence of other changes at cutoff; assess manipulation, predetermined covariates and local density, estimation/bandwidth and bias-aware uncertainty/sensitivity. Target the effect at cutoff; no universal parallel-trends test.
- ITS: justify interruption timing and proposed level/slope/lag effect, sufficient pre/post information for the intended inference, stable measurement, concurrent events, seasonality and serial dependence; use suitable uncertainty and trend/timing sensitivity. A comparison series may strengthen the design but is not universally mandatory.
- Matching: justify selection on measured pretreatment covariates, overlap, balance, unmeasured-confounding limitations and suitable sensitivity/dependence. Balance does not prove absence of confounding. Do not match on treatment consequences.
- Aggregate synthetic control: justify donor eligibility and exposure/spillover exclusions, adequate preperiod fit/data, anticipation and comparative context; disclose weights and assess relevant placebo/fit and donor sensitivity, with appropriate inference limits. Good prefit is not proof of a valid counterfactual. Weighting, IV, fuzzy/discrete/multicutoff RD and advanced variants need additional appropriate methodological review. This rubric does not provide estimators or judge those variants using an irrelevant branch. Randomized trials are excluded.
Scoring Criteria
The question, intended effect and population, assignment mechanism, timing and counterfactual strategy are coherent and justified for the stage; alternatives and feasibility limits are explicit.
The question and comparison strategy are usable; one minor target or timing detail needs clarification without changing the claim.
A coherent question and candidate strategy exist, but a material assignment or comparison limitation needs resolution and limits claims.
An observed mismatch between assignment, comparator or target effect threatens the stated claim.
The strategy cannot identify the stated effect, or a before/after change without a defensible counterfactual is presented as causal.
The selected design assumptions have a defensible context-specific rationale and appropriate planned or completed diagnostics; remaining untestable assumptions and sensitivity needs are explicit.
The identification rationale and material selection risks are addressed; one minor diagnostic explanation remains.
Identification is plausible only within clearly stated material limitations that constrain the conclusions; further evidence or analysis is needed.
A known selection, timing, manipulation, spillover or concurrent-change problem undermines the causal claim and is not adequately addressed.
The draft treats an unsupported key assumption as established or ignores a known identification failure; apparent balance, pretrends or fit are claimed to prove causality.
Definitions, timing, comparable measurement and data access/coverage are supported for the stage; missingness and attrition risks and feasible precision assessment are explicit without universal sample rules.
Measurement and data feasibility support the question; one minor coverage or timing explanation needs clarification.
Data are usable for a bounded claim, but a material measurement, coverage or precision limitation is acknowledged and constrains interpretation.
An observed measurement, attrition or data-coverage defect materially distorts the comparison and is inadequately addressed.
The supplied data demonstrably cannot measure the claimed outcome/comparison, yet the intended claim is retained.
The analysis targets the stated effect and design; dependence and uncertainty are justified; relevant robustness and sensitivity checks and reporting rules are specified for the stage.
The main analysis and uncertainty rationale are sound; one minor sensitivity or reporting detail remains.
A defensible analytic outline exists but a material uncertainty or robustness limitation remains explicit and limits conclusions.
An observed analysis or dependence error creates misleading precision or effect interpretation that the draft does not address.
The proposed or reported analysis cannot support the causal inference, or selective reporting or false certainty drives the conclusion.
Stage-appropriate safeguards and applicable review/access requirements are explicit; inference and transportability limits are clear; actions, owners and timing are supported or labeled proposals, with unresolved use conditions visible.
Safeguards and bounded interpretation are sound; one minor action or reporting detail needs clarification.
The use boundary is stated but a material noncritical safeguard or follow-up detail needs confirmation; readiness is not overclaimed.
An observed safeguard or interpretation defect creates material risk or generalizes beyond the supported population without adequate qualification.
The draft endorses unsafe sensitive-data use, invents approval or completed validation, or claims a documentation score resolves an inference or operational blocker.
Score Interpretation
The dimension profile and blockers take precedence. Preserve the original denominator; omit totals under any active evidence gap, conflict or N/A. A justified alternative design is not penalized for lacking another design's diagnostic. Anchors and bands are editorial documentation conventions, not source-owner endorsements or a validated scale. Apply the full Review prompt for scoring.
Methodological References
- HM Treasury, Magenta Book Annex A, May 2026, A2.2, A2.4, A2.6-A2.8, provides family-specific practical guidance, used beyond the UK as methodological support rather than a universal legal standard.
- Cattaneo, Idrobo and Titiunik, Foundations, manuscript 2019/published 2020, sections 2, 4, 5, supports sharp RD checks and local interpretation.
- Abadie 2021, sections 5-7, supports synthetic-control feasibility, donors, fit and sensitivity.
- Callaway and Sant'Anna, author manuscript 2020/published 2021, introduction and section 2, supports appropriate target effects and comparison choices with staggered timing.
Related Work
The existing Design a Quasi-Experimental Evaluation creation prompt is a useful starting point. Methodology Rigor provides a broader review; this rubric adds conditional identification and analysis checks. Broad Evaluation Designer planning does not establish causal identification. Use additional qualified methods review for advanced variants outside the branches above.