You are an expert M&E evaluator with experience assessing methods sections in evaluation, research, and assessment reports. Score the methods section I have attached or pasted using the rubric below. Read it as a competent peer reviewer would: could you replicate or critique the study from this section alone?
SCORING RUBRIC - Methods Section Quality
Score each dimension 1-5 using these criteria:
DIMENSION 1: Design Justification
| 5 | All elements present. Overall design is named (for example, theory-based evaluation, contribution analysis, mixed-methods cross-sectional, quasi-experimental with comparison group, realist evaluation, outcome harvesting) and justified with a clear rationale tied to the evaluation purpose and questions. Reader understands why this design and not an alternative. |
|---|---|
| 4 | Design named and justified. Justification may be brief but links to purpose or questions are present. |
| 3 | Design named but not justified, or justified in generic terms without linking to the specific evaluation context. |
| 2 | Design label appears but reader cannot tell what was actually done. No justification. |
| 1 | No design named. Methods section reads as a list of data sources without an overall approach. |
DIMENSION 2: Data Collection Detail
| 5 | All elements present. Each instrument is described (purpose, format, length, respondent type). Sampling is documented (sample frame, size, selection method, response rates where applicable). Procedures are described (training, piloting, data collection period, mode). A competent reader could assess the rigor of the data collection. |
|---|---|
| 4 | Most elements present. Instruments and sampling described; procedures partial (for example, piloting mentioned but not detailed). |
| 3 | Instruments and sample sizes named but sampling method and procedures generic. Reader cannot fully assess rigor. |
| 2 | Data sources listed at high level only ("KIIs and FGDs were conducted"). Sample sizes given but selection method absent. Procedures absent. |
| 1 | Data collection described in one or two sentences with no detail. |
DIMENSION 3: Analysis Approach
| 5 | All elements present. Analytic approach for each data type is specified: quantitative (descriptive statistics, inferential tests, software, missing data handling); qualitative (coding approach, framework or grounded, software, inter-rater process); mixed-methods (integration strategy, sequencing). Reader can trace how raw data became findings. |
|---|---|
| 4 | Analytic approach specified for each data type with minor gaps (for example, qualitative coding approach stated but inter-rater process not described). |
| 3 | Analytic approach mentioned but generic ("data were analyzed thematically," "quantitative data were analyzed using Excel"). Reader does not see how raw data became findings. |
| 2 | Analysis described in one sentence. No distinction between data types. |
| 1 | No analysis approach described. |
DIMENSION 4: Quality Assurance
| 5 | All elements present. Validity safeguards (triangulation procedures named, instrument validation, pilot testing). Reliability safeguards (inter-rater agreement, transcription protocols, audit trail). Ethical safeguards (consent process, anonymization rules, safeguarding measures, sensitive-topic handling, data security, IRB or equivalent where applicable). |
|---|---|
| 4 | Most quality assurance elements present. One category lighter than others (for example, ethics described but inter-rater process absent). |
| 3 | Quality assurance mentioned but generic ("ethical principles were followed," "triangulation was used"). Specific procedures absent. |
| 2 | Quality assurance limited to a single sentence on ethics. Validity and reliability not addressed. |
| 1 | No quality assurance described. |
DIMENSION 5: Reproducibility
| 5 | All elements present. A competent reader could replicate or critique the study from the methods section alone. Instruments referenced or annexed (annex pointers given). Sampling decisions documented. Analytic decisions documented. Deviations from the original plan noted with rationale. |
|---|---|
| 4 | Most reproducibility elements present. One element light (for example, instruments referenced but not annexed). |
| 3 | Reader could replicate the broad approach but key procedural decisions missing. Instruments not referenced. |
| 2 | Reader cannot replicate. Methods section is descriptive rather than procedural. |
| 1 | Methods section is a label without procedural content. Reproduction impossible. |
OUTPUT FORMAT:
| Dimension | Score (1-5) | Evidence | Priority Revision |
|---|---|---|---|
| Design Justification | |||
| Data Collection Detail | |||
| Analysis Approach | |||
| Quality Assurance | |||
| Reproducibility |
Total: X/25 Band: Strong (22-25) / Adequate (17-21) / Needs Revision (11-16) / Substantial Revision (5-10) Single Most Important Revision: [One specific sentence]
For any dimension scored 1 or 2, name the specific procedural detail that is missing and suggest the text that should be added.
METHODS SECTION TO SCORE:
(See the attached or pasted document.)
Document to review: Attached, or pasted below. Base everything above on it and do not invent details that are not in it.