At a Glance
Why Does AI-Generated M&E Content Fail?
AI produces M&E content that reads fluently and often looks correct at first pass. It fails on inspection in seven predictable ways. Each has a specific detection method and a specific fix.
| Failure mode | What it looks like, and how often |
|---|---|
| 1. Fabricated statistics | Baseline values, targets, or citations that sound specific but have no source. Most AI outputs with numerical fields. |
| 2. Hallucinated framework claims | Asserted compliance with a standard the AI never actually checked. Common when prompts ask for donor-aligned output. |
| 3. Logic breaks | Outputs labeled as outcomes; intermediate outcomes that don't produce the stated long-term outcome. Common in logframes and theory-of-change drafts. |
| 4. Vague specifics | "Survey" as a means of verification; placeholder language like "appropriate training". Common in MEL plan and data collection content. |
| 5. Generic content disconnection | Text that reads right but does not reflect your program's specific theory of change. Moderate in narrative and proposal drafts. |
| 6. Donor compliance errors | Wrong version of a donor handbook, outdated MER indicators (MER is the PEPFAR HIV reporting indicator set, now under the U.S. State Department), or fabricated donor requirements. Moderate in donor-facing content. |
| 7. Context mismatches | Cultural, geographic, or seasonal assumptions that don't fit the program. Common in pastoralist, conflict, fragile contexts. |
How long checking takes depends on the stakes and how many sources you must trace. A short internal note needs little; a donor report needs more. No fixed time or saving applies to every case. Skipping verification is how AI-assisted M&E work fails review. For the broader AI-assisted workflow, see writing logframe for proposal with AI.
When Does AI Output Verification Matter Most?
AI output verification is non-negotiable for:
Donor submissions. Proposals, MEL plans, evaluation reports, and compliance deliverables all fail donor review when they contain fabricated statistics, broken logic, or hallucinated framework claims. Reviewers catch these quickly because they read in the AI pattern.
Evaluation methodology. Methodology sections with AI-suggested counterfactuals, sampling designs, or analytical approaches must be verified against actual study feasibility. AI will cheerfully propose RCTs (randomized controlled trials, which assign people to groups by chance) for programs that cannot run them.
Compliance documents. Anything claiming alignment with ADS 201, MER, JMP (the WHO/UNICEF water and sanitation monitoring program), INEE (education in emergencies standards), or similar frameworks needs to be checked against the actual current framework version. AI training data lags; framework versions change.
Published deliverables. Blog posts, case studies, website content, and published evaluation summaries carry reputation risk. Fabricated numbers in public content damage credibility.
Lighter verification is acceptable for internal drafts, brainstorm outputs, and early-stage exploration where no one will act on the content without a human pass first.
What Happens When Nobody Checks?
A hallucination is text that sounds right but is invented. The risk is not only made-up facts. It also includes overconfident summaries, invented agreement between sources, and false precision. Professional settings have already paid for it.
- Courts. In 2023 a US federal court sanctioned two lawyers who filed a brief citing six court decisions that ChatGPT had invented (Mata v. Avianca). A public database kept by researcher Damien Charlotin lists hundreds of court and tribunal decisions worldwide that involved hallucinated AI content, and the count keeps growing.
- Evaluation consulting. In 2025 Deloitte Australia agreed to partly refund the Australian government for a review of a welfare compliance system worth about AU$440,000, after an academic found fabricated references and a fabricated quote from a court judgment. The corrected report disclosed that a generative AI tool had been used.
- Evaluation offices. The World Bank's Independent Evaluation Group and the International Fund for Agricultural Development's Independent Office of Evaluation tested large language models (the AI systems behind chatbots) on core evaluation tasks. The models classified text well, but completeness and relevance in extracting information and synthesizing it remained a challenge. Output can be good on average and still miss what matters.
The danger is greatest in qualitative synthesis (summarizing interviews and open-ended answers), literature review and reporting, where readers may never see the original material that would let them challenge the output. The control is workflow design, not general suspicion. Anchor every claim to a source, have a reviewer challenge the draft, and set rules for which tasks may rely on AI. Treat summary text as draft until it is checked against the evidence. If the chain from source to claim cannot be rebuilt, the claim should not reach the final product.
What Is the Short Version for a Small Team?
If you have one laptop and one afternoon, do these five steps on any AI draft before it leaves your desk, and keep the notes in a two-column table (field, your notes).
| Field | Your notes |
|---|---|
| Every number checked against its source file (done by whom) | |
| Every quote or citation found in the real document | |
| Any claim of "meets the donor standard" checked against the donor's own document | |
| Every spreadsheet formula or script tried by hand on 3 to 5 test rows | |
| Second person who read the full draft |
The seven checks in the next tab show how to do each step in detail.
The 7 Checks
Check 1: Fabricated Statistics
AI fills numerical fields with plausible-sounding numbers. Baseline values, targets, prevalence rates, research citations, and budget estimates all show the pattern.
What it looks like. A generated MEL plan says "38% of households in the target region practice safe water storage at baseline." Looks specific. Sounds defensible. Was invented. If the AI did not have access to your baseline data, it did not have the baseline. If it did not have access to current donor reference data, it did not have the current baseline rate.
How to detect. Any specific numerical claim in AI output needs a source citation or it is suspect. Ask: where does this number come from? If you cannot trace the number to an actual supplied dataset or an inspected primary source that supports the exact claim, population, period, and units, treat it as fabricated.
How to fix. Replace every AI-invented number with one of: (a) your actual data, (b) a cited source from published reference, or (c) an explicit placeholder ("baseline: TBD, to be collected in program year 1 via KAP survey"). Never ship AI-generated numbers as if they were real.
Check 2: Hallucinated Framework Claims
AI will claim an indicator "aligns with a named framework indicator" or "meets JMP safely managed criteria" without having actually checked. Framework names are common enough in training data that AI recognizes them; the specific current definitions are not.
What it looks like. A logframe draft includes "Indicator: Number of men and women with improved utilization of family planning services (MER PREV_FP_ACCEPT)." The indicator number looks real. The description matches a pattern. The actual MER indicator does not exist with that code.
How to detect. Pull up the actual framework document (ADS 201, MER Indicator Handbook, JMP service ladders, OECD DAC criteria, etc.) and verify every claimed indicator code, name, and definition. Takes 2-5 minutes per claimed alignment.
How to fix. Remove or correct any framework claims that don't check out. If an indicator is custom, label it as custom. If it maps approximately to a standard framework indicator, note the approximation rather than claiming alignment.
Check 3: Logic Breaks
AI-generated theories of change and logframes often have broken vertical logic: outputs labeled as outcomes, outcomes that don't plausibly produce the stated long-term outcome, activities aggregated strangely into outputs.
What it looks like. The outcome row says "500 women trained in business skills." That is an output (a count of deliverables), not an outcome (an applied change). The outcome should read something like "Trained women apply business skills to start or grow enterprises within 6 months."
How to detect. Walk the matrix bottom-up. Do activities produce outputs? Do outputs produce outcomes? Do outcomes contribute to the goal? Where any step breaks (or where outputs and outcomes are confused), the matrix needs revision. See how to write a logframe for the full vertical-logic test.
How to fix. Rewrite the confused rows manually. Do not prompt the AI to fix the logic without first articulating what is broken; AI tends to produce a second draft with the same structural error in slightly different wording.
Check 4: Vague Specifics
AI produces placeholder-quality specifics where precise specifics are needed. The most common: means of verification written as "survey" or "monitoring data" with no instrument, frequency, or responsibility specified.
What it looks like. Logframe column 3 says "Survey." Logframe assumption says "External conditions remain supportive." Training description says "Appropriate training for beneficiaries." All sound reasonable. None are executable.
How to detect. For every specific field in AI output, ask: is this detailed enough that someone picking up the document in 6 months could execute it? If the answer is no, it is a placeholder, not a specification.
How to fix. Replace vague language with named instruments, specific frequencies, and named responsibilities. "Survey" becomes "Annual household survey (KAP module, September) administered by field enumerator team, reviewed by M&E manager." See means of verification for the specificity bar.
Content & Compliance
Check 5: Generic Content Disconnection
AI produces fluent text that could describe many programs but doesn't specifically describe yours. The warning sign: reading the AI output and feeling like it could have been written for a different program with minimal editing.
What it looks like. A MEL plan narrative describes a generic "community health initiative" with standard methods, standard risks, and standard mitigation. Nothing in the text ties to your specific geography, partner relationships, delivery model, or theory of change. The program could swap the name and it would fit another program.
How to detect. Read through the AI draft asking: what is specific to this program? If the answer is "the program name and location," the content is disconnected. Compare the AI draft to your theory of change, your program description, and your partner set. If the AI draft does not reflect those specifics, it is generic.
How to fix. Rewrite to tie every section to your program's distinctive features. This is not optional; generic content reads as inexperience to donor reviewers and does not pass rigorous evaluation planning.
Check 6: Donor Compliance Errors
AI training data reflects framework versions from the training cutoff. Donor frameworks update regularly: ADS 201 chapters revise, MER indicators update annually, the SPHERE Handbook was last revised in 2018, SDG indicators get technical revisions most years. AI commonly cites outdated versions confidently.
What it looks like. A proposal MEL section cites "ADS 201.3.2.4 requires..." but the current chapter structure has different numbering. Or a reference-sheet template claims requirements without the applicable agreement or source version. Or a SPHERE-aligned indicator uses pre-2018 wording.
How to detect. For any framework citation, pull up the current framework document (donor website, UN body, sector coordination platform) and verify the version, chapter number, and specific requirement. Particularly important for bilateral-donor submissions where reviewers know the current framework cold.
How to fix. Update citations to current versions. If the AI output reflects outdated guidance, rewrite the affected sections. Check for currency at every submission cycle, not just the first time.
Check 7: Context Mismatches
AI defaults to generic development-sector assumptions. Pastoralist contexts, conflict-affected contexts, urban informal settlements, and programs in countries with non-standard administrative structures all trip up AI in predictable ways.
What it looks like. A food security MEL plan for pastoralist communities assumes monthly data collection rounds tied to calendar months. Pastoralists are transhumant; calendar months don't map to food security cycles for them. Or a governance program in a conflict-affected area uses "community consultation" methods that assume a stable civic space the context does not have.
How to detect. For every method, frequency, and assumption in AI output, check: does this fit the context you actually work in? If the AI has applied a generic development-sector template to a non-standard context, it needs revision.
How to fix. Rewrite the methods, frequencies, and assumptions to fit your context. Run the revised sections past a local M&E advisor or sector specialist before finalizing.
Sector Examples
Sector Examples
The scenarios below are illustrative. Their figures, costs, and review outcomes are teaching assumptions, not reported studies or evidence of donor approval.
Health: Vaccination Coverage Survey, East Africa
A health program used AI to draft the methodology section for a vaccination coverage survey. The AI produced a clean methodology citing WHO EPI cluster sampling and specific design effect calculations (the extra sample needed because people in the same cluster tend to answer alike). Verification caught two issues: (a) the specific DEFF (design effect) value cited (1.6) had no source, and typical DEFF for the program's geography is 1.8-2.0; (b) the AI claimed the methodology "aligns with WHO EPI cluster sampling methodology" but the actual WHO EPI manual revised its guidance in 2018 and the AI output reflected pre-2018 conventions. Revision took 45 minutes; the original draft took 20 minutes to generate. The final methodology was defensible in donor review.
Education: Learning Outcomes Assessment, South Asia
A program drafted a learning outcomes assessment section using AI. The AI produced a methodology aligned to "INEE standards" and cited specific assessment instruments. Verification found that two cited instruments were real but for different age groups than the program targeted, one "INEE minimum standard" citation was to an outdated version, and the proposed sampling design assumed equal school sizes when actual schools varied 50-400 students. Revision required ~90 minutes across the 3 issues. Final design was appropriate to context.
WASH: Rural Water Program, West Africa
A WASH program drafted a MEL plan section using AI. The AI proposed quarterly household surveys across 80 villages with a sample of 2,000 households per round. Verification caught that (a) the proposed sample size was not supported by the M&E budget (which would cover roughly 800 households per round at local rates), (b) the AI claimed JMP service ladder alignment but used pre-JMP-revision category definitions, and (c) the means of verification was listed as "survey" without any instrument specified. Revision cut the sample to feasible levels, updated JMP definitions, and specified the actual instrument. Took ~60 minutes.
Food Security: Pastoralist Livelihoods, Sahel
A livelihoods program used AI to draft the seasonal data collection schedule. The AI produced a standard quarterly schedule tied to calendar quarters. Verification with a local M&E advisor caught the context mismatch: the communities follow a transhumance pattern with migration phases that don't map to calendar quarters. Q2 (April-June) corresponds to late dry season and early migration, not to a stable data collection window. Revision restructured the schedule around migration phases (pre-migration, migration, post-migration, wet season) and rewrote indicator specifications to support seasonal disaggregation. Took ~2 hours including advisor consultation.
Mistakes & Checklist
Common Mistakes
Mistake 1: Skipping verification under deadline pressure. When the proposal is due in 48 hours, the temptation is to ship AI output with minimal review. This is the single most damaging habit; verification time is not optional. Build it into the workflow up front.
Mistake 2: Treating AI outputs as "mostly right." AI output is not mostly right. It is fluent. Those are different qualities. An AI-drafted logframe may have the right structure and an unknown share of the specifics correct. Verification is not polish; it is substantive quality control.
Mistake 3: Not running SMART checks on AI-suggested indicators. Every AI-suggested indicator should run through SMART validation before acceptance. See SMART indicators deep-dive. Skipping this step means vague-verb indicators slip into the MEL plan.
Mistake 4: Accepting AI-generated targets. Targets in AI output are placeholders. Every target needs to be validated against your baseline and your program's actual capacity. Targets copied unverified from AI drafts are how programs commit to unachievable deliverables.
Mistake 5: Trusting framework citations. AI cites frameworks confidently and often incorrectly. Every framework claim (ADS chapter, MER indicator code, JMP category, DAC criterion, INEE standard, SPHERE minimum) needs to be verified against the current framework document.
Mistake 6: Not checking quantitative claims. Percentages, sample sizes, budget allocations, and timelines all get fabricated by AI. Every number needs a source or a flag that it is a placeholder.
Mistake 7: Using AI output for final donor submission without human rewrite. Even verified AI output reads as AI output. For final donor submission, a human rewrite pass that adjusts voice, adds program-specific texture, and smooths structural quirks is not optional.
Mistake 8: Confusing AI fluency for AI accuracy. AI writes fluently. Donor reviewers read quickly. The combination creates false confidence that the document is right. Slow down and verify; fluency is not a quality signal.
AI Output Verification Checklist
Run through this for any AI-assisted M&E deliverable before submission.
For AI-drafted indicators:
- Every indicator passes SMART validation (specific, measurable, achievable, relevant, time-bound)
- Every claimed framework alignment (the applicable framework, JMP, Sphere, SDG, etc.) verified against current framework document
- Every baseline value verified against actual data or flagged as placeholder
- Every target checked against baseline and program capacity
- Every data source is feasible given program budget
For AI-drafted logframes and theories of change:
- Vertical logic walked bottom-up; each row plausibly produces the row above it
- Outputs and outcomes not confused
- Every means of verification specifies an actual instrument + frequency + responsibility
- Every assumption is testable during implementation
- Theory of change and logframe are internally consistent
For AI-drafted narrative and reports:
- Text is specific to this program, not generic
- Program specifics (geography, theory of change, partners, delivery model) reflected throughout
- Every quantitative claim has a source or is explicitly flagged
- Framework citations verified against current versions
- Voice consistency checked against organization standards
For AI-drafted spreadsheet formulas and scripts:
- Every function name exists in your version of Excel or the language (AI invents plausible ones)
- Every column header or cell range the formula refers to exists in your file
- Result matches a hand calculation on 3 to 5 test rows
- Blank cells, text in number columns, and duplicate keys tested, not just clean rows
For AI-drafted data collection plans:
- Methods feasible given program budget and capacity
- Sample sizes calculated correctly (design effect applied, non-response buffer included)
- Sampling approach fits program context (pastoralist, conflict, urban informal, etc.)
- Instruments named specifically, not "survey" or "questionnaire"
Pre-ship gates:
- Full document re-read by human, not just spot-checked
- Context appropriateness confirmed with local advisor if non-standard context
- Donor compliance requirements confirmed against current donor framework
- Human rewrite pass completed for donor-facing content (voice, polish, program-specific texture)
For the broader AI-assisted drafting workflow, see writing logframe for proposal with AI and how to write a MEL plan. For the quality gate framework AI verification fits within, see data quality assurance and the 5 data quality dimensions. For the AI-assisted step-by-step playbooks this verifies output from, see the playbooks library.