The 7 AI Output Failure Modes
AI produces M&E content that reads fluently and often looks correct at first pass. It fails on inspection in seven predictable ways. Each has a specific detection method and a specific fix.
| # | Failure mode | What it looks like | How often it happens |
|---|---|---|---|
| 1 | Fabricated statistics | Baseline values, targets, or citations that sound specific but have no source | Most AI outputs with numerical fields |
| 2 | Hallucinated framework claims | Asserted compliance with a standard the AI never actually checked | Common when prompts ask for donor-aligned output |
| 3 | Logic breaks | Outputs labeled as outcomes; intermediate outcomes that don't produce the stated long-term outcome | Common in logframes and theory-of-change drafts |
| 4 | Vague specifics | "Survey" as a means of verification; placeholder language like "appropriate training" | Common in MEL plan and data collection content |
| 5 | Generic content disconnection | Text that reads right but does not reflect your program's specific theory of change | Moderate in narrative and proposal drafts |
| 6 | Donor compliance errors | Wrong ADS version, outdated MER indicators, fabricated donor requirements | Moderate in donor-facing content |
| 7 | Context mismatches | Cultural, geographic, or seasonal assumptions that don't fit the program | Common in pastoralist, conflict, fragile contexts |
A systematic check for each mode takes 30-40% of the time the AI saved in drafting. Skipping verification is how AI-assisted M&E work fails review. For the broader AI-assisted workflow, see writing logframe for proposal with AI.
When Verification Matters Most
AI output verification is non-negotiable for:
Donor submissions. Proposals, MEL plans, evaluation reports, and compliance deliverables all fail donor review when they contain fabricated statistics, broken logic, or hallucinated framework claims. Reviewers catch these quickly because they read in the AI pattern.
Evaluation methodology. Methodology sections with AI-suggested counterfactuals, sampling designs, or analytical approaches must be verified against actual study feasibility. AI will cheerfully propose RCTs for programs that cannot run them.
Compliance documents. Anything claiming alignment with ADS 201, MER, JMP, INEE, or similar frameworks needs to be checked against the actual current framework version. AI training data lags; framework versions change.
Published deliverables. Blog posts, case studies, website content, and published evaluation summaries carry reputation risk. Fabricated numbers in public content damage credibility.
Lighter verification is acceptable for internal drafts, brainstorm outputs, and early-stage exploration where no one will act on the content without a human pass first.
Check 1: Fabricated Statistics
AI fills numerical fields with plausible-sounding numbers. Baseline values, targets, prevalence rates, research citations, and budget estimates all show the pattern.
What it looks like. A generated MEL plan says "38% of households in the target region practice safe water storage at baseline." Looks specific. Sounds defensible. Was invented. If the AI did not have access to your baseline data, it did not have the baseline. If it did not have access to current donor reference data, it did not have the current baseline rate.
How to detect. Any specific numerical claim in AI output needs a source citation or it is suspect. Ask: where does this number come from? If you cannot trace the number to a file you uploaded, a URL the AI referenced, or a widely published statistic, treat it as fabricated.
How to fix. Replace every AI-invented number with one of: (a) your actual data, (b) a cited source from published reference, or (c) an explicit placeholder ("baseline: TBD, to be collected in program year 1 via KAP survey"). Never ship AI-generated numbers as if they were real.
Check 2: Hallucinated Framework Claims
AI will claim an indicator "aligns with USAID MER PREV_FP_ACCEPT" or "meets JMP safely managed criteria" without having actually checked. Framework names are common enough in training data that AI recognizes them; the specific current definitions are not.
What it looks like. A logframe draft includes "Indicator: Number of men and women with improved utilization of family planning services (MER PREV_FP_ACCEPT)." The indicator number looks real. The description matches a pattern. The actual MER indicator does not exist with that code.
How to detect. Pull up the actual framework document (ADS 201, MER Indicator Handbook, JMP service ladders, OECD DAC criteria, etc.) and verify every claimed indicator code, name, and definition. Takes 2-5 minutes per claimed alignment.
How to fix. Remove or correct any framework claims that don't check out. If an indicator is custom, label it as custom. If it maps approximately to a standard framework indicator, note the approximation rather than claiming alignment.
Check 3: Logic Breaks
AI-generated theories of change and logframes often have broken vertical logic: outputs labeled as outcomes, outcomes that don't plausibly produce the stated long-term outcome, activities aggregated strangely into outputs.
What it looks like. The outcome row says "500 women trained in business skills." That is an output (a count of deliverables), not an outcome (an applied change). The outcome should read something like "Trained women apply business skills to start or grow enterprises within 6 months."
How to detect. Walk the matrix bottom-up. Do activities produce outputs? Do outputs produce outcomes? Do outcomes contribute to the goal? Where any step breaks (or where outputs and outcomes are confused), the matrix needs revision. See how to write a logframe for the full vertical-logic test.
How to fix. Rewrite the confused rows manually. Do not prompt the AI to fix the logic without first articulating what is broken; AI tends to produce a second draft with the same structural error in slightly different wording.
Check 4: Vague Specifics
AI produces placeholder-quality specifics where precise specifics are needed. The most common: means of verification written as "survey" or "monitoring data" with no instrument, frequency, or responsibility specified.
What it looks like. Logframe column 3 says "Survey." Logframe assumption says "External conditions remain supportive." Training description says "Appropriate training for beneficiaries." All sound reasonable. None are executable.
How to detect. For every specific field in AI output, ask: is this detailed enough that someone picking up the document in 6 months could execute it? If the answer is no, it is a placeholder, not a specification.
How to fix. Replace vague language with named instruments, specific frequencies, and named responsibilities. "Survey" becomes "Annual household survey (KAP module, September) administered by field enumerator team, reviewed by M&E manager." See means of verification for the specificity bar.
Check 5: Generic Content Disconnection
AI produces fluent text that could describe many programs but doesn't specifically describe yours. The warning sign: reading the AI output and feeling like it could have been written for a different program with minimal editing.
What it looks like. A MEL plan narrative describes a generic "community health initiative" with standard methods, standard risks, and standard mitigation. Nothing in the text ties to your specific geography, partner relationships, delivery model, or theory of change. The program could swap the name and it would fit another program.
How to detect. Read through the AI draft asking: what is specific to this program? If the answer is "the program name and location," the content is disconnected. Compare the AI draft to your theory of change, your program description, and your partner set. If the AI draft does not reflect those specifics, it is generic.
How to fix. Rewrite to tie every section to your program's distinctive features. This is not optional; generic content reads as inexperience to donor reviewers and does not pass rigorous evaluation planning.
Check 6: Donor Compliance Errors
AI training data reflects framework versions from the training cutoff. Donor frameworks update regularly: ADS 201 chapters revise, MER indicators update annually, SPHERE standards revised in 2018 and 2024, SDG indicators get technical revisions most years. AI commonly cites outdated versions confidently.
What it looks like. A proposal MEL section cites "ADS 201.3.2.4 requires..." but the current chapter structure has different numbering. Or a PIRS template includes fields USAID no longer requires. Or a SPHERE-aligned indicator uses pre-2018 wording.
How to detect. For any framework citation, pull up the current framework document (donor website, UN body, sector coordination platform) and verify the version, chapter number, and specific requirement. Particularly important for bilateral-donor submissions where reviewers know the current framework cold.
How to fix. Update citations to current versions. If the AI output reflects outdated guidance, rewrite the affected sections. Check for currency at every submission cycle, not just the first time.
Check 7: Context Mismatches
AI defaults to generic development-sector assumptions. Pastoralist contexts, conflict-affected contexts, urban informal settlements, and programs in countries with non-standard administrative structures all trip up AI in predictable ways.
What it looks like. A food security MEL plan for pastoralist communities assumes monthly data collection rounds tied to calendar months. Pastoralists are transhumant; calendar months don't map to food security cycles for them. Or a governance program in a conflict-affected area uses "community consultation" methods that assume a stable civic space the context does not have.
How to detect. For every method, frequency, and assumption in AI output, check: does this fit the context you actually work in? If the AI has applied a generic development-sector template to a non-standard context, it needs revision.
How to fix. Rewrite the methods, frequencies, and assumptions to fit your context. Run the revised sections past a local M&E advisor or sector specialist before finalizing.
Sector Examples
Health: Vaccination coverage survey, East Africa
A health program used AI to draft the methodology section for a vaccination coverage survey. The AI produced a clean methodology citing WHO EPI cluster sampling and specific design effect calculations. Verification caught two issues: (a) the specific DEFF value cited (1.6) had no source, and typical DEFF for the program's geography is 1.8-2.0; (b) the AI claimed the methodology "aligns with WHO EPI cluster sampling methodology" but the actual WHO EPI manual revised its guidance in 2020 and the AI output reflected pre-2020 conventions. Revision took 45 minutes; the original draft took 20 minutes to generate. The final methodology was defensible in donor review.
Education: Learning outcomes assessment, South Asia
A program drafted a learning outcomes assessment section using AI. The AI produced a methodology aligned to "INEE standards" and cited specific assessment instruments. Verification found that two cited instruments were real but for different age groups than the program targeted, one "INEE minimum standard" citation was to an outdated version, and the proposed sampling design assumed equal school sizes when actual schools varied 50-400 students. Revision required ~90 minutes across the 3 issues. Final design was appropriate to context.
WASH: Rural water program, West Africa
A WASH program drafted a MEL plan section using AI. The AI proposed quarterly household surveys across 80 villages with a sample of 2,000 households per round. Verification caught that (a) the proposed sample size was not supported by the M&E budget (which would cover roughly 800 households per round at local rates), (b) the AI claimed JMP service ladder alignment but used pre-JMP-revision category definitions, and (c) the means of verification was listed as "survey" without any instrument specified. Revision cut the sample to feasible levels, updated JMP definitions, and specified the actual instrument. Took ~60 minutes.
Food security: Pastoralist livelihoods, Sahel
A livelihoods program used AI to draft the seasonal data collection schedule. The AI produced a standard quarterly schedule tied to calendar quarters. Verification with a local M&E advisor caught the context mismatch: the communities follow a transhumance pattern with migration phases that don't map to calendar quarters. Q2 (April-June) corresponds to late dry season and early migration, not to a stable data collection window. Revision restructured the schedule around migration phases (pre-migration, migration, post-migration, wet season) and rewrote indicator specifications to support seasonal disaggregation. Took ~2 hours including advisor consultation.
Common Mistakes
Mistake 1: Skipping verification under deadline pressure. When the proposal is due in 48 hours, the temptation is to ship AI output with minimal review. This is the single most damaging habit; verification time is not optional. Build it into the workflow up front.
Mistake 2: Treating AI outputs as "mostly right." AI output is not mostly right. It is fluent. Those are different qualities. An AI-drafted logframe may have the right structure and 40% of the specifics correct. Verification is not polish; it is substantive quality control.
Mistake 3: Not running SMART checks on AI-suggested indicators. Every AI-suggested indicator should run through SMART validation before acceptance. See SMART indicators deep-dive. Skipping this step means vague-verb indicators slip into the MEL plan.
Mistake 4: Accepting AI-generated targets. Targets in AI output are placeholders. Every target needs to be validated against your baseline and your program's actual capacity. Targets copied unverified from AI drafts are how programs commit to unachievable deliverables.
Mistake 5: Trusting framework citations. AI cites frameworks confidently and often incorrectly. Every framework claim (ADS chapter, MER indicator code, JMP category, DAC criterion, INEE standard, SPHERE minimum) needs to be verified against the current framework document.
Mistake 6: Not checking quantitative claims. Percentages, sample sizes, budget allocations, and timelines all get fabricated by AI. Every number needs a source or a flag that it is a placeholder.
Mistake 7: Using AI output for final donor submission without human rewrite. Even verified AI output reads as AI output. For final donor submission, a human rewrite pass that adjusts voice, adds program-specific texture, and smooths structural quirks is not optional.
Mistake 8: Confusing AI fluency for AI accuracy. AI writes fluently. Donor reviewers read quickly. The combination creates false confidence that the document is right. Slow down and verify; fluency is not a quality signal.
AI Output Verification Checklist
Run through this for any AI-assisted M&E deliverable before submission.
For AI-drafted indicators:
- Every indicator passes SMART validation (specific, measurable, achievable, relevant, time-bound)
- Every claimed framework alignment (USAID F, MER, JMP, SPHERE, SDG, etc.) verified against current framework document
- Every baseline value verified against actual data or flagged as placeholder
- Every target checked against baseline and program capacity
- Every data source is feasible given program budget
For AI-drafted logframes and theories of change:
- Vertical logic walked bottom-up; each row plausibly produces the row above it
- Outputs and outcomes not confused
- Every means of verification specifies an actual instrument + frequency + responsibility
- Every assumption is testable during implementation
- Theory of change and logframe are internally consistent
For AI-drafted narrative and reports:
- Text is specific to this program, not generic
- Program specifics (geography, theory of change, partners, delivery model) reflected throughout
- Every quantitative claim has a source or is explicitly flagged
- Framework citations verified against current versions
- Voice consistency checked against organization standards
For AI-drafted data collection plans:
- Methods feasible given program budget and capacity
- Sample sizes calculated correctly (design effect applied, non-response buffer included)
- Sampling approach fits program context (pastoralist, conflict, urban informal, etc.)
- Instruments named specifically, not "survey" or "questionnaire"
Pre-ship gates:
- Full document re-read by human, not just spot-checked
- Context appropriateness confirmed with local advisor if non-standard context
- Donor compliance requirements confirmed against current donor framework
- Human rewrite pass completed for donor-facing content (voice, polish, program-specific texture)
For the broader AI-assisted drafting workflow, see writing logframe for proposal with AI and how to write a MEL plan. For the quality gate framework AI verification fits within, see data quality assurance and the 5 data quality dimensions. For the AI-assisted step-by-step playbooks this verifies output from, see the playbooks library.