At a Glance
An AI tool can read a transcript and return a list of themes in seconds. The hard question is whether each theme is true to the words. This page runs a trace on a fictional example, using the eight excerpts from from interview notes to findings. For general checks on AI work in M&E, see verifying AI outputs.
At a Glance
| Check | Question | Typical Failure |
|---|---|---|
| Verbatim | Does the quoted wording appear in the transcript? | A paraphrase presented as a quotation |
| Support | Does each cited excerpt say what the theme says? | An excerpt placed under a theme it contradicts |
| Overreach | Is the strength of the claim supported by the evidence cited? | "Most" from a handful of cases |
| Omission | What passages fit no theme? | A concern mentioned once is dropped |
| Contradiction | Which passages run against the theme? | Only supporting examples are shown |
| Stability | Does a rerun give the same result? | Themes shift between runs |
Worked Check
Worked Check: A Fictional AI Summary
Everything here is invented. The summary below was written for this page to show typical problems. It was not produced by a named model, and it is not evidence about the accuracy of any tool.
Imagine the eight excerpts (E1 to E8) were given to an AI tool with this request: "Identify the main themes in why participants left, and cite supporting quotations." The fictional reply:
Scheduling conflicts. "Participants left because of scheduling conflicts (E1, E2, E3, E5, E6)."
Transport costs. "Transport costs were a barrier (E4)."
Prevalence. "Most participants left because of scheduling."
A quotation. P1: "I felt behind and ashamed."
Facilitator support. "Facilitator support was the key factor in staying (E2)."
Summary line. "Overall, a later start time would solve the problem."
Tracing Each Suggestion
Open each excerpt and compare.
| AI Suggestion | Trace | Verdict | Action |
|---|---|---|---|
| 1. Scheduling conflicts, citing E1, E2, E3, E5, E6 | E1 and E6 describe a clash with the start time. E3 is mainly about care, though the 5:30 limit is time-related. E2 describes the clash but the person stayed. E5 says "the timing was fine." | Partly supported; E5 is misplaced and E2 is a counter-case | Remove E5. List E2 as a case where the clash did not lead to leaving. Split E3 into CARE and TIM |
| 2. Transport costs, citing E4 | E4 mentions the walk back in the dark. No excerpt mentions cost | Unsupported | Delete. Code E4 as travel safety instead |
| 3. "Most participants left because of scheduling" | Four people described a clash or care limit; one person left with no clash. The sample is six and was chosen on purpose | Overreach | Replace with a statement about the six interviews and drop "most" |
| 4. P1: "I felt behind and ashamed." | E1 says "I feel behind." The word "ashamed" does not appear | Altered quotation | Replace with the verbatim words. Never keep a quote you cannot find |
| 5. Facilitator support was the key factor in staying, citing E2 | One case. E4 also stayed and mentions no facilitator | Overreach | State as a possibility from one case |
| 6. A later start would solve the problem | E4 and E8 raise the walk home after dark; later starts could make it worse | Contradicted | Remove. Report the tension |
The omission check finds more. Ask the tool, or search yourself, for passages assigned to no theme. E7 (the person would return if children had somewhere to go) and E8 (a later start means a walk home after 8) are cited nowhere in the summary. Two other concerns appear only in distorted form: E5's difficulty with the material and embarrassment is filed under scheduling, and E4's walk home in the dark is filed as transport cost. The analyst, not the tool, decides whether these become new codes. Here they do, as in the other worked case.
Checks and Records
Checks to Run Every Time
Number the excerpts yourself before the tool sees them, and require IDs in every answer.
Search for each quotation in the transcript text.
Open every cited excerpt and write one line on whether it supports the label.
Ask for contradicting passages, then read them yourself. See negative case analysis.
List unassigned passages and decide what to do with each.
Compare claim wording to evidence. Words such as most, many, always and key need a basis in the data.
Run it again. Different runs can give different themes. Differences show where the tool is unstable.
Protect people. Remove names and identifying details, use only tools approved for the material, and confirm consent covers the use. See data management and ethics in M&E.
Keep interpretation human. In reflexive approaches, Braun and Clarke treat the researcher's interpretive work as the method itself; a tool's output does not replace it. See positionality and reflexivity.
What the Studies Suggest
The research is early, small and specific to particular tools. Morgan re-analyzed two datasets with ChatGPT and found that it reproduced concrete, descriptive themes better than subtle, interpretive ones. Tai and colleagues asked a model to apply a codebook and give evidence, ran the same texts 160 times to observe changes between runs, and compared the results with human coders. Xiao and colleagues combined a codebook with GPT-3 for deductive coding of one dataset and reported fair to substantial agreement with expert coding.
These studies do not show that a current tool will perform well on your transcripts, your languages or your codebook. A cautious design choice follows from them, though the studies do not test it: AI is used most defensibly for deductive coding against an explicit codebook, with evidence required, and with a human check. Agreement with a human coder, where measured, speaks to consistency in that design and does not show the interpretation is right.
Recording What You Did
Keep a short log with the analysis:
- Tool name and version, and the date.
- The exact prompt and the codebook or instructions supplied.
- Which excerpts were provided, by ID.
- What the tool returned.
- Each human change: suggestions rejected, quotations corrected, codes added.
- The memo that explains the final claim. See analytic memos.
If the log cannot be written, the use of the tool should not be reported as if it were part of a reviewable method.
Using It
Common Mistakes
Accepting a quotation because it sounds right. Fabricated or altered wording is the easiest error to miss.
Letting the tool decide the approach. It will code in whatever way the prompt implies, and a prompt that supplies a codebook asks for deductive coding. Name the approach first; see choosing a qualitative analytic tradition.
Treating agreement with the tool as validation. Agreement shows similarity, not truth.
Letting a fluent summary stand for analysis. Tidy narrative can hide missed cases.
Skipping the record. Without a log, no one can tell what the tool and the people each did.