Rubric Evidence
Two small synthetic studies. Finding the planted error was near the top with or without a rubric, and full repair differed in one study only.
Read This First
- Both studies use invented documents with planted problems. They test a narrow conformance task, not real reviews.
- Both are small and exploratory. No statistical test is reported and no ranking of prompts is claimed.
- Results are reported as counts with their denominators. Null results are shown beside the positive one.
What These Studies Do Not Show
- That rubrics reduce false accusations. The two studies disagree, and the newer one found none in either arm.
- That rubrics work in general, are valid, or are usable by practitioners.
- Results on real documents, on other models, or on calibrated scores.
Planted Errors in Report Excerpts, November 9, 2026
Question: When a model reviews a report excerpt that has one planted error, does a short rubric in the request help it find the error and propose a full repair?
Model: Claude Sonnet 5.5 (claude-sonnet-5-5), direct API
Cases and Denominators
- 11 synthetic report excerpt families, each with an altered version (one planted error) and an unaltered version.
- 2 repeats per excerpt and prompt, 44 replies per prompt, 88 coded replies in all. Repeats are not independent cases.
- Denominators below are replies: 22 on altered excerpts and 22 on unaltered excerpts for each prompt.
Results
| Outcome | Ordinary Review Prompt | Rubric-Guided Review Prompt |
|---|---|---|
| Planted Error FoundBoth prompts found it in most replies. At this level the study cannot show a difference. | 19 of 22 | 21 of 22 |
| Full Repair of the Error | 10 of 22 | 15 of 22 |
| Partial Repair | 10 of 22 | 6 of 22 |
| False Accusation on Unaltered ExcerptsNeither prompt made one, so this study says nothing for or against a rubric effect here. | 0 of 22 | 0 of 22 |
| Unsupported Findings | 0 of 381 | 0 of 367 |
- On 11 synthetic report excerpts with one planted error each, Sonnet 5.5 with the rubric fully repaired the error in 15 of 22 replies. Without it, 10 of 22.
- Counted by family, the rubric did better on full repair in 5 families, tied in 5 and did worse in 1. That meets the rule set before the run (at least 3 more wins than losses), by a small margin.
- Both prompts found the planted error in most replies (21 of 22 and 19 of 22).
Null and Ceiling Results
- False accusations on unaltered excerpts: 0 of 22 with the rubric and 0 of 22 without. No evidence either way.
- Finding the planted error: 21 of 22 against 19 of 22. Both near the top, so no detection benefit is claimed.
- Unsupported findings: none in either arm (0 of 367 and 0 of 381). This may reflect the model, or coders leaning toward "neutral" for weak points.
A Miss in Both Prompts: Excerpt F12
In excerpt F12, the introduction says "five additional groups" but lists four professions, and a later page says "four groups". The error was missed in 3 of the 4 altered-excerpt replies (2 of 2 without the rubric, 1 of 2 with it).
Limitations
- One model, 11 synthetic families, 2 repeats each. Exploratory and not power-calculated. No p-values are reported.
- Planted errors are mostly mechanical cross-checks (a number, word or count against another place in the excerpt). The rubric prompt carries a cross-checking rule, so a gain may come from that guidance and not from the rubric format.
- Coding was done by two Claude coders on Claude replies. There was no human reference coding. 22 of 88 replies were coded twice, with full agreement on detection, repair level and false accusation.
- Blinding was partial: the rubric wording can reveal the condition. Coders guessed the condition wrongly twice and correctly never.
Method Detail
- Run record: 111 model calls (19 diagnostic, 92 main), no refusals, no unparseable replies. Strict JSON came back in 0 of 88 replies; all were recoverable.
- Three scoring-only amendments (A1 to A3) were made after delivery checks stopped the run. Each changed how replies were scored, none changed what was sent to the model.
- Exact quotations: 877 of 879 (ordinary prompt) and 813 of 816 (rubric prompt).
Three Synthetic Case Families, November 5, 2026
Question: Does a compact rubric package change what a model finds and proposes when it reviews short synthetic documents, compared with an ordinary request and with equally detailed prose?
Model: GPT-4.1 (gpt-4.1-2025-04-14)
Cases and Denominators
- 6 synthetic inputs: an adequate and an altered or missing-evidence version of 3 case families (a success rate that must use all 60 enrolled households, unique households versus installation events, and an absent versus supplied data-collection annex).
- 2 responses per input per condition: 12 responses per condition, 6 on altered inputs and 6 on adequate inputs. 72 study responses plus 6 delivery checks.
- Denominators are responses, not documents.
Results
| Outcome | First Condition | Second Condition |
|---|---|---|
| Planted Target Found | 6 of 6ordinary request | 6 of 6compact rubric |
| Full Repair | 3 of 6ordinary request | 3 of 6compact rubric |
| False Accusation on Adequate InputThe ordinary request called a supplied annex missing twice. Those two responses are repeats of one input, not two independent cases. | 2 of 6ordinary request | 0 of 6compact rubric |
| Full RepairEqually detailed prose gave the same main outcomes as the rubric. | 2 of 6equal-detail prose | 2 of 6compact rubric |
| Full RepairMixed: false missing-evidence calls were 1 of 6 without examples and 0 of 6 with them, but complete numeric repairs were more common without. | 3 of 6rubric without examples | 2 of 6rubric with examples |
- Every condition found every planted problem (6 of 6 each), so detection cannot show improvement.
- The one visible difference: the ordinary request wrongly called a supplied annex missing in 2 of 6 adequate responses, the rubric in 0 of 6.
Null and Ceiling Results
- Detection: 6 of 6 in every condition.
- Rubric against equally detailed prose: no advantage for the rubric layout.
What Went Wrong Anyway
Strict JSON came back in 42 of 72 study responses; the other 30 needed recovery. Of 266 quotations in findings, 248 were exact text from the input and 18 were not. Reviews still raised unneeded concerns, for example asking for a frequency the document already stated.
Limitations
- Three case families, one model snapshot, 12 responses per condition. Repeats are not independent.
- The rubric package has several parts, so the comparison tests the package, not the layout alone.
- Scored against a key written before any output was seen, in one analyst’s non-blinded reading.
- These are conformance checks on invented documents, not a test of correct overall scores or of the library rubrics.
Method Detail
- Run record: 78 model calls, no retries, every response recoverable. Comparisons: A, ordinary request against compact rubric; B, rubric against equally detailed prose; C, rubric with and without worked examples.
Where the Two Studies Differ
| Question | November 5 | November 9 | Reading |
|---|---|---|---|
| False Accusations | GPT-4.1: 2 of 6 without rubric, 0 of 6 with | Sonnet 5.5: 0 of 22 and 0 of 22 | Not replicated. No support for the earlier signal on the newer model. |
| Full Repair | GPT-4.1: 3 of 6 and 3 of 6 | Sonnet 5.5: 10 of 22 without, 15 of 22 with | Conflict. A difference appears in the newer study and not in the pilot. |
| Detection | GPT-4.1: 6 of 6 in every condition | Sonnet 5.5: 19 of 22 and 21 of 22 | Consistent. Near the top in both. |
Use the Rubrics and Tell Us What You See
Try a rubric on your own document and compare it with what you would have flagged. The most useful feedback is specific: the document type, what the review caught, what it missed or invented, and whether the score matched your judgment. Please leave out confidential or client material.