How to Evaluate Advocacy and Policy Influence Without Overclaiming

Why Advocacy Is Easy to Overclaim

Advocacy rarely works alone. Policy changes when several things line up: other organizations push the same message, an official is willing, the budget cycle opens a window, or a national rule forces action. The path is not a straight line, and it can take years. An advocacy team that sees its recommendation appear in a new policy is tempted to say "our work led to it." That may be partly true. It is seldom provable as stated.

Overclaiming costs credibility with the people who decide on funding and the officials the team needs to work with. The fix is not to claim less than happened. It is to show what happened at each step, say who else was involved, and match the strength of the claim to the strength of the evidence.

This page covers the evaluation of advocacy aimed at decision-makers. For the methods it relies on, see contribution analysis, outcome harvesting and attribution vs contribution.

Follow the Evidence Chain

Break the effort into links, from what the team did to what changed for people. For each link, decide what evidence would show it happened.

Two rules keep the claim honest. First, claim only up to the last link where you hold evidence. If you have evidence of access and attention but not position or adoption, the claim is about access and attention. Second, the earlier links are not proof of the later ones. A well-attended briefing does not show that a policy followed because of it.

Choose Measures for Each Link

Policy influence is easier to measure when the measure fits the link. Use a mix of counts, records and judgments.

A measure is not an end in itself. Counting briefings says what was done, not what was achieved. Choose a small number of measures for the links that matter most in your theory of change, and record the assumptions that must hold between them.

Methods That Fit Advocacy

Outcome harvesting. Starts from the changes observed in policy and practice, then works backward to find out whether and how the team contributed. It suits advocacy, where the outcomes cannot all be predicted. Each outcome is written as a statement (who changed what, when, and where) and checked with people who know, ideally some not connected to the team. See outcome harvesting.

Contribution analysis. Builds a contribution story from the theory of change, gathers evidence for each link, and tests other explanations. It ends with a statement of how strong the contribution case is. See contribution analysis.

Process tracing. Asks, for each key claim, what evidence would be found if the claim were true and what would be found if it were false, then looks for it. It helps when one decision is the focus. See process tracing.

Interviews with informed outsiders. Ask decision-makers, other advocates and observers about what shaped the decision. Include people who disagreed with the team, and record what they say about other influences.

A decision log. Keep a dated record from the start: contacts, meetings, drafts and changes in text. Evidence collected as events happen is far stronger than recollection years later.

Test the Other Explanations

For every claimed outcome, list the other things that could explain it, and look for evidence on each. A short table is enough.

If the policy would probably have happened anyway, say so. If the team's contribution was to speed it up or to shape the detail, that is a valid and useful finding. Check the assumptions in the theory of change as well: an advocacy theory often assumes decision-makers listen to evidence, and that assumption can be tested.

Word the Claim to Match the Evidence

Language should signal strength. A simple ladder:

Avoid "caused," "led to," and "was responsible for" unless the evidence supports sole or main cause. "Contributed to" is stronger when followed by how: what the team did, and what the evidence is. State who else was involved and what was not found.

Steps to Plan the Evaluation

1. Write the policy change sought, and by when it could realistically happen.
2. Draw the chain of links from activity to effect on people.
3. Choose evidence and a measure for each link that matters.
4. Set up a decision log and assign someone to keep it.
5. Plan how outcomes will be harvested and who will be asked to confirm them.
6. List the other explanations to test.
7. Review the evidence at set points and update the claim.
8. Write up the claim using the ladder, with the other actors named.

Worked Example: A District Health Budget Line

Healthy Futures is a fictional child nutrition program. Its team advocates for the district council to add a community nutrition screening line to the annual health budget.

The team's decision log shows three briefings to the district health committee (activity and contact). The committee chair then asks for a technical note (access). The note is cited in the minutes of a council session (attention). The next draft budget includes a screening line (adoption). The following year, funds for the line are released (implementation).

The team tests other explanations. A national health circular that year asks districts to plan for community screening. A local teachers' group also lobbied on child health. Interviews with a council member, a committee clerk and a teachers' group leader are used to find what shaped the decision.

The interviews suggest the circular made the line likely, and the briefings shaped how the line was designed: it targets children under two, as the team recommended. The team's claim is therefore: "The program contributed to the design of the screening budget line, alongside a national circular that required districts to plan for screening. Evidence: committee minutes, draft budget text and three interviews." It does not claim that the program caused the line to exist.

Common Mistakes

• Reporting outputs, such as briefings held, as if they were policy results.
• Claiming adoption without saying who else pushed for it.
• Taking the earlier links as proof of the later ones.
• Starting to collect evidence only after the policy changed.
• Interviewing only the team and its allies.
• Using "caused" where "contributed to" is all the evidence supports.