Evidence-backed AI code review: what a trustworthy report needs

Most AI code reviews are summaries wearing the clothes of judgment.

They restate the diff, identify familiar patterns, and produce a confident paragraph. That can save time. It cannot establish that the change satisfies the task, respects an architectural decision, preserves a critical behavior, or came from a safe autonomous execution.

A trustworthy review needs evidence.

Begin with claims the system can support

Every statement in a review belongs in one of three categories:

1. Verified fact: directly supported by code, a test result, a repository event, an execution record, or another observed source.
2. Inference: a conclusion drawn from supported facts, with uncertainty made visible.
3. Recommendation: a proposed action based on facts and inferences.

This vocabulary prevents a fluent model from turning “I found a similar file” into “this service is definitely affected.”

Separate execution assurance from outcome assurance

AI-generated code creates two independent review surfaces.

A safe run can produce broken code. Passing tests can come from an agent that accessed data it should never have seen. A serious report keeps the layers separate and computes a verdict from both.

Extract requirements before judging the diff

The task is the beginning of the evidence ledger. Convert it into explicit, testable claims.

For “Implement expiration checks for password-reset tokens,” the requirements might be:

• valid, unexpired tokens remain accepted;
• expired tokens are rejected;
• successful use invalidates the token;
• existing login behavior remains unchanged.

Now the review can connect each requirement to evidence. Three supported requirements and one unverified requirement is useful. “Looks good overall” is not.

Connect the change to the Engineering Graph

Diff-only review misses the system around the diff. A graph-grounded review can trace:

changed function
  → CALLED_BY → authentication route
  → USES → token store
  → GOVERNED_BY → security decision
  → COVERED_BY → reset-token tests
  → OWNED_BY → Identity team

That path can expose sensitive systems, missing coverage, relevant reviewers, and historical context without stuffing the entire repository into a prompt.

Make missing evidence visible

“Not recorded” and “unverified” are legitimate results.

If the agent runtime was not connected, the report should not invent execution assurance. If generated tests did not run, their count should not be zero—it should be unknown. If a requirement has no supporting check, show the gap and its effect on the verdict.

Conservative truth builds more trust than artificial completeness.

A useful report is layered

The first screen should answer four questions:

• What changed?
• What does it affect?
• What evidence supports the analysis?
• What still needs a human?

Details can expand into requirements, test runs, findings, agent actions, evidence sources, and cost. The report should meet the reviewer at the decision, then let them inspect the path.

The verdict must explain itself

“High risk” is not useful unless the reviewer can see why. A verdict should name the decisive factors:

Needs human review. Authentication behavior changed. Three of four requirements are verified. Reused-token invalidation is unsupported, and one generated test for that behavior failed. No secrets or blocked network calls were observed.

That is specific enough to act on and compact enough to post in GitHub.

AI review should reduce uncertainty, not perform certainty

The goal is not to replace human judgment with model confidence. It is to arrive at human judgment with the code, history, relationships, requirements, and tests already organized.

That is the difference between another AI reviewer and an independent evidence layer. One comments on code. The other explains what the team can support, what it cannot, and why.