A risk score can help a reviewer decide where to spend attention. It can also become a decorative number that no one trusts.
The difference is explainability.
“Risk: 74” is not a decision. “High risk because authentication behavior changed, one requirement is unverified, and the affected checkout path has no connected test” is.
Risk is consequence multiplied by uncertainty
Large diffs are not automatically risky, and small diffs are not automatically safe. A one-line permission change can have more consequence than a generated-file rewrite.
A practical model separates two dimensions:
- Consequence: what important systems and behaviors the change can reach.
- Uncertainty: how much supporting evidence is missing, stale, or contradictory.
The Engineering Graph helps establish consequence. The evidence ledger helps establish uncertainty.
Use features reviewers recognize
Build the score from signals that can be explained in plain language:
| Signal | Why it matters |
|---|---|
| Sensitive path changed | Authentication, payments, migrations, secrets, and infrastructure have higher consequence |
| Downstream dependents | More affected services increase coordination and failure surface |
| Contract changed | API, schema, and event changes can cross repository boundaries |
| Coverage gap | A requirement or affected path lacks connected test evidence |
| Ownership gap | No current owner or experienced reviewer is established |
| Incident history | Similar components have caused production failures |
| Execution anomaly | An agent encountered blocked calls, failed commands, or unexpected file access |
| Change novelty | The path differs materially from prior safe changes |
Avoid opaque features that correlate with risk but cannot support an action.
Keep observed and inferred signals separate
“Migration file changed” can be observed directly. “This migration may lock a hot table” is an inference. “Request database review” is a recommendation.
Store them separately and show how each affects the score. This prevents a speculative model judgment from carrying the same weight as a failed test.
Score paths, then summarize the pull request
A pull request often contains several change paths with different risk.
PR #481
├─ auth/token.ts → HIGH: sensitive behavior + failed generated test
├─ docs/reset.md → LOW: documentation only
└─ telemetry.ts → MEDIUM: new outbound event + owner unresolvedScore the paths first. Aggregate the result using the highest-consequence unsupported path, not a simple average that lets safe documentation dilute a dangerous migration.
Calibrate with outcomes
The first weights will be assumptions. Treat them that way.
After merge, connect the analysis to outcomes: rollback, incident, failed deployment, bug report, or quiet operation. Compare predicted risk with observed results. Adjust weights carefully and preserve versions so old scores remain explainable.
Do not train on “merged” as a synonym for “safe.” Teams merge risky changes deliberately.
Make the score actionable
Every material contributor should map to a next step:
- missing contract coverage → run or add the named test;
- sensitive path → request the relevant owner;
- unresolved dependency → inspect the graph path;
- agent execution gap → connect or review the execution record;
- migration risk → verify rollback and backfill strategy.
If a signal cannot change reviewer behavior, question whether it belongs in the product.
Show the minimum useful interface
Lead with the verdict and decisive reasons:
Needs human review · High risk Authentication logic changed. Three of four requirements are verified. test_reused_token_rejected failed. The affected service has one current owner and no rollback evidence.Let the reviewer expand change surface, affected graph paths, requirements, tests, execution, and cost. The number can remain as a compact comparison tool, but it should never be the only explanation.
A score should compress evidence, not replace it
Risk scoring earns trust when an engineer can disagree with a factor, inspect its source, and correct the graph. That is the product standard: a concise signal at the top, durable evidence underneath, and no mystery in between.