AI Code Review
Type: ExplanationCreated: Team: Engineering
draft
Overview
No off-the-shelf Claude review option fits our setup. This document explains why, and what we should build instead.
Scope note: we do not review PRs from outside contributors. Everything below covers our own team's PRs only.
1. The standard ways to review code with Claude
| Option | Trigger | Cost | Pros | Cons |
|---|---|---|---|---|
/code-review in Claude Code | Developer runs it | Plan usage | No setup, any git host, can auto-fix, cost spread across each dev's seat | Manual; single-repo; ignores REVIEW.md |
/code-review ultra | Developer runs it | $5–25/run in credits | Deepest review available | Paid per run; Team gets no free runs |
| Managed Code Review (Claude GitHub App) | Automatic per repo | $15–25/review in credits | Zero maintenance, highest quality, tunable via REVIEW.md | Paid; GitHub only; single-repo; no external context |
claude-code-action (DIY GitHub Actions) | Automatic on PR events | Free on subscription OAuth token | Full control; can integrate external systems | A toolkit, not a solution — see §2 |
@claude mentions | Developer comments on a PR | Plan usage | Great for follow-up questions; can implement fixes | Not a gate; human-initiated |
2. Why the simple options don't work for us
Our requirements:
- Automatic on every PR
- Funded by existing Team Premium seat usage — no new spend
- Composite context: a feature spans the main repo plus multiple submodules
- Cross-org: public main repo in one GitHub org, private submodules in another
- Reviews against the Skapp PM module task's acceptance criteria
- Reviews UI work against the Figma design
| Option | 1. Auto | 2. Free | 3. Submodules | 4. Cross-org | 5/6. External |
|---|---|---|---|---|---|
Local /code-review | ✗ | ✓ | ✗ | ✓ | ✗ |
ultra | ✗ | ✗ | ✗ | ✓ | ✗ |
| Managed Code Review | ✓ | ✗ | ✗ | partial | ✗ |
@claude mentions | ✗ | ✓ | ✗ | ✓ | ✗ |
| DIY GitHub Actions | ✓ | ✓ | only if we build it | only if we build it | only if we build it |
The blockers:
- Cost rules out the two best options. Managed Code Review and
ultrabill usage credits separately from plan usage. Neither can be funded from a Team seat. - Nothing understands our submodule layout. Every packaged option scopes to one repository's diff. Our parent PR is usually just a pointer bump, and a review of one submodule can't see the interface it breaks in another.
- The cross-org split is a security problem. Reviewing private submodules from the public repo means putting a private-org credential in a public repository's secrets and emitting private code into world-readable logs and public comments. Not acceptable.
- No packaged option sees our PM module or Figma. Managed Code Review reads the repo,
CLAUDE.md, andREVIEW.md— nothing external.
On DIY GitHub Actions
It looks like it passes, but only because three of its cells say "if we build it." claude-code-action is a toolkit for running Claude in CI, not a review product. Out of the box it reviews one repository's diff and knows nothing about our submodules, our PM module, or Figma — exactly the same ceiling as every other option in the table.
Specifically, adopting it as-is fails us three ways:
- The obvious placement is the insecure one. The natural instinct is to put the workflow in the main repo, where the parent PR lives. That is the one place it must not go, because it would need a private-org credential stored in a public repository. Making it safe means inverting the design so reviews run from the private side — a deliberate architecture decision, not a config flag.
- Everything of value to us is custom work. Submodule checkout across two orgs, branch discovery, splitting the review so cross-repo findings land somewhere useful, fetching PM tasks, fetching Figma nodes. None of this ships in the box.
- It concentrates cost on one person. The OAuth token that makes it free is tied to whoever generated it, so the whole team's reviews draw against one seat's usage limits. That is both a capacity ceiling and an unresolved licensing question.
So DIY Actions is not a fifth option alongside the others. It is the foundation of the custom solution in §3, and the work in §3 is what turns it from a toolkit into something that meets our requirements.
3. What will work
The three viable builds
| A. Actions, hosted runners | B. Actions, self-hosted on EC2 (recommended) | C. Local desktop poller | |
|---|---|---|---|
| How | Workflows in the private submodule repos only, authenticated with a subscription OAuth token | Same workflows, our own always-on runner registered to the private org | Script or scheduled task on a dev machine |
| Cost | $0–25/mo | $0 | |
| Effort | ~1 day | ~2–3 days | ~1 day to prototype |
| Gets us | 1, 2, 5, 6; partial 3 | All of A, plus safe cross-org credentials and cheap composite checkouts | A fast way to test whether the reviews are any good |
| Doesn't | Cross-org credential still has to reach the private repos | Must be registered to the private org only, never the public repo | One laptop becomes team infrastructure; fails silently |
Option C is a prototyping step, not a production process.
Two reviews - not one
Inline comments must attach to lines in that PR's diff, so a single job can't give both cross-repo context and correctly-placed comments. We run:
- Per-repo review — each submodule PR reviewed normally, comments inline where the author is working.
- Composite integration review — one run with all repos checked out, looking only at cross-repo risk (changed signatures with callers elsewhere, contract drift, migration ordering, dependency skew). Posts one summary comment.
External context
- Skapp PM module — we own it, so fetching a task is an internal integration with no third-party auth constraint. Low risk.
- Figma — the official Figma MCP is OAuth-only, rejects personal access tokens, and can't run headless. Use the Figma REST API instead. Realistic scope is token drift, missing variants and states, and text mismatches. True visual fidelity needs rendering and image diffing — a separate project.
Recommended plan
Build Option B, phased:
- Per-repo review on one submodule, hosted runners — two weeks measuring seat usage
- Add the composite integration review
- Add PM task acceptance-criteria checking
- Add Figma token and variant checking
- Move to the EC2 self-hosted runner once the workflow is stable
Open items
- Seat licensing — whether one Team Premium seat may fund automation reviewing other people's code, and whether a seat can be provisioned for a service account. Needs written confirmation from Anthropic. This gates the recommendation.
- Programmatic billing risk — a change moving Actions and SDK usage onto separate credits was announced for June 2026 and paused, with advance notice promised. Fallback on API-key billing is roughly $1–3 per review.
- Usage headroom — unknown until measured; that's what phase 1 is for.
- Zero Data Retention — if enabled on our account, several options are unavailable. To confirm.