Tracking Agent docs
How the Tracking Agent PR check and Approve work
On every pull request, the Tracking Agent works out what tracking the change needs and writes it as a proposal. Most changes need none and pass with "No tracking change needed". When it proposes something, a person approves in Datalyze or with an @datalyze apply comment, and the agent builds the change on the branch, checks it, and reports.
What happens on a pull request?
- ReviewClaude reads the change and writes the proposal and a report comment.
- ApproveA reviewer keeps, edits or rejects each item, answers questions and approves.
- BuildThe agent changes the plan and the calls in your own wrappers.
- VerifyThe rules run, and the verifier checks the result cold.
- MergeYour team's normal review merges the pull request.
The review runs when a pull request is opened, pushed to, reopened or marked ready for review. Its proposal is a file on the branch, .datalyze/proposals/pr-<n>.json, so the decision stays in the pull request's history. The report goes in one comment, which later runs edit rather than add to.
What does the Tracking Agent's review check?
- What the change does for users, in product terms, before deciding anything.
- Every existing event, user property and identify or reset call it can reach, and how the change affects each one: a firing moment moved, a screen removed or merged, a step inserted, an enum or type changed, an event renamed in place.
- Each item in a fixed order: no change, extend, modify, deprecate, new. It stops at the first that fits.
- Whether a new event should exist: if nobody can write the Monday-morning question it answers, it isn't proposed. More than five new events, or two for one screen, need a written reason each.
- A rename or deprecation comes with its replacement event and a dual-fire date in the same proposal, so dashboards don't break on the day it ships.
- A new thing a person can do, such as an export button, always gets an item or a visible question, never a buried note.
Problems the change doesn't touch are left to the audit. The playbook chapter on creating a tracking plan covers deciding which events are worth tracking, the same test the review applies. If the tracking itself needs designing first, our analytics setup service builds it.
How do I approve a tracking proposal in Datalyze?
- Open the review screen from the pull request: the Awaiting approval status and the report comment both link to it, and the Review inbox lists every proposal waiting in your repos.
- For each change, choose Keep or Reject. Why opens it: edit the proposed entry, give a reason for a rejection (Datalyze reads it on its next review), or add a Note for Datalyze the build must follow.
- Answer the questions (the agent's recommended answer is picked already). Under Anything else, write whatever else you want done, in your own words.
- Save to stop and come back later: it commits your decisions to the branch without starting anything.
- Build it when you're done: it approves the proposal and builds it. Undecided items count as kept.
Build it commits the edited proposal to the pull request branch with your own GitHub sign-in, in a commit that starts Datalyze review: approved by @you. The run on that commit checks you have write access and are in reviewers, posts "Approved by @you, building it", and builds it. Your decisions aren't reviewed again.
How do I approve with a comment?
Comment @datalyze apply on the pull request. Every item not rejected counts as approved, and every answered question is binding. The same rule as Approve applies: you need write access, and to be in reviewers when that list is set.
Any other @datalyze comment on an issue or pull request is a free-form request, such as @datalyze why is checkout_started proposed?. It runs Claude with the plugin and the workflow's own token. If it pushes to the pull request, the new commit gets its own review.
What does the agent do after a proposal is approved?
- Builds exactly the approved items: the plan entries and the tracking calls, through the wrappers your code already uses, nothing else.
- Runs the rules (
validateandscan, judged by your base branch's rules) and fixes until they pass. It never edits the rules or_conventions.yamlto pass. - Gives the verifier only the code diff, the plan diff and your decisions, never its own reasoning. It applies the verifier's blockers and fixes, at most two rounds.
- Keeps every verified item and holds back only items with an open question, so one bad item never loses the rest. Held-back questions appear on the review screen to answer.
- Commits with the Datalyze GitHub App, so your CI runs on the commit, and posts the report. Its first line is
Implemented and verified,Implemented, N held backorNeeds a person.
What does each datalyze/tracking status mean?
| State | Description | When |
|---|---|---|
| success | No tracking change needed | The review found nothing the change needs. Nothing waits for a person |
| pending | Awaiting approval | A proposal is waiting. The status links to its review screen |
| pending | Approved by @login, building it | Someone approved in Datalyze, and the run on that commit is building it |
| success | Implemented and verified | Every approved item is built, the rules pass and the verifier signed off |
| success | Implemented, N held back | Every verified item is committed; the held-back items' questions are in the comment and on the review screen |
| failure | Needs a person: … | The rules still fail, the verifier's blockers are open after two rounds, or the run stopped early. The report comment says what |
| failure | Needs a person: commits after the proposal haven't been reviewed yet | A build was asked for while newer commits haven't had their review. Wait for it, then approve again |
| success | First tracking plan: merging it approves it | The first audit's plan pull request |
| success | No tracking plan yet: the first audit writes it | A pull request before the first plan, such as the setup pull request |
| success | Tracking Agent inactive, not checked | The subscription lapsed or the key was revoked. The pull request passes |
GitHub shows these as commit statuses on the pull request's head commit. A pull request from a fork gets no datalyze/tracking status at all: GitHub gives its run no secrets and no OIDC token, so it can't get the Datalyze App's token to post one. On a repo that has gone live, push the branch to the repo itself.
What does the datalyze/rules check do?
It runs on every push to a pull request, with no model: the plan's hard rules (naming, vocabulary, structure) and a scan of the code against the plan, judged by the rules on your base branch. It passes with "Passes", fails with the number of problems and the first one, and passes with "No tracking plan yet" before the first plan. "Couldn't run" means the base branch or its history wasn't checked out.
Which pull requests does the review skip?
| Pull request or commit | What happens |
|---|---|
The first plan (datalyze/first-plan) and fix pull requests (datalyze/fix-audit-<date>) | No review: the audit or the build already produced and verified them. datalyze/rules still runs |
| A commit the Datalyze GitHub App made | No review of its own commits |
| A Save or Approve commit from Datalyze on top of Datalyze's own commits | No new review: those are the reviewer's decisions |
A person's push to any branch, datalyze/ ones included | Reviewed as usual |
| A fork's pull request | No review and no datalyze/tracking status, since GitHub gives it no secrets |
Questions about the PR check
Does every pull request cost Claude usage?
Every pull request with a tracking plan gets a review, which runs Claude on your runner with your Anthropic key; most end with "No tracking change needed". Usage is billed to your Anthropic account, not by Datalyze.
Can the author approve their own tracking?
Yes, when reviewers is empty or names them. To require someone else, list the people or @org/team who may approve in .datalyze/config.yaml.
What if I disagree with a proposed event?
Reject it, with a reason. The next review reads the reason and won't propose it again unless the change alters why it was rejected.
What happens if someone pushes after I approve?
The new push gets its own review. A build waits until the commits under your decisions are ones Datalyze has reviewed, so nothing unreviewed is built.