Tracking Agent docs
How to set up the Tracking Agent
Set up the Tracking Agent in Datalyze's onboarding: connect GitHub, pick repos, add your Anthropic and analytics keys, open the setup PR and pay. Merging the setup pull request starts the first audit, which opens your first tracking plan as a pull request of its own. Merging that one approves the plan, and every pull request after it gets the PR check.
What do I need before setting up the Tracking Agent?
- A Datalyze account. Setup itself is free; pull requests are checked once you subscribe ($250 a month). Reviewers don't need an account: they sign in with GitHub.
- Admin access to the GitHub repos, so setup can install the Datalyze GitHub App, save secrets, open the setup pull request and, later, make the checks required.
- An Anthropic API key. Claude runs on your GitHub runner with it, and the usage is billed to your Anthropic account. Make one in the Anthropic Console, or name an organisation secret that already holds one.
- To block merges on a private repo, a paid GitHub plan (Pro, Team or Enterprise) on the account or organisation that owns it. Without it the agent still checks and reports on every pull request. Public repos work on any plan.
- Optional: a read-only key for Mixpanel, PostHog or Amplitude, so the audit can compare the plan with what actually arrives.
How do I set up the Tracking Agent?
- Connect GitHub. Sign in with GitHub at /tracking-agent/start. Your GitHub token stays on Datalyze's server, never in the page.
- Choose repos. Tick the repos to check. Only repos you're an admin of are listed.
- Anthropic key. Paste your Anthropic API key, or give the name of an organisation secret that holds it.
- Analytics. Pick Mixpanel, PostHog or Amplitude, the region and project ID, and a read-only key. Or pick None yet, and the agent works from the code alone.
- Google Sheet. Optional: share a Sheet with Datalyze and paste its link. The Google Sheet covers it.
- Context. Eight questions about your product: where tracking is sent, what it does and which actions matter, what's in scope, naming, privacy, volume, the properties every event needs, and the questions to answer first. The answers go in
.datalyze/context.md, which every run reads. Four more, and all twelve answers, are on the Setup page's Context for the agent row. - Install. Install the Datalyze GitHub App on the chosen repos when GitHub asks. Setup then saves the secrets to each repo, including a workspace key it makes and never shows.
- Setup pull request. Setup opens one pull request per repo. Its run checks the keys you saved:
datalyze/modelreads Anthropic's model list once, anddatalyze/analyticsmakes one read-only request to your analytics tool. No model runs. Later changes on the Setup page are checked the same way: a new tool, region or project opens a small pull request that runs both checks, and new keys alone start a shortcheckrun on your default branch. - Pay. Subscribe with Stripe, then merge the setup pull request.
Setup starts from defaults: anyone with write access may approve tracking, the Datalyze naming conventions (snake_case object_action names such as post_created), and no Google Sheet. Change the conventions in analytics/registry/_conventions.yaml; the playbook chapter on creating a tracking plan covers choosing them.
Which read-only analytics key do I make?
| Tool | Key to make | Secrets it's saved as |
|---|---|---|
| Mixpanel | A service account with the Consumer role on the project, plus the project ID and data region (US, EU or India) | MIXPANEL_SA_USERNAME, MIXPANEL_SA_SECRET |
| PostHog | A personal API key with read access to queries, plus the project ID and your PostHog host | POSTHOG_API_KEY |
| Amplitude | The project's API key and secret key, plus its region | AMPLITUDE_API_KEY, AMPLITUDE_SECRET_KEY |
A read-only key is enough: the agent never writes to your analytics tool. If the tool refuses the key, the audit says so and the dashboard shows Update your Mixpanel key (or PostHog, or Amplitude) until you save a new one on the Setup page.
What happens when the setup pull request is merged?
- The first audit starts by itself, on the push that brings the workflow to your default branch. If it didn't, use Run the first audit on the dashboard or the Setup page.
- It opens your first plan as a pull request from
datalyze/first-plan: the registry written from your code (and your analytics data, if connected), the tracking profile, the baseline and the audit. Itsdatalyze/trackingpasses with "First tracking plan: merging it approves it". - Merge it. That approves the plan. The audit's findings are on the audit screen, where you can pick the ones to fix.
- Go live on the Setup page. It adds a repository ruleset that makes
datalyze/trackinganddatalyze/rulesrequired on your default branch.
Until a repo goes live, the checks run and report on every pull request but don't block a merge. The setup pull request, and any pull request opened before the first plan exists, pass with "No tracking plan yet: the first audit writes it". The audit covers what the first audit writes.
What does the setup pull request add?
| File | What it is |
|---|---|
.github/workflows/datalyze.yml | The workflow. It runs on pull requests, on @datalyze comments, on each push to your default branch (to rebuild the Sheet) and on runs Datalyze starts (an audit, a build, fixes) |
.datalyze/config.yaml | Where the plan is, who may approve, and which analytics tool the audit reads |
.datalyze/context.md | What your team told the agent about the product. Edit it like any file, or on the Setup page |
.datalyze/.gitignore | Keeps the scan's cache out of git |
analytics/registry/_conventions.yaml | Your naming rules, over the Datalyze defaults |
CLAUDE.md | A short block that sends tracking questions to the plugin: "what does this branch need" runs /datalyze:review. It replaces v1's block in AGENTS.md where there is one |
.claude/settings.json | Offers the Datalyze plugin to anyone who opens the repo in Claude Code. Your other settings stay |
Nothing in your product code changes. The first audit adds the rest of the registry on its own pull request. Reference lists every file the agent keeps.
Which secrets does Tracking Agent setup save?
| Secret | What it holds | When |
|---|---|---|
DATALYZE_KEY | Your workspace key, made by setup and saved without being shown. Each install makes its own, and older repos keep theirs | Always |
ANTHROPIC_API_KEY | Your Anthropic API key | Unless the key is in an organisation secret you named |
MIXPANEL_SA_USERNAME, MIXPANEL_SA_SECRET | A Mixpanel service account (read-only) | With Mixpanel |
POSTHOG_API_KEY | A PostHog personal API key with read access | With PostHog |
AMPLITUDE_API_KEY, AMPLITUDE_SECRET_KEY | The Amplitude project's keys | With Amplitude |
DATALYZE_SHEET_URL, DATALYZE_SHEET_TOKEN | Your Sheet's Apps Script receiver and its token | Only with the Apps Script route for the Sheet |
Each value is encrypted in your browser with the repo's public key before it's sent to GitHub through Datalyze's server, so only GitHub can open it. GitHub explains how secrets in GitHub Actions are stored and who can read them.
Who can approve tracking changes?
By default, anyone with write access to the repo. To narrow it, list GitHub logins or @org/team under reviewers in .datalyze/config.yaml. The list is read from the pull request's base branch, so a pull request can't change who approves it, and it applies to both ways of approving: Build it in Datalyze and an @datalyze apply comment.
# .datalyze/config.yaml
registry: analytics/registry
reviewers:
- priya-dev
- "@acme/analytics"Why does setup install the Datalyze GitHub App?
Commits pushed with the default Actions token don't start other workflows, so your CI would never run on the agent's commits; GitHub documents this under automatic token authentication. Commits pushed with a GitHub App's token do start them, so the agent commits as the Datalyze GitHub App.
You install the App only on the repos you choose. Its private key stays on Datalyze's server, which gives each run a token for its own repo, valid for an hour. While your subscription is active, the token can push the agent's commits and post its checks; without one, it can only post the two checks as passing, so a live repo keeps merging. The ruleset accepts the two checks only from the Datalyze App, so no other token can turn them green.
How do I move a v1 repo to v2?
On the Setup page, a repo that still runs v1 shows Update the workflow. It opens one pull request, "Move the Datalyze Tracking Agent to v2", that changes the workflow, adds the CLAUDE.md block and offers the plugin in .claude/settings.json. Your .datalyze/config.yaml, .datalyze/context.md and tracking plan stay as they are.
v2 runs on Claude only, so the repo needs an ANTHROPIC_API_KEY secret first; save one under Anthropic API key on the Setup page. Until the move merges, a v1 run passes with "not checked". A _map/ folder v1 wrote in the registry is ignored by v2 and can be deleted.
Questions about setting up the Tracking Agent
Does Tracking Agent setup change our product code?
No. The setup pull request adds a workflow, the .datalyze folder, the naming conventions, a CLAUDE.md block and the plugin setting. Tracking calls change later, in pull requests a person approved, written the way your code already writes them.
Who needs admin access during setup?
Only the person running setup: installing the Datalyze GitHub App, saving repo secrets and going live all need admin rights. Reviewers need only their normal access to the repo.
We already have a tracking plan in a spreadsheet. Can setup import it?
No. The first audit writes the registry from the tracking calls in your code, and from your analytics data if you connect it. Copy anything worth keeping from the old spreadsheet into the registry on the first plan's pull request, or later.
Can Datalyze build our tracking before we add the check?
Yes. If the tracking itself needs designing or rebuilding, our analytics setup service does that work, and the Tracking Agent keeps it correct afterwards.