Tracking Agent docs

How to set up the Tracking Agent

Set up the Tracking Agent in Datalyze's onboarding: connect GitHub, pick repos, add your Anthropic and analytics keys, open the setup PR and pay. Merging the setup pull request starts the first audit, which opens your first tracking plan as a pull request of its own. Merging that one approves the plan, and every pull request after it gets the PR check.

By Ansh AgrawalUpdated

What do I need before setting up the Tracking Agent?

  • A Datalyze account. Setup itself is free; pull requests are checked once you subscribe ($250 a month). Reviewers don't need an account: they sign in with GitHub.
  • Admin access to the GitHub repos, so setup can install the Datalyze GitHub App, save secrets, open the setup pull request and, later, make the checks required.
  • An Anthropic API key. Claude runs on your GitHub runner with it, and the usage is billed to your Anthropic account. Make one in the Anthropic Console, or name an organisation secret that already holds one.
  • To block merges on a private repo, a paid GitHub plan (Pro, Team or Enterprise) on the account or organisation that owns it. Without it the agent still checks and reports on every pull request. Public repos work on any plan.
  • Optional: a read-only key for Mixpanel, PostHog or Amplitude, so the audit can compare the plan with what actually arrives.

How do I set up the Tracking Agent?

  1. Connect GitHub. Sign in with GitHub at /tracking-agent/start. Your GitHub token stays on Datalyze's server, never in the page.
  2. Choose repos. Tick the repos to check. Only repos you're an admin of are listed.
  3. Anthropic key. Paste your Anthropic API key, or give the name of an organisation secret that holds it.
  4. Analytics. Pick Mixpanel, PostHog or Amplitude, the region and project ID, and a read-only key. Or pick None yet, and the agent works from the code alone.
  5. Google Sheet. Optional: share a Sheet with Datalyze and paste its link. The Google Sheet covers it.
  6. Context. Eight questions about your product: where tracking is sent, what it does and which actions matter, what's in scope, naming, privacy, volume, the properties every event needs, and the questions to answer first. The answers go in .datalyze/context.md, which every run reads. Four more, and all twelve answers, are on the Setup page's Context for the agent row.
  7. Install. Install the Datalyze GitHub App on the chosen repos when GitHub asks. Setup then saves the secrets to each repo, including a workspace key it makes and never shows.
  8. Setup pull request. Setup opens one pull request per repo. Its run checks the keys you saved: datalyze/model reads Anthropic's model list once, and datalyze/analytics makes one read-only request to your analytics tool. No model runs. Later changes on the Setup page are checked the same way: a new tool, region or project opens a small pull request that runs both checks, and new keys alone start a short check run on your default branch.
  9. Pay. Subscribe with Stripe, then merge the setup pull request.

Setup starts from defaults: anyone with write access may approve tracking, the Datalyze naming conventions (snake_case object_action names such as post_created), and no Google Sheet. Change the conventions in analytics/registry/_conventions.yaml; the playbook chapter on creating a tracking plan covers choosing them.

Which read-only analytics key do I make?

ToolKey to makeSecrets it's saved as
MixpanelA service account with the Consumer role on the project, plus the project ID and data region (US, EU or India)MIXPANEL_SA_USERNAME, MIXPANEL_SA_SECRET
PostHogA personal API key with read access to queries, plus the project ID and your PostHog hostPOSTHOG_API_KEY
AmplitudeThe project's API key and secret key, plus its regionAMPLITUDE_API_KEY, AMPLITUDE_SECRET_KEY

A read-only key is enough: the agent never writes to your analytics tool. If the tool refuses the key, the audit says so and the dashboard shows Update your Mixpanel key (or PostHog, or Amplitude) until you save a new one on the Setup page.

What happens when the setup pull request is merged?

  1. The first audit starts by itself, on the push that brings the workflow to your default branch. If it didn't, use Run the first audit on the dashboard or the Setup page.
  2. It opens your first plan as a pull request from datalyze/first-plan: the registry written from your code (and your analytics data, if connected), the tracking profile, the baseline and the audit. Its datalyze/tracking passes with "First tracking plan: merging it approves it".
  3. Merge it. That approves the plan. The audit's findings are on the audit screen, where you can pick the ones to fix.
  4. Go live on the Setup page. It adds a repository ruleset that makes datalyze/tracking and datalyze/rules required on your default branch.

Until a repo goes live, the checks run and report on every pull request but don't block a merge. The setup pull request, and any pull request opened before the first plan exists, pass with "No tracking plan yet: the first audit writes it". The audit covers what the first audit writes.

What does the setup pull request add?

FileWhat it is
.github/workflows/datalyze.ymlThe workflow. It runs on pull requests, on @datalyze comments, on each push to your default branch (to rebuild the Sheet) and on runs Datalyze starts (an audit, a build, fixes)
.datalyze/config.yamlWhere the plan is, who may approve, and which analytics tool the audit reads
.datalyze/context.mdWhat your team told the agent about the product. Edit it like any file, or on the Setup page
.datalyze/.gitignoreKeeps the scan's cache out of git
analytics/registry/_conventions.yamlYour naming rules, over the Datalyze defaults
CLAUDE.mdA short block that sends tracking questions to the plugin: "what does this branch need" runs /datalyze:review. It replaces v1's block in AGENTS.md where there is one
.claude/settings.jsonOffers the Datalyze plugin to anyone who opens the repo in Claude Code. Your other settings stay

Nothing in your product code changes. The first audit adds the rest of the registry on its own pull request. Reference lists every file the agent keeps.

Which secrets does Tracking Agent setup save?

SecretWhat it holdsWhen
DATALYZE_KEYYour workspace key, made by setup and saved without being shown. Each install makes its own, and older repos keep theirsAlways
ANTHROPIC_API_KEYYour Anthropic API keyUnless the key is in an organisation secret you named
MIXPANEL_SA_USERNAME, MIXPANEL_SA_SECRETA Mixpanel service account (read-only)With Mixpanel
POSTHOG_API_KEYA PostHog personal API key with read accessWith PostHog
AMPLITUDE_API_KEY, AMPLITUDE_SECRET_KEYThe Amplitude project's keysWith Amplitude
DATALYZE_SHEET_URL, DATALYZE_SHEET_TOKENYour Sheet's Apps Script receiver and its tokenOnly with the Apps Script route for the Sheet

Each value is encrypted in your browser with the repo's public key before it's sent to GitHub through Datalyze's server, so only GitHub can open it. GitHub explains how secrets in GitHub Actions are stored and who can read them.

Who can approve tracking changes?

By default, anyone with write access to the repo. To narrow it, list GitHub logins or @org/team under reviewers in .datalyze/config.yaml. The list is read from the pull request's base branch, so a pull request can't change who approves it, and it applies to both ways of approving: Build it in Datalyze and an @datalyze apply comment.

yaml
# .datalyze/config.yaml
registry: analytics/registry
reviewers:
  - priya-dev
  - "@acme/analytics"

Why does setup install the Datalyze GitHub App?

Commits pushed with the default Actions token don't start other workflows, so your CI would never run on the agent's commits; GitHub documents this under automatic token authentication. Commits pushed with a GitHub App's token do start them, so the agent commits as the Datalyze GitHub App.

You install the App only on the repos you choose. Its private key stays on Datalyze's server, which gives each run a token for its own repo, valid for an hour. While your subscription is active, the token can push the agent's commits and post its checks; without one, it can only post the two checks as passing, so a live repo keeps merging. The ruleset accepts the two checks only from the Datalyze App, so no other token can turn them green.

How do I move a v1 repo to v2?

On the Setup page, a repo that still runs v1 shows Update the workflow. It opens one pull request, "Move the Datalyze Tracking Agent to v2", that changes the workflow, adds the CLAUDE.md block and offers the plugin in .claude/settings.json. Your .datalyze/config.yaml, .datalyze/context.md and tracking plan stay as they are.

v2 runs on Claude only, so the repo needs an ANTHROPIC_API_KEY secret first; save one under Anthropic API key on the Setup page. Until the move merges, a v1 run passes with "not checked". A _map/ folder v1 wrote in the registry is ignored by v2 and can be deleted.

Questions about setting up the Tracking Agent

Does Tracking Agent setup change our product code?

No. The setup pull request adds a workflow, the .datalyze folder, the naming conventions, a CLAUDE.md block and the plugin setting. Tracking calls change later, in pull requests a person approved, written the way your code already writes them.

Who needs admin access during setup?

Only the person running setup: installing the Datalyze GitHub App, saving repo secrets and going live all need admin rights. Reviewers need only their normal access to the repo.

We already have a tracking plan in a spreadsheet. Can setup import it?

No. The first audit writes the registry from the tracking calls in your code, and from your analytics data if you connect it. Copy anything worth keeping from the old spreadsheet into the registry on the first plan's pull request, or later.

Can Datalyze build our tracking before we add the check?

Yes. If the tracking itself needs designing or rebuilding, our analytics setup service does that work, and the Tracking Agent keeps it correct afterwards.