Tracking Agent docs

How the Tracking Agent audit and Fix work

The Tracking Agent audit compares your tracking plan, what your code sends and what your analytics tool received, and explains every problem with evidence. Run it from Datalyze, from GitHub Actions, or in Claude Code. Then tick the findings to fix and press Fix: the agent opens one fix pull request per product area.

By Ansh AgrawalUpdated

What does the Tracking Agent audit compare?

SourceWhere it comes from
The planThe registry on your default branch, analytics/registry/ unless .datalyze/config.yaml says otherwise
The codeEvery track, identify, reset and alias call and every user-property write, through any wrapper, client and server. Claude reads the code around each call; the scan confirms it sees them all
The dataA snapshot of what your analytics tool received in the audit window (the last 30 days), through your read-only key. Every figure carries the query that produced it

Without an analytics key, the audit compares the plan and the code alone, and says so at the top. The audit never changes product code; fixes go through pull requests you choose.

How do I run an audit?

WhereHow
DatalyzeRun an audit on the repo's audit screen. It runs on the repo's GitHub Actions, against your default branch
GitHub ActionsActions, then Datalyze, then Run workflow, with the job audit (a manual `workflow_dispatch` run)
Claude Code/datalyze:audit, or ask: "audit the checkout events", "what changed in tracking since 1 October". See the plugin on a laptop

An audit can cover the whole repo, one product area, a list of events, or what changed since a date. Runs started from Datalyze cover the whole repo.

What does the first audit write?

The first audit runs when the plan has no events yet, and writes the plan before it compares anything:

  • One area file per product area, such as Checkout or Billing, never per folder. Each event says when it fires and when it doesn't, with example values from the code's literals and the data. UNCONFIRMED only where neither has one, and each becomes an open question.
  • The tracking profile, _tracking_profile.yaml: your SDKs, your wrappers, and a real example of each kind of call, so the scan finds every call and later changes copy your style.
  • User properties, super properties and identity, from what the code actually does.
  • The vocabulary, _properties.yaml, with synonyms for properties spelled two ways.
  • The baseline, _baseline.json, which accepts today's problems so later pull requests are judged only on what they add.

It opens all of it as one pull request from datalyze/first-plan. Event names stay exactly what the code sends, since that's what the data holds; a name that breaks your convention is a finding, fixed later without breaking dashboards.

What do the audit's buckets mean?

Every event and user property lands in one bucket. The audit screen groups findings by them.

BucketWhat it means
Arriving with issuesArrives, but with a problem: a property missing or rarely filled, a value outside the plan, the wrong type, double-firing, or the wrong platform
Broken: nothing arrivesIn the plan and the code, and data that should show it shows nothing
Never arrivedIn the plan and the code for over seven days, and never seen in the data
Stopped arrivingArrived before, then stopped, with the last day it was seen
Not arriving yet (under 7 days old)A new call with no data yet. Not counted as broken
Planned, not in the codeIn the plan as planned, with no call in the code
Gone from the codeIn the plan as live, but the call is gone
In the plan since the first audit, gone from the codeAn old problem the baseline accepted
Deprecated, still arrivingDeprecated in the plan, but data still arrives, past its dual-fire date
Deprecated, still in the codeDeprecated in the plan, but the call is still there
Not in the planSent by the code and arriving, but not in the plan
Not in the plan, and nothing arrivesIn the code only, with no data
Arriving from somewhere elseArrives, but no code in this repo sends it: another service, or old app versions
Marked planned, already arrivingStill marked planned in the plan, though it already arrives

A call less than seven days old with no data is "not arriving yet", never broken, and stays out of the health score. The health score is the share of items that are healthy.

How does the verifier check the audit's findings?

Before anything is written, a separate verifier subagent tries to disprove every finding, from the code and data each one cites. What it disproves is dropped, and the report says so; what it can't decide becomes an open question. A finding of "broken" needs data that backs it: a call that looks correctly wired, with no data to judge, isn't called broken.

Where do the audit results go?

Into analytics/.datalyze/audit.json, which the audit screen reads, and audit.md, a readable report with the three things to fix first. Both live on the datalyze/audit branch and are replaced by each audit. The first audit's are also in its plan pull request.

Each finding's title starts with the event or property name, and carries why it happens, the evidence (a file and line with the code quoted, or the query and its result) and the fix. The playbook chapter on data accuracy as a system covers the alerts and checks that catch drift between audits. For an audit done by people, with a plan to fix what it finds, there's our analytics audit service.

How do I fix audit findings?

  1. Merge the first plan's pull request. Until then, the audit screen shows the findings and questions but doesn't let you act on them.
  2. Open the repo's audit screen from the dashboard. Answer the Open questions (the audit's recommendation is picked, and Decide later leaves one for the next audit), tick the findings to fix, pick Look into it for anything under No data yet, and write anything else you want done under Anything else.
  3. Press Save to keep them without changing the repo, or Build it to build them. You need write access to the repo.
  4. The agent builds it all in one run and opens one pull request, on datalyze/fix-audit-<date>. Its description says what each answer, finding and your note changed, and anything held back. Your answers are also recorded in .datalyze/decisions.yaml, so no review, fix or later audit asks them again.
  5. Review and merge the pull request through your team's normal review. A second Fix for the same audit adds to the same pull request.

Fix pull requests aren't reviewed again by the PR check: the build already checked them with the rules and the verifier. datalyze/rules still runs on them.

Questions about the audit

Does the Tracking Agent audit change our code?

No. It writes the audit files, and on the first audit the plan, on their own branches. Code changes only through the fix pull request you start with Fix.

How often should we run an audit?

After a release that touched tracking, or monthly. The PR check catches what each change does; the audit catches what drifted anyway, such as an event that stopped arriving from old app versions.

Can we audit without connecting an analytics tool?

Yes. It compares the plan and the code, and says at the top that there was no data, so the data checks and the health score are skipped.

Why is a new event "not arriving yet" rather than broken?

A call merged less than seven days ago may not have shipped or been used yet. It's checked again by the next audit.