Acting on what you find

How to run experiments at different stages?

Most companies collect insights and never test them. You might not have the traffic for a textbook A/B test, and that is fine. Run a smaller version instead, such as a before-and-after comparison or a group of users held back, and keep running it.

By Ansh Agrawal3 min readUpdated

Why don't companies run experiments on their insights?

Most of the companies I work with are genuinely interested in their data. Very few of them are fast or hands-on with experimentation.

So they end up sitting on a pile of insights they never act on. Everyone is busy launching the next feature and fixing the last one. The experiments never get set up.

I am not going to cover how to run one here. No maths on how many users you need, no working out how sure you can be of a result, none of the technical detail.

The point is simpler than that. If you have a data person producing insights and you are not spending time to experiment around them, you are wasting the insight. You never find out how your users actually behave. You never find out what works and what does not.

What if you do not have the traffic for an A/B test?

Skip the textbook test and run a smaller version. You may be thinking you do not have the traffic for this. Most founders who say that to me are not wrong.

The textbook version is an A/B testSplitting users between two versions and comparing the result. The textbook version needs more traffic than most startups have, which is an argument for running a smaller test rather than for running none.Glossary. You split your users at random into two groups. One group sees the change, the other does not, and you compare the two. If a proper A/B test would take three months to give you a result, do not run one.

IfThen
A proper A/B test would give you a result in timeRun it: split users at random into two groups and compare them
A proper A/B test would take three monthsShip to everyone and compare the one number it should move, before and after
You can keep a slice of users on the old versionHold them back and watch what they do differently
You want to know why people dropped offAsk ten of them

How do you run an experiment without an A/B test?

Run the smaller version instead:

  • Before and after: Ship the change to everyone. Pick the one number it should move, then compare the weeks before and after.
  • Hold some users back: Keep a slice of users on the old version while everyone else gets the change. Watch what they do differently. PostHog's holdouts documentation describes the same setup, a percentage of users excluded from the change, and uses it to see the long-term effect of a change over a few months.
  • Just ask: Ask ten people who dropped off why they dropped off.

None of that is as clean as an A/B test, but all of it beats shipping on instinct and never looking again.

Four ways to test a change, by how much traffic each needs: an A/B test (random split, compare the two groups) needs lots of traffic, holding some users back on the old version needs more traffic, before and after (ship to everyone, compare the weeks before and after) works on low traffic, and just asking ten people who dropped off why works on any traffic. If a proper A/B test would take three months, run a smaller version instead.

Is experimenting worth it when most experiments fail?

Yes, even though it is hard. Run 10 experiments and 9 of them will fail.

But those 9 failures are 9 things you now know do not work in your product. Do that for long enough and you understand your product well enough to know what to do next.

Ten experiment cards numbered 01 to 10: nine are marked with a cross and Learned what doesn't work, and card 07 is highlighted as the one win. Run 10 experiments and 9 will fail, but nine failures are nine things you now know about your product.

Common questions

Can you run experiments without much traffic?

Yes, just not textbook A/B tests. Run smaller, simpler versions and keep running them. The alternative most teams choose is not testing at all, which teaches you nothing.

Why don't companies act on their insights?

Everyone is busy launching the next feature and fixing the last one, so experiments never get set up. The insights pile up and nobody ever finds out whether they were right.

What is a holdout group in experimentation?

A slice of users you keep on the old version while everyone else gets the change. You watch what they do differently, which tells you what the change did.

How many experiments fail?

Run 10 experiments and 9 of them will fail. Each failure is one more thing you know does not work in your product, and enough of them tell you what to do next.