How to run experiments at different stages?
Most companies collect insights and never test them. You might not have the traffic for a textbook A/B test, and that is fine. Run a smaller version instead, such as a before-and-after comparison or a group of users held back, and keep running it.
Why don't companies run experiments on their insights?
Most of the companies I work with are genuinely interested in their data. Very few of them are fast or hands-on with experimentation.
So they end up sitting on a pile of insights they never act on. Everyone is busy launching the next feature and fixing the last one. The experiments never get set up.
I am not going to cover how to run one here. No maths on how many users you need, no working out how sure you can be of a result, none of the technical detail.
The point is simpler than that. If you have a data person producing insights and you are not spending time to experiment around them, you are wasting the insight. You never find out how your users actually behave. You never find out what works and what does not.
What if you do not have the traffic for an A/B test?
Skip the textbook test and run a smaller version. You may be thinking you do not have the traffic for this. Most founders who say that to me are not wrong.
The textbook version is an A/B testSplitting users between two versions and comparing the result. The textbook version needs more traffic than most startups have, which is an argument for running a smaller test rather than for running none.Glossary. You split your users at random into two groups. One group sees the change, the other does not, and you compare the two. If a proper A/B test would take three months to give you a result, do not run one.
| If | Then |
|---|---|
| A proper A/B test would give you a result in time | Run it: split users at random into two groups and compare them |
| A proper A/B test would take three months | Ship to everyone and compare the one number it should move, before and after |
| You can keep a slice of users on the old version | Hold them back and watch what they do differently |
| You want to know why people dropped off | Ask ten of them |
How do you run an experiment without an A/B test?
Run the smaller version instead:
- Before and after: Ship the change to everyone. Pick the one number it should move, then compare the weeks before and after.
- Hold some users back: Keep a slice of users on the old version while everyone else gets the change. Watch what they do differently. PostHog's holdouts documentation describes the same setup, a percentage of users excluded from the change, and uses it to see the long-term effect of a change over a few months.
- Just ask: Ask ten people who dropped off why they dropped off.
None of that is as clean as an A/B test, but all of it beats shipping on instinct and never looking again.

Is experimenting worth it when most experiments fail?
Yes, even though it is hard. Run 10 experiments and 9 of them will fail.
But those 9 failures are 9 things you now know do not work in your product. Do that for long enough and you understand your product well enough to know what to do next.

Common questions
Can you run experiments without much traffic?
Yes, just not textbook A/B tests. Run smaller, simpler versions and keep running them. The alternative most teams choose is not testing at all, which teaches you nothing.
Why don't companies act on their insights?
Everyone is busy launching the next feature and fixing the last one, so experiments never get set up. The insights pile up and nobody ever finds out whether they were right.
What is a holdout group in experimentation?
A slice of users you keep on the old version while everyone else gets the change. You watch what they do differently, which tells you what the change did.
How many experiments fail?
Run 10 experiments and 9 of them will fail. Each failure is one more thing you know does not work in your product, and enough of them tell you what to do next.