-
Randy Wattilete - 14 Mar, 2026
- Testing Methodology
The null hypothesis in A/B testing: what it means and why most tests prove it right
The null hypothesis says your change made no difference. That's it. When you run an A/B test, the null hypothesis is the default assumption that Version A and Version B perform the same. Your test's job is to prove this assumption wrong. Think of it like a courtroom. The null hypothesis is "innocent until proven guilt ...
- Randy Wattilete
- 14 Mar, 2026
- Testing Methodology
Sequential testing: when to stop your A/B test early
Sequential testing lets you check A/B testing results as data comes in. Without getting tricked by random noise. Instead of waiting for a pre-set ...
- Randy Wattilete
- 13 Mar, 2026
- Testing Methodology
How to design a marketing experiment (even if most of them fail)
To design a marketing experiment: start with a business question, write a hypothesis you can be wrong about, pick one metric, figure out how many visitors you n ...
- Randy Wattilete
- 13 Mar, 2026
- Testing Methodology
Minimum detectable effect in A/B testing: how to pick the right one for your business
Minimum detectable effect (MDE) is the smallest real improvement your A/B test can reliably catch. If your page actually converts 5% better but your test's ...
- Randy Wattilete
- 13 Mar, 2026
- Testing Methodology
Statistical power and power analysis for A/B tests: the planning step most teams skip
Power analysis answers a simple question: do I have enough visitors to trust my A/B test results? It tells you how many visitors each version needs and how ...
- Randy Wattilete
- 13 Mar, 2026
- Testing Methodology
Type 1 vs type 2 errors in A/B testing: what they are and why they cost you money
A type 1 error means your A/B test says "we have a winner" when there's no real difference. A type 2 error means your test says "nothing happened" when there ac ...
- Randy Wattilete
- 12 Mar, 2026
- Testing Methodology
A/B test sample size formula: how to calculate it (with worked examples)
The A/B test sample size formula is: n = (Zα/2 + Zβ)² × [p₁(1−p₁) + p₂(1−p₂)] / (p₁−p₂)² If that looks like gibberish, you're not al ...
- Randy Wattilete
- 12 Mar, 2026
- Testing Methodology
A/B testing conversion rate: how to measure, track, and actually improve it
Your A/B testing conversion rate is the percentage of visitors who do what you want (buy, sign up, click) in each version of your test. Compare the rates betwee ...
- Randy Wattilete
- 12 Mar, 2026
- Testing Methodology
Bayesian A/B testing: what it is, when it helps, and when it's overkill
Bayesian A/B testing measures which version of a page works better using a probability that updates as data comes in. Instead of waiting for some magic number o ...
- Randy Wattilete
- 12 Mar, 2026
- Testing Methodology
Landing page split testing: the full playbook from page selection to results
Landing page split testing means showing two versions of a page to different visitors, then keeping the version that converts better. Half your traffic sees the ...