Testing Methodology

Statistical power and power analysis for A/B tests: the planning step most teams skip

Power analysis answers a simple question: do I have enough visitors to trust my A/B test results? It tells you how many visitors each version needs and how long to run the test. All before you start. Skip it and you'll waste weeks on a test that was never going to give you a clear answer. Most power analysis guide ...

🔢

🎲

Type 1 vs type 2 errors in A/B testing: what they are and why they cost you money

A type 1 error means your A/B test says "we have a winner" when there's no real difference. A type 2 error means your test says "nothing happened" when there ac ...

🔢

🧮

A/B test sample size formula: how to calculate it (with worked examples)

The A/B test sample size formula is: n = (Zα/2 + Zβ)² × [p₁(1−p₁) + p₂(1−p₂)] / (p₁−p₂)² If that looks like gibberish, you're not al ...

📊

📈

A/B testing conversion rate: how to measure, track, and actually improve it

Your A/B testing conversion rate is the percentage of visitors who do what you want (buy, sign up, click) in each version of your test. Compare the rates betwee ...

📐

🎲

Bayesian A/B testing: what it is, when it helps, and when it's overkill

Bayesian A/B testing measures which version of a page works better using a probability that updates as data comes in. Instead of waiting for some magic number o ...

📄

🔀

Landing page split testing: the full playbook from page selection to results

Landing page split testing means showing two versions of a page to different visitors, then keeping the version that converts better. Half your traffic sees the ...

🧪

🔀

Multivariate testing: when it works, when it doesn't, and what to do instead

Multivariate testing lets you test multiple page elements at the same time. Different headlines, images, and buttons, all mixed and matched to find the winning ...

🔀

🧪

Split testing meaning: what it is, the 3 types, and when each one matters

Split testing means showing two (or more) versions of something to different visitors, then keeping the one that gets better results. That's it. You change a he ...

⚠️

🧪

11 A/B testing mistakes that quietly kill your results

You're running A/B tests. That already puts you ahead of most websites. But running tests and running them well are two different things. Bad tests are worse ...