Control group: what it is, why it matters, and how it works in any experiment

What is a control group?

A control group is the group in an experiment that doesn’t receive any new treatment. It stays exactly as-is. The whole point is to give you something stable to compare your changes against.

Without that baseline, you can’t tell whether your change caused the result or something else did.

Think of it like a cooking experiment. You make two batches of cookies. One batch uses the original recipe (that’s your control group). The other swaps butter for coconut oil.

If the coconut oil batch tastes better, you know why. Without the original recipe as your baseline, you’re just guessing.

In split testing, the control group is your current page. Version A. The one that’s already live. You’re comparing it against a new version (Version B) to see which performs better.

You’ll also hear people call it the “comparison group” or “baseline group.” Same idea, different label.

How control groups work in experiments

Here’s the basic flow, whether you’re running a clinical trial or testing a new headline on your website:

1. Start with your population. That’s everyone in the experiment. Website visitors, email subscribers, patients in a study.
2. Split them randomly into two groups. Random is the key word. No cherry-picking. No putting your best customers in one group.
3. Leave one group alone. That’s your control group. They see your current page, get no email, receive the placebo pill.
4. Change one thing for the other group. That’s your treatment group (sometimes called the experimental group or the variant). They see the new headline, get the promotional email, take the new medication.
5. Measure the same thing for both groups. Conversion rate, recovery time, purchase rate, whatever matters.
6. Compare. If the treatment group does better than the control group, your change probably worked.

The random split is what makes this trustworthy. Without it, you’re just comparing apples to oranges. (Need help structuring this? Grab an A/B testing template and fill in the blanks.)

Both groups also need to run at the same time. Comparing this month’s results to last month’s isn’t a real control. Too many things change between months (seasons, holidays, a competitor launching a sale).

That’s why designing a proper marketing experiment always includes a simultaneous control.

A/B testing tools like Kirro handle the random split automatically. You pick what to change, the tool divides your visitors 50/50 and tracks the results. No spreadsheet required.

Our take: The hardest part of control groups isn’t the concept. It’s the patience. People want to skip ahead and ship the change. But the whole point is waiting for the data, and the data needs a baseline to mean anything.

Control groups in science vs. marketing

The concept of a control group has been around for almost 300 years. In 1747, a ship’s doctor named James Lind tested six different scurvy treatments on 12 sailors aboard the HMS Salisbury.

He gave each pair a different remedy (citrus fruit, cider, vinegar, seawater, and two others), kept their conditions identical, and compared what happened. The citrus pair recovered in six days. Everyone else stayed sick.

That’s a control group in action. And the logic hasn’t changed since.

In 1935, statistician Ronald Fisher formalized the idea. He proved that random assignment eliminates bias and makes results trustworthy. His work at an agricultural research station (testing crop yields) became the foundation of every clinical trial and A/B test run today.

Here’s how the same principle plays out in two different worlds:

	Science	Marketing
What’s tested	New drug, therapy, medical device	New headline, page layout, email
Control receives	Placebo pill or standard treatment	Current page or no campaign
Sample size	Hundreds to thousands	Thousands to millions
Duration	Months to years	Days to weeks
Ethics review	Required (you can’t give a placebo when a proven treatment exists)	None required
Cost per test	High (staff, regulatory, time)	Near zero
What’s at stake	Human health	Revenue

The FDA requires clinical trials to include a control group. Their official guidance (ICH E10) defines five types of controls and when each is appropriate.

Marketing has no such rules. Nobody will stop you from shipping a redesign without testing it. But that doesn’t mean you should skip the control.

At Booking.com, a team of 1,800+ runs over 25,000 A/B tests per year. Ninety percent fail. Meaning 9 out of 10 ideas don’t improve anything.

Without a control group for each test, they’d have shipped thousands of changes that either did nothing or made things worse.

Google runs 10,000+ experiments a year. Microsoft’s Bing team once ran a “low priority” A/B test on ad headline formatting. It led to a 12% revenue increase, translating to over $100 million annually.

The test sat ignored for six months before someone ran it. No one predicted it would be worth nine figures.

Our take: The science-vs-marketing gap is mostly about stakes, not method. A marketer who skips control groups makes the same logical error as a scientist who skips them. They just lose money instead of lives.

Types of control groups

Not all control groups work the same way. The FDA defines five types for clinical trials, and marketing has borrowed (and adapted) most of them.

Type	Science example	Marketing example	When to use it
Placebo control	Patients get a sugar pill	Visitors see the unchanged page	Standard A/B test
Active control	Patients get the current best drug	You test against your best-performing page	When you already know something works
No-treatment control	Patients receive nothing	Customers get no email at all (holdout testing)	Measuring if a campaign works at all
Historical control	Compare to past patient data	Compare this month to last month	Weak. Conditions change. Avoid if possible
Positive control	A known-to-work drug verifies the setup	An A/A test confirms your tool is working	Validating your experiment setup
Waitlist control	Patients wait their turn for treatment	Prospects on a waitlist vs. immediate access	SaaS trials, product launches

The most common mistake? Treating historical controls as just as good as concurrent ones. Comparing your February numbers to March isn’t a real experiment.

A 2024 meta-analysis of 333 clinical trials found that even the type of control you pick changes the results. Waitlist controls, for example, systematically made treatments look more effective than they actually were.

The type of control group you choose matters almost as much as having one at all.

Control group vs. experimental group

This is the most common question in any science class (and honestly, in plenty of marketing meetings too).

	Control group	Experimental group
What it receives	No change. The original.	The new version, treatment, or variable.
Its role	Baseline for comparison	Tests whether the change makes a difference
Also called	Baseline group, comparison group	Treatment group, test group, variant
In A/B testing	Version A (your current page)	Version B (the new version)
How many?	Usually one	Can be multiple (B, C, D…)

You can run tests with multiple experimental groups. That’s what multivariate testing does. You test several changes at once, each against the same control.

The difference between A/B testing and multivariate testing comes down to how many things you’re changing at a time. But you typically only need one control group. It’s the anchor.

Control group examples

Clinical trial

Two hundred patients with the same condition are split randomly. One hundred get the new drug. One hundred get a sugar pill (placebo).

After 12 weeks, doctors compare recovery rates. The sugar pill group is the control. If recovery is 60% in the drug group vs. 30% in the control group, the drug likely works.

Email marketing (holdout)

You have 10,000 subscribers. You send a promotional email to 9,000 of them. The other 1,000 get nothing. That held-back group is your control.

If 4% of the email group buys and only 2% of the holdout group buys, your email drove the difference.

This is how Optimove discovered that a campaign appearing to generate $40,000 in revenue was actually only responsible for $17,100. The other $22,900 would have happened anyway.

Website A/B test

Fifty thousand visitors come to your landing page. Half see the original (control). Half see a version with a new headline (treatment). You track your conversion rate for both.

If the new headline converts at 4.2% vs. the original’s 3.5%, you’ve got a winner.

Tools like Kirro set this up in minutes. You pick the page, make the change in a visual editor, and the tool handles the random split and metrics tracking. No developer needed.

Advertising

Alibaba ran an experiment with 555,800 customers. The treatment group got personalized product recommendations. The control group got generic ones.

Removing personalization caused a 75% drop in click-through rates and an 81% drop in purchases. The control group proved personalization wasn’t just correlated with sales. It caused them.

Why control groups matter

At Microsoft, former technical fellow Ronny Kohavi found that only about one-third of experiments improve their target metric. Another third have no effect. And the last third actually make things worse.

That’s from Trustworthy Online Controlled Experiments, the industry’s reference book on experimentation.

Without a control group, you’d ship all three types equally.

Mastercard’s 2024 State of Business Experimentation report backs this up: 46% of retail ideas don’t break even when properly tested. Almost half. If those ideas had been rolled out without testing, someone would have called them “innovations.”

The control group is what separates “we think this worked” from “we know this worked.”

Google runs 10,000+ experiments per year. Amazon runs 12,000+. Booking.com runs 25,000. The average company? Two to three tests per month.

Companies that test more win more, but only because each test has a control group that catches the losers before they ship.

Even 77% of organizations that run A/B tests on their websites are doing better than most. Only 0.2% of all websites use A/B testing tools at all. The bar is low.

If you want to start testing, it doesn’t have to be complicated. Set up a simple A/B test with your current page as the control, make one change, and see what happens.

Common control group mistakes

Contamination

Your control group accidentally gets exposed to the treatment. Maybe a promotional email gets forwarded to holdout customers. Maybe a feature change leaks through shared infrastructure.

Meta, Microsoft, and Booking.com have reported that unaddressed contamination can reduce measured lift by 30 to 70%. Your test might show a 5% improvement when the real number is 15%, or vice versa.

Too-small control groups

A control group needs enough people to detect real differences. For A/B tests, a 50/50 split gives you the most statistical power (your test’s ability to spot a real difference when one exists).

For marketing holdouts, 5% works for campaigns targeting 10,000+ customers. Smaller campaigns need 10-20%. Running a test with fewer than 1,000 visitors? You’re flipping a coin and calling it data.

Non-random assignment

Putting your best customers in one group and everyone else in another isn’t an experiment. It’s wishful thinking.

Random assignment is what makes the groups comparable. Skip it and your results are meaningless.

Checking results too early

Looking at your test results before you have enough data leads to false positives (Type 1 errors). The numbers look promising on day two, you call a winner, and a week later the effect disappears.

This is called “peeking,” and it’s one of the most common A/B testing mistakes. Wait for your test to reach the sample size set by your minimum detectable effect (the smallest improvement worth detecting).

Comparing “before” to “after” instead of using a simultaneous control

This month’s conversion rate is higher than last month’s. Must be the redesign, right? Not necessarily.

Seasonal traffic, a competitor going offline, a viral social post. Any of these could explain the change. A true control group runs at the same time as your treatment group. That’s the only way to isolate the variable.

Skipping the control entirely

Some marketers skip control groups because they don’t want to “leave money on the table” by not showing the campaign to everyone. But without a control, you never know if the campaign actually worked.

You might be optimizing something that has zero real impact.

The fix is simple. Set up a quick A/B test with a proper control group and let the numbers tell you what’s actually working.

Our take: If you’re not willing to set aside a small group as a control, you’re not really testing. You’re just making changes and hoping.

FAQ

What is a control group in an experiment?

A control group is the group in an experiment that receives no new treatment or change. It stays the same while the experimental group gets the thing you’re testing.

By comparing the two, you can figure out whether your change actually caused the result. In website testing, the control group sees your current page (Version A), and the experimental group sees the new version (Version B).

The null hypothesis (your starting assumption that there’s no difference between groups) is what your test tries to disprove.

What is a good example of a control group?

An email holdout is one of the clearest examples. Say you have 10,000 subscribers and you’re running a promotional campaign. Send the email to 9,000 (your treatment group) and hold back 1,000 (your control group).

If 4% of the email group purchases but only 2% of the control group does, you know the email drove about 200 extra sales. Without the holdout, you’d have assumed all 360 purchases were caused by the campaign.

How do you identify a control group?

The control group is the one that doesn’t receive the new treatment or change. In an A/B test, it’s Version A, your current page. In a clinical trial, it’s the placebo group.

Ask yourself: “Which group stays the same?” That’s your control.

What is a control group in science?

A science control group follows the same rules as any other, just with higher stakes. In a clinical trial, the control group might receive a placebo (sugar pill) while the treatment group gets the actual medication.

The FDA requires clinical trials to include a properly designed control group. In laboratory research, the control keeps all variables constant except the one being tested.

Same principle as marketing: isolate the variable, measure both groups, compare.

Can you run an experiment without a control group?

Technically, yes. Synthetic controls let you build an artificial comparison group from historical data. Quasi-experiments compare similar (but not randomly assigned) groups.

Multi-armed bandit testing shifts traffic toward the winning version dynamically, without a fixed control.

Netflix uses geographic experiments when user-level controls aren’t feasible. But these approaches need more data, more assumptions, and more statistical skill.

For most businesses running website tests or evaluating A/B testing alternatives, a simple control group is still the gold standard. Easiest path to results you can trust.

What is the difference between a control group and an experimental group?

The control group receives no change. The experimental group receives the new treatment.

In an A/B test, the control is your current page and the experimental group sees the variation. You can have multiple experimental groups (testing different headlines, images, or layouts), but you typically only need one control. It’s your anchor. Everything else gets measured against it.