Landing Page A/B Testing: What to Test and When

A/B testing sounds simple: show version A to half your visitors, version B to the other half, see which converts better. In practice, most landing page tests produce inconclusive results — not because testing doesn't work, but because people test the wrong things, stop tests too early, or run multiple changes at once.

This guide walks through a practical framework: what to test first, how to structure a valid test, and how to avoid the mistakes that waste weeks of traffic on data you can't trust.

What Is A/B Testing a Landing Page?

A/B testing (also called split testing) shows two versions of a landing page to different segments of visitors at the same time. One element is changed between the two versions — everything else stays identical. After enough visitors see each version, you compare conversion rates to determine which performs better.

The goal isn't just to find a winner. It's to understand why one version outperforms the other, so you can apply that learning to future pages.

The Testing Priority Framework

Not everything on a landing page is equally worth testing. The elements with the biggest impact on conversion should be tested first. Testing button colour before you've tested your headline is a common and expensive mistake — the headline drives far more conversion lift than any visual tweak.

Test in this order:

Tier 1 — Biggest impact (test these first)

1. Headline
2. Primary call to action (button copy + placement)
3. Offer or value proposition
4. Hero image or video

Tier 2 — Medium impact

5. Form length and fields

6. Social proof (reviews, trust badges, testimonials)

7. Subheadline and supporting copy

8. Page structure (above-fold layout)

Tier 3 — Marginal impact (test last)

9. Button colour

10. Font size

11. Background colour

12. Icon styles

Always complete a Tier 1 test before moving to Tier 2. A page with a weak headline will underperform regardless of how much you optimise the button colour.

Tier 1: Headline Testing

Your headline is the first thing visitors read and the single largest driver of conversion rate. A/B testing your headline can produce conversion lifts of 20–100% — more than almost any other change.

What to test:

• Pain-point vs benefit: Compare a headline that names the problem ("Stop Losing Money to Google Ads Setup Errors") against one that leads with the outcome ("Launch a Google Ads Campaign That Actually Converts").

• Specific vs broad: "Cut Your Google Ads CPC by 30% in 30 Days" vs "Improve Your Google Ads Performance". Specific claims almost always outperform vague ones.

• Question vs statement: "Are You Making These Google Ads Mistakes?" vs "The 7 Google Ads Mistakes Costing You Money". Questions can work well for cold traffic; statements often work better for warm retargeting audiences.

How to structure the test: Change only the headline. Keep the subheadline, hero image, CTA, and everything below the fold identical between versions. Run the test until you reach statistical significance (at least 100 conversions per variant, minimum 2 weeks of data).

Tier 1: CTA Testing

Your call to action button — its copy, size, placement, and whether it appears above the fold — has a direct impact on how many visitors take action.

Button copy to test:

Generic CTAs consistently underperform specific ones:

Placement testing:

• CTA above the fold vs below the fold
• Single CTA vs repeated CTA (above fold + bottom of page)
• Sticky CTA bar vs inline button

Form vs no form: For some offers, removing the form and replacing it with a single CTA button (that reveals the form on click) reduces friction enough to lift conversions meaningfully.

Tier 1: Offer Testing

Sometimes the conversion problem isn't copy or design — it's the offer itself. Testing different offers is often the highest-impact change you can make.

Examples of offer tests:

• Free trial vs free demo vs free audit
• 14-day trial vs 30-day trial
• "Get the checklist" vs "Get the checklist + video walkthrough"
• Price anchoring (showing a crossed-out higher price) vs no anchor
• Single product vs bundle

Tier 2: Form Length Testing

Every field you add to a form reduces the number of people who complete it. The optimal form length depends on what you're offering and how warm your traffic is.

• Common test: Remove one field at a time and measure conversion rate impact. For most lead generation pages, name + email outperforms name + email + phone + company.

• Lead quality consideration: A shorter form generates more leads but often lower quality. A longer form generates fewer leads but typically better qualified ones. Test both and measure lead quality downstream — not just form completion rate.

Tier 2: Social Proof Testing

The type, placement, and format of social proof affects how credible and trustworthy your page feels to a first-time visitor.

What to test:

• Testimonials with photos vs text-only testimonials
• Review count ("Rated 4.8/5 by 240 marketers") vs specific testimonials
• Placement: social proof near the CTA vs above the fold
• Logo strip of known clients vs individual testimonials
• Video testimonial vs written review

How to Run a Valid A/B Test

Step 1: Define your hypothesis before you test

A hypothesis forces you to think clearly about why you expect one version to outperform. Format: "Changing [X] to [Y] will increase [metric] because [reason]."

Example: "Changing the headline from 'Google Ads Guide' to 'Stop Wasting Your Google Ads Budget' will increase form submissions because it speaks directly to the visitor's primary pain point."

Step 2: Calculate the sample size you need

Running a test with 200 visitors and declaring a winner is meaningless. Use a sample size calculator (VWO and Optimizely both offer free ones) before launching. For a page converting at 5%, you typically need 800–1,200 visitors per variant to detect a 20% improvement with 95% confidence.

Step 3: Set a minimum test duration

Never run a test for fewer than 2 full weeks — even if you hit your sample size sooner. Day-of-week and time-of-day effects can skew results significantly. A test that ran only on weekdays may not reflect weekend behaviour.

Step 4: Test one variable at a time

This is the most important rule. If you change the headline AND the CTA AND the hero image in the same test, you can't know which change drove the result. One variable per test, always.

Step 5: Use a proper testing tool

Google Optimize shut down in 2023. Current reliable options:

• VWO — most comprehensive, paid
• Optimizely — enterprise-grade
• Convert — mid-market option
• AB Tasty — good for marketers without developer support
• HubSpot A/B testing — built in if you're on a HubSpot paid plan

Avoid testing by manually switching pages — the results won't be valid because you can't guarantee equal traffic split.

Common A/B Testing Mistakes

1. Stopping the test too early. A test showing 60% confidence after 3 days is not a valid result. Wait for 95% confidence or a predetermined end date.
   
   

2. Running too many tests simultaneously. If the same visitors can see both Test A and Test B on different pages, the results contaminate each other. Run one test per page at a time.
   
   

3. Ignoring seasonal effects. A test run entirely during a sale period or a quiet summer week may not reflect normal conversion behaviour.
   
   

4. Testing without enough traffic. If your landing page gets 50 visitors per month, A/B testing is not your priority. Focus on getting more traffic first.
   
   

5. Declaring a winner too quickly. A variant that's 40% better after 50 conversions often regresses to a much smaller lift or no difference at all once you scale.
   

Frequently Asked Questions

How long should an A/B test run? A minimum of 2 weeks regardless of traffic volume, and until you reach 95% statistical confidence with at least 100 conversions per variant. Use a sample size calculator to set expectations before you start.

Can I A/B test a page that gets low traffic? You can, but results take much longer to reach significance. If your page gets fewer than 500 visitors per month, consider multivariate testing tools that use Bayesian statistics (like VWO) — they can produce useful directional results with smaller samples.

Does A/B testing help SEO? Done correctly, no — A/B testing does not negatively impact SEO. Google's guidelines allow A/B testing as long as you're not cloaking (showing different content to Google than to users). Use canonical tags pointing to the original URL on test variants.

What's the difference between A/B testing and multivariate testing? A/B testing compares two complete page versions. Multivariate testing tests multiple variables simultaneously across many page versions. Multivariate requires significantly more traffic to produce valid results but can identify interactions between elements.

Should I test on mobile and desktop separately? Where possible, yes. Mobile and desktop visitors behave differently — a CTA that works well on desktop may be too far down on mobile. Many testing tools allow you to segment results by device.

Related reading: How to Create a Landing Page That Converts | How to Optimize Landing Pages for SEO | Tips for Lead Generation on Landing Pages