--- title: "Test SEO changes on matched page groups so you roll out only what moves organic traffic" description: "Split pages into matched control and variant groups, size the test for statistical power, isolate outside noise, and scale SEO experiments safely." canonical: "https://scalewithsearch.com/articles/seo-ab-testing-methods" date: "2026-03-20" modified: "2026-09-25" --- ## Site navigation - [Scale With Search](https://scalewithsearch.com/) - Real estate - Real estate - [Real estate](https://scalewithsearch.com/for/real-estate) - Work - Start here - [Send your brief](https://scalewithsearch.com/work#send-your-brief) - [Prepare your six-question brief](https://scalewithsearch.com/work#prepare-your-six-question-brief) - Build - [Site build, content library with SEO, signal desk](https://scalewithsearch.com/work) - For your business - Trades and home services - [Auto body and collision shops](https://scalewithsearch.com/for/auto-body-and-collision-shops) - [Foundation and home repair contractors](https://scalewithsearch.com/for/foundation-and-home-repair) - [Garage door and fencing contractors](https://scalewithsearch.com/for/garage-door-and-fencing-contractors) - [HVAC contractors](https://scalewithsearch.com/for/hvac-contractors) - [Janitorial and commercial cleaning companies](https://scalewithsearch.com/for/janitorial-and-commercial-cleaning) - [Locksmiths](https://scalewithsearch.com/for/locksmiths) - [Moving companies](https://scalewithsearch.com/for/moving-companies) - [Pest control companies](https://scalewithsearch.com/for/pest-control-companies) - [Plumbing and electrical contractors](https://scalewithsearch.com/for/plumbing-and-electrical-contractors) - [Restoration and water or fire damage companies](https://scalewithsearch.com/for/restoration-and-water-fire-damage) - [Roofing companies](https://scalewithsearch.com/for/roofing-companies) - [Towing companies](https://scalewithsearch.com/for/towing-companies) - [Tree services and landscaping companies](https://scalewithsearch.com/for/tree-services-and-landscaping) - [Solar installers](https://scalewithsearch.com/for/solar-installation) - [General contractors](https://scalewithsearch.com/for/general-contractors-and-construction) - [Paving, concrete, and flooring contractors](https://scalewithsearch.com/for/paving) - Practices and professional services - [Bookkeeping and tax practices](https://scalewithsearch.com/for/bookkeeping-and-tax-practices) - [Dental practices](https://scalewithsearch.com/for/dental-practices) - [Family and criminal defense law firms](https://scalewithsearch.com/for/family-and-criminal-defense-law-firms) - [Med spas and aesthetics practices](https://scalewithsearch.com/for/med-spas-and-aesthetics) - [Personal injury law firms](https://scalewithsearch.com/for/personal-injury-law-firms) - [Veterinary clinics](https://scalewithsearch.com/for/veterinary-clinics) - [Gyms and fitness studios](https://scalewithsearch.com/for/fitness) - [Therapy and outpatient health practices](https://scalewithsearch.com/for/therapy-and-outpatient-health) - [Medical billing companies](https://scalewithsearch.com/for/medical-billing) - [Insurance agencies](https://scalewithsearch.com/for/insurance-agencies) - [Financial advisors](https://scalewithsearch.com/for/financial-advisors) - [Property management companies](https://scalewithsearch.com/for/property-management) - [Recruiting and staffing agencies](https://scalewithsearch.com/for/recruiting-and-staffing) - [Architects and interior designers](https://scalewithsearch.com/for/architects-and-interior-designers) - [Logistics and supply chain companies](https://scalewithsearch.com/for/logistics-and-supply-chain) - Agencies, MSPs, and manufacturing - [IT and managed service providers](https://scalewithsearch.com/for/it-and-managed-service-providers) - [Machine shops and precision manufacturers](https://scalewithsearch.com/for/machine-shops-and-precision-manufacturing) - [Marketing agencies and freelancers](https://scalewithsearch.com/for/marketing-agencies-and-freelancers) - [SEO agencies and consultants](https://scalewithsearch.com/for/seo-agencies-and-consultants) - [Small manufacturers and fabricators](https://scalewithsearch.com/for/small-manufacturers-and-fabricators) - Restaurants, shops, studios, and nonprofits - [Restaurants and hospitality businesses](https://scalewithsearch.com/for/restaurants-and-hospitality) - [Retail stores and ecommerce sellers](https://scalewithsearch.com/for/retail-and-ecommerce) - [Photographers, event planners, and travel agents](https://scalewithsearch.com/for/photographers) - [Churches and nonprofits](https://scalewithsearch.com/for/churches-and-nonprofits) - [All industries](https://scalewithsearch.com/for/) - Learn - For your office - [Office job guides](https://scalewithsearch.com/guides/) - [Browser calculators](https://scalewithsearch.com/tools/) - Start here - [How it works](https://scalewithsearch.com/how-it-works) - [Free Starter Kit](https://scalewithsearch.com/kit/business-memory-starter-kit.zip) - [Synthetic specimen](https://scalewithsearch.com/specimen/working-session-specimen.zip) - Guides - [The Complete Guide to Business Memory for AI Agents](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [The Complete Small-Business Guide to AI Agent Governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [The Complete Guide to Leaving Vendor AI Memory](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - Articles by cluster - [Business memory](https://scalewithsearch.com/articles/business-memory-for-ai-agents-guide) - [Agent governance](https://scalewithsearch.com/articles/ai-agent-governance-guide-small-business) - [Migration and ownership](https://scalewithsearch.com/articles/leaving-vendor-ai-memory-guide) - For machines - [llms.txt](https://scalewithsearch.com/llms.txt) - [llms-full.txt](https://scalewithsearch.com/llms-full.txt) - [Machine view](https://scalewithsearch.com/?view=machine) - Company - Evidence - [Proof](https://scalewithsearch.com/proof) - Company - [About](https://scalewithsearch.com/about) # Test SEO changes on matched page groups so you roll out only what moves organic traffic. A team rewrites the title tags on every product page. Organic traffic rises 8% the next month. Nobody can say whether the titles caused it, because a core update, a seasonal lift, and a competitor's outage happened in the same weeks. An SEO split test answers that question. It compares treated pages against a control group, so it shows causation instead of correlation. A conversion A/B test splits users between page versions. An SEO test splits pages into a control group and a variant group. It changes the variants only and measures the organic difference over several weeks. ## Know why a conversion A/B test does not work for SEO A conversion test randomizes each visitor to a page version. SEO cannot do that. Googlebot and Bingbot crawl URLs, not user sessions, and each URL has one indexed version. Different content for bots and users is cloaking, which search engines penalize. A JavaScript content swap after page load adds indexing delays and measurement noise. That constraint forces a different design. Divide URLs into groups that match on historical traffic, and change one group. Wait for search engines to recrawl and re-rank, then compare organic sessions between the groups. The method trades speed for validity. A test runs 2 to 8 weeks, depending on crawl frequency and traffic volume. As of September 2026, SearchPilot and Semrush's SplitSignal offer this method as platforms. They automate URL grouping and the significance calculation. The main risk is not a false positive. It is opportunity cost. A 6-week title tag test delays other experiments. Poor grouping adds bias that inflates or hides the real effect. An underpowered test ends without an answer. The [SEO testing in CI/CD guide](/articles/seo-testing-ci-cd-pipeline) covers the related job of catching regressions before release, which is a different kind of test. ## Build matched control and variant groups A valid split needs control and variant pages that behaved the same way before the change. Template type gives a coarse match: product pages with product pages, blog posts with blog posts, category pages with category pages. Historical organic sessions, conversion rate, and a link-strength metric such as Ahrefs URL Rating give a tighter one. Follow these steps to prepare the page pool: 1. Export organic landing page data from Google Analytics 4 for the last 90 days. 2. Keep pages with 100 or more monthly organic sessions, so the test has enough data. 3. Calculate each page's coefficient of variation: the standard deviation of its sessions divided by its mean. 4. Exclude pages with a coefficient above 0.8, because high volatility hides real changes. Then assign pages to groups. Two methods work. **Propensity score matching.** Train a logistic regression model on page features. Use sessions, bounce rate, page depth, and an authority metric from Moz Pro or Ahrefs. Pages with similar scores behave similarly under normal conditions. Pair them, and put one of each pair in each group. The treatment effect then stands out when you apply the change. **Stratified random sampling.** On a site with thousands of pages, perfect matching may not be practical. Divide the pages into ten bands (deciles) by organic traffic. Assign half of each band at random to control and half to variant. This keeps the traffic distribution even, and the randomization protects against variables you did not measure. Do not split by site section or URL pattern unless the traffic behaves the same. If all `/blog/` URLs are variants and all `/products/` URLs are controls, the section and the change are mixed together. You cannot tell whether a result came from your edit or from the difference between sections. The number of pages you need depends on traffic. For template-level tests, aim for 50 or more pages per group. Fewer pages can work when each has high traffic. Ten pages with 5,000 monthly sessions each give more power than 100 pages with 100 sessions each. Calculate the minimum from the effect size to detect, the baseline traffic, and the historical variance. ## Match the test design to the change ### Title tags A title test measures change in click-through rate from the results page. Group pages by their current position, because click-through curves differ steeply by position. Click-through rate falls steeply from position 1 to position 10. Published curves shift as search features change, so check a current source, such as Advanced Web Ranking's CTR data. Tests on position-1 pages need larger samples, because click-through variance is higher at the top. Apply the new titles to the variant pages. Wait 2 to 3 weeks for Google to recrawl and update the results. Then compare click-through rate between groups in Google Search Console. To control for query changes, check branded queries separately. Or use impression-weighted click-through rate to absorb ranking movement. Google truncates titles by pixel width, and the visible length differs between mobile and desktop results. Segment the analysis by device. ### Structured data A markup test measures rich result appearances and the clicks they bring. Choose markup the page type is eligible for, such as Product or Review snippets. Do not build a test on FAQ or HowTo markup. Since August 2023, Google shows FAQ rich results only for well-known government and health sites. Google removed HowTo rich results in September 2023. Add the markup to the variant pages and validate it with the Rich Results Test and the Schema Markup Validator. Track rich result appearances with Semrush Position Tracking or the Search Console search appearance filter. Measure incremental clicks by comparing clicks on results with the feature against clicks on plain results. ### Internal links An internal link test moves link equity to target pages. Add contextual links from strong pages to the variant URLs. Track Ahrefs URL Rating weekly, and measure the organic session lift after 4 to 6 weeks. Check that control and variant pages gained similar external backlinks during the test. The internal links must be the only difference. ### Content expansion A content test adds depth to thin pages. The variant pages get 800 to 1,200 more words on secondary keywords from Ahrefs Keywords Explorer. Track rankings for both primary and secondary terms. Estimate incremental sessions as the ranking gain multiplied by search volume and the expected click-through rate for the new position. ## Calculate significance and size the test SEO tests run in noisy conditions. A winner declared too early ships a change that does nothing; a test run too long delays a real improvement. ### Use a Bayesian model A Bayesian framework updates the probability that the variant beats the control as data arrives. When that probability passes 95%, you have enough evidence to roll out. Google's CausalImpact package for R implements Bayesian structural time-series models that account for the trend before the change. Feed it the control group's sessions as a predictor for the variant group's sessions. Fit the model on the pre-change period. Then compare the actual post-change sessions with the model's prediction of what would have happened without the change. The difference is the causal effect, with a credible interval. ### Use a simpler frequentist check For a simpler analysis, calculate the percent difference between groups: ``` lift % = (variant sessions - control sessions) / control sessions * 100 ``` Then run a two-sample t-test on daily session counts for the two groups after the change. A p-value below 0.05 means the difference is unlikely to be random. ### Size the test before you start The minimum detectable effect (MDE) sets the test length. For two groups, the number of observations per group is approximately: ``` n per group = 2 × (z_alpha/2 + z_beta)² × sigma² / delta² ``` With a 5% significance level and 80% power, `(1.96 + 0.84)²` is about 7.84. A worked example: control pages average 1,000 weekly sessions each, with a standard deviation of 200 (20%). You want to detect a 10% lift, so `delta` is 100. Then n = 2 × 7.84 × 40,000 / 10,000, which is about 63 page-weeks per group. With 10 pages per group, that is about 6.3 weeks. With 20 pages per group, it is about 3.2 weeks. This formula treats each page-week as independent. Weeks from the same page are correlated, and pages differ from each other, so real tests need more data than the formula gives. Estimate the standard deviation from your own pre-change data, and add a margin. The test length is also never shorter than the time Google needs to recrawl and re-rank the variants. Aim for 80% power, an 80% chance of detecting a real effect. An underpowered test leads you to reject changes that work. Estimate the sample size first with an online calculator or R's `pwr.t.test()`. As a guide, allow at least 4 to 6 weeks for crawling, indexing, and ranking. Pages with 10,000 or more monthly sessions can detect a 10% lift in 3 to 4 weeks; lower-traffic pages need 6 to 8. A holiday retailer may need 8 to 12 weeks to cover its seasonal cycle. ## Isolate outside variables ### Algorithm updates An algorithm update during the test can corrupt the result. A lift in the variants can come from the update instead of your change. Analyze branded and non-branded impressions separately, because updates rarely move branded search. If non-branded impressions jump 40% across control and variant alike, the algorithm changed; extend the test. When a core update lands mid-test, pause the measurement and wait 2 to 3 weeks for volatility to settle. Then extend the post-change period to collect clean data. Compare the two groups during the update. If both moved equally, the update hit all pages, and the test design still holds. The [algorithm update response guide](/articles/google-algorithm-updates-role-response) covers how to read an update's effect. ### Seasonality Seasonal cycles look like treatment effects. E-commerce sites spike in December. B2B sites dip in summer, when decision-makers take vacation. Remove the trend with week-over-week percentage changes instead of absolute session counts. Or add lagged sessions as covariates in a regression model. ### Competitors Competitor moves, such as link campaigns or content refreshes, hit both groups equally only if the pages compete for similar queries. Analyze low-competition (Ahrefs Keyword Difficulty below 30) and high-competition (above 60) segments separately. ### Crawl lag On large sites, crawl capacity delays the indexing of variant pages. Suppose Googlebot crawls 10,000 pages a day and you change 5,000 variants. Expect 2 to 3 weeks before all changes are indexed. Check indexing with searches such as `site:example.com intitle:"variant title"`, or with the Search Console URL Inspection API. Do not start the measurement until at least 80% of the variants show the new content in search. ### Devices Device segments can hide opposite effects. Markup displays differently across devices. Analyze by device category, and consider separate tests when mobile makes up more than 60% of organic sessions. Google indexes with Googlebot Smartphone, so confirm that each variant renders correctly on mobile. Google retired its Mobile-Friendly Test in December 2023. Use a live test in the URL Inspection tool, or Lighthouse, instead. ## Scale tests with automation Manual test management breaks down beyond 2 or 3 tests at once. Automation pays back when you run 10 or more tests across several templates. A managed platform such as SearchPilot handles grouping, deployment, and measurement. It applies changes at the edge, for example through a CDN worker, so the origin servers do not change. The [edge SEO guide](/articles/edge-seo-cdn-workers) explains how edge deployment works. A custom build can use Google Cloud Functions or AWS Lambda to change HTML at edge locations. Store control and variant assignments in Redis for lookups under 10 ms. Send events to Google Analytics 4 through the Measurement Protocol, with a custom dimension for the test group. A test lifecycle in Python can follow these steps: 1. Query BigQuery for candidate pages above the traffic threshold. 2. Match pages with propensity scores in `scikit-learn`. 3. Store the control and variant assignments in Cloud Firestore. 4. Deploy edge functions that change the meta tags or markup. 5. Wait 21 days for indexing and ranking to settle. 6. Pull Google Analytics 4 data through the Google Analytics Data API. 7. Calculate Bayesian credible intervals with `PyMC`, the library formerly called PyMC3 (renamed with version 4 in 2022). 8. Show the results in Looker Studio dashboards. Keep each experiment's configuration in Git as a YAML file with its pages, changes, start date, and success metrics. Code review catches grouping errors before launch, and the file history records which changes worked on which page types. Connect the results to revenue. Join Google Analytics 4 sessions to transaction data from Shopify or Salesforce. Calculate revenue per session for each group, and project annual revenue from the session lift. Report the result in this form: "Rolling out the title tag change to all product pages projects [amount] in additional annual revenue, with a 95% credible interval of [low] to [high]." The [SEO analytics setup guide](/articles/seo-analytics-setup-guide) covers the tracking this join depends on. ## Avoid five mistakes that produce wrong answers **Peeking.** If you check results daily and stop when the variant leads, you add bias. Significance moves as data accumulates, and a day-5 winner can fall back by day 21. Fix the test length from the power calculation before you start, and check only at set milestones. **Several changes at once.** A combined change of titles, markup, and content shows that the combination works, not which part did it. Test one change at a time, or use a factorial design, which needs about four times the sample. **Transfer across page types.** A title change that lifts product page click-through does not always work on blog posts, because intent differs. A product query such as "buy red shoes" is commercial. A blog query such as "how to clean red shoes" is informational. **Many tests at once without correction.** Twenty tests at a 0.05 threshold produce about one false positive by chance. Apply a Bonferroni correction: divide alpha by the number of tests (0.05 / 20 = 0.0025). Or use a false discovery rate method. **Homepage tests without care.** Branded search dominates homepage traffic, so analyze branded and non-branded queries separately. A parallel homepage on a staging domain risks duplicate content and mixes in the difference between domains. Treat its results as weak evidence. ---- ```text .|########||. .|########||. .|########||. |##||. .||##|. |##||. .||##|. |##||. .||##|. |#|. .|#|. |#|. .|#|. |#|. .|#|. |#| |#| |#| |#| |#| |#| .#| |#. .#| |#. .#| |#. .#. .#| .#. .#| .#. .#| |#. .#| |#. .#| |#. .#| |# ...... #| |# ...... #| |# ...... #| .# ||#########| #| .# ||#########| #| .# ||#########| #| .#. |######||######|. .#. .#. |######||######|. .#. .#. |######||######|. .#. #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# #. .##|###|##|#|######| .# || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || || |##|#||||||||||||#||#| || # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. # |#||||||||||||||||||||#| #. || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # # .#||||||||||||||||||||||||#| # || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || || ....|||#||||||##|#|||#|#||##||||....|. || #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# #. . ....|# ....#||| ||| .#||| ....#. .# # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. # . ||||#| .#####|| . .#| .#||| ||||# #. .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || .| ||||| #. |#|||||. || #. |#||. .||||| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || || |.... #. ....|#. |. ...|. ....|| || #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# #. .||||||##||||||##||####|||||||#||||||#| .# # .||####|###########################|. # # .||####|###########################|. # # .||####|###########################|. # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#||||||.#.|| # |. ..# #| #|||||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||||| .. |# ## |#| ...#|#|||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#|||#|# .# .#| #| ##|.#..#|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # .#||||||############|######|||||#| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # # |...||#||||||#||||#|#||||||#|||||#||| |.| # #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# #. .. ||||# .|||##|. |#| .|| || .|||#. #| # .# || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || | ...|| ...##| | #| .|. |. ##### .. .| || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || || ||||| || ||||#| . |. |. |#. ||||| |#| || || .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. .# |.....#|....|#.||||.|||##.|#|....||.#||.#. #. #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# #.|||||||#############################| ||| .# || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || || ##||#||#||||||#||#||#||#||##. || # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # # .#|||||||||#||#|||||||||||#| # || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || || |#||||||||||||||||||||||#| || .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. .# |#||||||||||||||||||||#| #. || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || || |#||||||||||||||||||#| || #. .######|#|#########| .# #. .######|#|#########| .# #. .######|#|#########| .# .#. |#####||||#####| .#. .#. |#####||||#####| .#. .#. |#####||||#####| .#. |# ||########|| #. |# ||########|| #. |# ||########|| #. |# ...... #| |# ...... #| |# ...... #| |#. .#| |#. .#| |#. .#| |#. .#. |#. .#. |#. .#. .#| |#. .#| |#. .#| |#. |#| |#| |#| |#| |#| |#| .|#|. .|#|. .|#|. .|#|. .|#|. .|#|. |##||. .||##| |##||. .||##| |##||. .||##| .||########||. .||########||. .||########||. Scale With Search 2026 [scalewithsearch.com](https://scalewithsearch.com) ```