How to Use Pairwise Testing for Cross-Browser Coverage

Pairwise testing can reduce a large browser and device matrix while guaranteeing that every allowed pair of values across every pair of modeled factors appears in at least one test. Define the factors and valid values, encode impossible combinations as constraints, generate a pairwise covering set, and run each row through your browser test runner. Keep targeted tests for critical journeys and known browser-specific risks: pairwise coverage does not test every full configuration or guarantee detection of failures that require three or more interacting conditions.

1. Define the supported environments

Start with the environments your product intends to support. Use audience or support data if available; there is no universally correct browser matrix. Decide whether your goal is coverage across browser engines, branded browsers, operating systems, device classes, or all of these.

For example, a team might model these finite factors:

• Browser engine: Chromium, Firefox, WebKit
• Form factor: desktop, mobile
• Viewport class: narrow, wide
• Locale: primary, secondary
• Authentication: signed out, signed in

This is an example shape, not a recommended universal matrix. Include a factor only when it can affect the feature under test. If behavior depends on a branded browser, enterprise policy, extension, codec, or platform capability, model and run that environment explicitly; an engine label alone may not represent it.

2. Make the model valid

Each factor needs a finite set of values. Add constraints for combinations that cannot occur or are unsupported, such as pairing a mobile-only device profile with a desktop-only operating system. Constraints prevent the generator from spending rows on impossible states. Keep constraints understandable and review them alongside the model.

Pairwise means covering each allowed value pair between each pair of distinct factors at least once. It does not mean that every two values within the same factor are paired, and it does not guarantee coverage for interactions among three or more factors. The [ISTQB syllabus](https://www.istqb.in/pdf/Advanced%20Level%20Syllabus%202019-1%20Test%20Analyst.pdf) describes pairwise testing as testing all pairs of parameter values while avoiding all combinations.

3. Generate pairwise rows with PICT

[PICT](https://github.com/microsoft/pict/blob/main/doc/pict.md) is a command-line combinatorial test generator. Create a model file, for example browser.pict:

Browser: Chromium, Firefox, WebKit
FormFactor: Desktop, Mobile
Viewport: Narrow, Wide
Locale: Primary, Secondary
Auth: SignedOut, SignedIn

Run PICT with the model file as input:

pict browser.pict

PICT’s default generation order is two, so this produces pairwise combinations. To request three-way coverage, use /o:3:

pict browser.pict /o:3

Inspect the generated rows before execution. Confirm that each row is valid, that the labels map to real environments, and that the suite covers the pairs you intended. When the model has constraints or needs coverage above pairs, review PICT’s [model documentation](https://github.com/microsoft/pict/blob/main/doc/pict.md) and consider [NIST ACTS](https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software), which supports constraints and variable-strength coverage.

4. Map rows to browser runs

Generation does not execute tests. Connect each generated row to an automation environment. [Playwright projects](https://playwright.dev/docs/browsers) let a suite run against configured browser builds and projects. Its [emulation options](https://playwright.dev/docs/emulation) include viewport, user agent, touch, locale, timezone, geolocation, permissions, and color scheme.

A small project configuration can make the supported browser choices explicit. Save as playwright.config.js:

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  projects: [
    { name: 'chromium-desktop', use: { ...devices['Desktop Chrome'] } },
    { name: 'firefox-desktop', use: { ...devices['Desktop Firefox'] } },
    { name: 'webkit-desktop', use: { ...devices['Desktop Safari'] } },
    {
      name: 'webkit-mobile',
      use: {
        ...devices['iPhone 13'],
        browserName: 'webkit',
        locale: 'en-US',
      },
    },
  ],
});

Install the browser builds compatible with the Playwright version in your project, then run the suite:

npx playwright install
npx playwright test

This configuration illustrates projects, not a complete pairwise runner: it does not consume the PICT output or encode every example factor. For a production matrix, keep the model as the source of truth and map each generated row to a project and context configuration. If many rows share a browser project, pass row-specific settings such as locale, viewport, or authentication state as test configuration. Preserve the generated row with test results so failures can be reproduced.

Playwright’s browser builds track Playwright releases; keep the package and installed browser versions aligned in CI. Playwright WebKit is not a branded Safari binary, and media codec availability can vary by operating system. Emulation tests configured properties, but does not prove that every physical device behaves identically. Use a real target platform where the behavior depends on that platform.

5. Add coverage for risk beyond pairs

Pairwise suites can miss failures that require three or more simultaneous conditions. Add targeted tests for critical journeys, security-sensitive states, regressions, and known browser differences. Increase interaction strength for high-risk factor groups, or use exhaustive combinations when the relevant factor space is small enough to run and maintain.

NIST’s [combinatorial testing project](https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software) summarizes studies reporting fault detection comparable to exhaustive testing with test sets 20 to 700 times smaller. That is a broad summary across combinatorial-testing studies, not a browser-specific result or a promise for any particular model. NIST’s [SP 800-142](https://csrc.nist.gov/pubs/sp/800/142/final) explains the methods and their limits.

6. Keep the suite useful in CI

1. Version-control the factor model, constraints, generator version, and browser configuration.
2. Regenerate rows when supported environments or relevant settings change.
3. Run the same generated suite in CI and locally, and record the row with each result.
4. Separate broad pairwise coverage from fast smoke tests and risk-focused tests so teams can choose the right run for each change.
5. Review failures in the exact browser build and platform where they occurred before treating an emulated configuration as equivalent.

More factors and values can increase the row count, but there is no fixed number of cases or universal reduction ratio. The practical cost includes both test execution and the time needed to maintain meaningful, reproducible environments. Raise coverage strength selectively where the consequence of a missed interaction is high.

Or skip the browser setup

For screenshots of test states, [ScreenshotNeo](https://screenshotneo.com) offers a website screenshot API and MCP server. Its [API documentation](https://screenshotneo.com/docs/) describes the available options.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`Screenshot request failed: ${res.status}`);
await Bun.write('shot.webp', new Uint8Array(await res.arrayBuffer()));

Cookie banners, popups, and chat widgets are removed before capture. Bot checks, blank pages, and failed loads are never billed. An MCP server lets AI agents take screenshots. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000. Sign up for 1,000 free screenshots a month, with no card.

Troubleshooting

Problem	Likely cause	Fix
Generated rows contain unsupported combinations	The model omits a constraint or uses ambiguous values.	Add explicit constraints and regenerate; review the full row before execution.
A browser project fails to launch in CI	The browser binary is missing or does not match the Playwright package version.	Install the compatible browser builds in the CI image and pin the package version.
A test passes in emulation but fails on a phone	The issue depends on a platform behavior not represented by emulated settings.	Reproduce on the target platform; add a focused real-device or platform test where needed.
A pairwise suite misses a combination bug	The fault requires a three-way or higher interaction, or a factor was omitted.	Expand the model, raise strength for the risky factors, and add a regression test.
WebKit behavior differs from Safari	Playwright’s WebKit build is not branded Safari, and platform behavior can vary.	Validate in branded Safari on the target platform when Safari-specific behavior matters.
The suite is still too slow	Rows run serially, setup is repeated, or the model includes irrelevant factors.	Remove irrelevant factors, reuse setup safely, and allocate CI capacity according to run-time needs; do not remove risk-critical coverage solely to shrink the matrix.

FAQ

How many browser combinations should I test?

There is no universal count. It depends on factor values, constraints, interaction strength, and the environments your product supports. Generate the suite from the reviewed model and measure its run cost.

Does pairwise testing cover Chrome, Firefox, and Safari?

It can cover modeled browser values in pairs with other factors. Choose actual browser channels and platforms based on the behavior you need to validate; Chromium, branded Chrome, WebKit, and Safari should not be treated as interchangeable in every case.

Is pairwise testing enough for release sign-off?

It is one coverage layer. Pair it with critical journey tests, regression tests, and platform-specific checks that reflect your product’s risk.