What Is a Regression Test in Software?

A regression test checks that software behavior that worked before still works after a change. The change might be new code, a bug fix, a configuration update, a database or data migration, a dependency upgrade, or an environment change. Regression testing looks for unintended side effects in existing functionality.

It is a testing activity applied at different levels, such as unit, API, integration, end-to-end, visual, and performance testing. Regression testing is not a separate level of testing.

Microsoft describes it as testing performed after changes or updates to a solution. The ISTQB definition similarly focuses on testing a previously tested program after modification to find defects in unchanged areas. See Microsoft’s testing strategy guidance and the ISTQB glossary.

1. What regression testing means

Imagine a checkout system that already supports payments, discount codes, shipping calculations, and order confirmation. A developer changes tax calculation. The regression work does not stop at checking the new tax result. It also checks related behavior that could have been affected: payment authorization, discounts, shipping totals, confirmation messages, and saved orders.

The purpose is to detect regressions: a previously correct behavior that became incorrect because of a change elsewhere.

• Change-related: it is triggered by a modification or relevant environment change.
• Evidence-based: tests compare actual behavior with an expected result.
• Scopeable: the suite can cover nearly everything or only the areas justified by risk.
• Repeatable: automated tests are useful when the same checks run after many changes.

2. Regression testing vs. confirmation testing (retesting)

Activity	Question answered	Typical target
Confirmation testing	Did the specific fix resolve the reported defect?	The failed test or reproduction steps for that defect
Regression testing	Did the change cause failures in other previously working behavior?	Unchanged features that could be affected by the change

After a bug fix, do both when appropriate: first confirm the fix, then run regression checks around the modified code and its dependencies. ASTQB’s presentation of ISTQB material describes subsequent regression testing as checking whether fixes cause failures elsewhere.

3. When should regression testing be done?

Run regression tests when a change could alter an existing workflow. Common triggers include:

• new features or significant refactoring;
• bug fixes, especially in shared components;
• configuration, feature-flag, permission, or environment changes;
• database schema, seed-data, or migration changes;
• library, browser, operating-system, or runtime upgrades;
• API contract, integration, or infrastructure changes;
• before releasing to production and after deployment when production-like checks are available.

For a small, isolated change, a focused set may be enough. For a high-risk release, use a broader suite. The decision should reflect business impact, technical coupling, and the cost of a missed failure.

4. How to choose regression scope

Microsoft’s guidance describes broad, impact-based, change-focused, and combined approaches. Each trades coverage against execution and maintenance effort.

Approach	Coverage	Effort	Risk
Broad suite	Most processes and features	Highest runtime and maintenance	Fewer blind spots, but slower feedback
Impact-based	Business-critical workflows first	Moderate	Lower-priority regressions may be missed
Change-focused	Code and integrations touched by the change	Lowest for small changes	Unrelated but coupled areas may fail unnoticed
Combined	Critical flows plus affected and nearby areas	Adjustable	Requires a maintained dependency and risk map

A practical selection sequence is:

1. List the user journeys and business processes where failure would matter most.
2. Map files, services, data, permissions, and integrations changed by the release.
3. Add tests for directly affected behavior and nearby shared components.
4. Include a small set of cross-system smoke checks.
5. Expand the suite when failures reveal an uncovered dependency.

Record why each test is in the regression set. This keeps the suite understandable when ownership or architecture changes.

5. A regression test workflow

1. Baseline: establish expected results from a known-good build or approved specification.
2. Analyze the change: inspect code, configuration, data, and dependency changes.
3. Select scope: choose broad, risk-prioritized, change-focused, or combined coverage.
4. Prepare data: use deterministic accounts, fixtures, dates, permissions, and external-service stubs where possible.
5. Run fast checks: execute unit and API tests first so obvious failures surface quickly.
6. Run workflow checks: exercise integration and end-to-end journeys.
7. Compare results: distinguish a product regression from an environment, data, timing, or test defect.
8. Triage and report: capture the failing step, build, inputs, logs, screenshots, and expected versus actual behavior.
9. Re-run after a fix: perform confirmation testing, then repeat the relevant regression scope.

6. Manual and automated regression tests

Regression testing can be manual or automated. Manual checks are useful for exploratory investigation, unusual visual behavior, and workflows that are changing rapidly. Automation is valuable for stable, repeated, important paths.

What to automate first

• critical login, purchase, payment, and data-saving paths;
• API contracts with deterministic inputs and outputs;
• calculations and validation rules;
• browser checks that run on every pull request;
• visual checks for pages where layout changes are costly.

Minimal Python example with pytest

import requests

BASE_URL = "https://example.test"

def test_checkout_total_still_includes_discount():
    response = requests.post(
        f"{BASE_URL}/api/checkout/quote",
        json={"sku": "book-1", "quantity": 1, "coupon": "SAVE10"},
        timeout=20,
    )
    assert response.status_code == 200
    body = response.json()
    assert body["currency"] == "USD"
    assert body["discount"] > 0
    assert body["total"] < body["subtotal"]

Keep assertions tied to the contract you intend to protect. Avoid asserting incidental fields that change frequently.

Minimal Node.js example

import assert from "node:assert/strict";

const response = await fetch("https://example.test/api/health");
assert.equal(response.status, 200);
const body = await response.json();
assert.equal(body.status, "ok");
console.log("Regression check passed");

Visual regression

A visual regression test captures a stable page or component and compares it with an approved baseline. Control viewport, device scale, fonts, data, animations, time, locale, and network responses to reduce noise. Review intentional design changes and update the baseline only after approval.

7. CI/CD strategy

Run a small, fast regression set on every change. Schedule longer suites for a nightly build, pre-release gate, or other cadence that fits their runtime. Azure testing guidance recommends integrating tests into CI/CD and scheduling full suites that are too slow for every commit.

• Publish machine-readable results and retain logs and artifacts.
• Retry only known transient failures; never hide repeatable product failures with retries.
• Quarantine a flaky test with an owner and removal deadline.
• Track test duration and failure history so the suite remains usable.
• Run critical post-deployment smoke checks against the deployed environment.

8. Visual regression screenshots with ScreenshotNeo

ScreenshotNeo can provide deterministic page images for visual regression pipelines. It is a website screenshot API and MCP server: a GET request returns PNG, JPEG, WebP, or PDF. Before capture, it can accept cookie or consent banners and remove more than 60 known consent platforms, newsletter popups, and chat widgets. Each step can be turned off.

Only clean shots are billed. Bot checks or CAPTCHAs, blank pages, timeouts, failed loads, and cache hits cost nothing, and the response identifies the result with X-Page-Verdict and X-Billed headers. The API supports full-page capture with lazy images loaded, element selectors, dark mode, device presets or custom viewports, retina scale, custom CSS and JavaScript, waits, blocked resources, headers, cookies, user agents, authorization, timezone, geolocation, transparent backgrounds, resizing, caching, signed links, asynchronous jobs, bulk capture, and a usage API.

For AI-driven test workflows, its MCP server exposes take_screenshot, get_page_info, and capture_pdf to Claude, Cursor, and other MCP clients. See the ScreenshotNeo site and API documentation.

9. Troubleshooting regression failures

Symptom	Likely cause	Fix
Only one test fails after a broad change	Real regression, stale fixture, or changed contract	Compare the expected behavior with the approved requirement and inspect the change boundary.
Failure is intermittent	Race condition, shared data, timing, or external dependency	Make data isolated, wait on a meaningful condition, and stub unstable services.
Visual diff shows the whole page changed	Fonts, viewport, animations, timestamps, ads, or consent UI differ	Pin the environment, disable motion, freeze time, and remove variable content before capture.
Tests pass locally but fail in CI	Different browser, locale, timezone, secrets, data, or service versions	Record environment details and reproduce with the CI container or image.
Test times out	Application failure, slow dependency, or an overly short timeout	Inspect server logs and network traces; wait for a specific state instead of adding arbitrary delays.
Screenshot includes a cookie banner or chat widget	The capture environment differs from a real visitor flow	Accept or remove the banner and disable overlays before taking the baseline.

10. Performance, reliability, and cost

• Performance: run unit and API checks before browser tests, parallelize independent tests, and reserve full suites for scheduled runs.
• Reliability: make tests deterministic, isolate data, pin dependencies, and preserve artifacts for failed runs.
• Maintenance: remove duplicate checks, keep selectors and contracts stable, and assign ownership for flaky tests.
• Scope cost: broad suites provide wider coverage but consume more execution and maintenance time; focused suites are cheaper but leave untested areas.
• Screenshot cost: ScreenshotNeo’s free plan includes 1,000 shots per month with no card. Paid plans start at $5 for 3,000 shots; yearly billing gives two months free. Every feature is available on every plan.

11. Or skip the browser setup

For a screenshot-based regression check, call ScreenshotNeo directly:

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

Cookie banners, popups, and chat widgets are removed before the shot. Bot checks, blank pages, and failed loads are never billed. An MCP server lets AI agents take screenshots. You get 1,000 screenshots a month free with no card, and paid plans start at $5 for 3,000. Create a free ScreenshotNeo account.

12. Frequently asked questions

Is regression testing only for automated tests?

No. It can be manual, automated, or a combination. Automation is most useful for stable, repeated, high-value checks.

Is regression testing performed only after a bug fix?

No. Feature work, refactoring, configuration, data, dependency, and environment changes can all trigger it.

Does a full regression suite test every possible behavior?

No suite can prove that every behavior is correct. A broader suite checks more paths but costs more to run and maintain.

What is a good first regression suite?

Start with critical user journeys, tests covering the changed code, and integrations that share data or services with the change. Expand it when risk or failures justify more coverage.

How is a visual regression baseline approved?

Review the difference against the intended design change, confirm functional tests still pass, and update the baseline with a recorded reason and owner.