How to Run BackstopJS Visual Tests in GitLab CI
Run BackstopJS visual regression tests in GitLab CI, publish JUnit results, and make failures block the pipeline reliably.
To run BackstopJS visual tests in GitLab CI, install the project’s pinned BackstopJS dependency, make the application reachable from the runner, and run npx backstop test against committed approved references. Enable BackstopJS’s CI report and upload its JUnit XML with artifacts:reports:junit. The test command’s exit code must be non-zero on a visual failure; GitLab’s JUnit report displays results but does not fail the job by itself.
1. Add BackstopJS and its configuration
Install BackstopJS as a project dependency and commit the lockfile so local and CI installs select the same version. The package snapshot for BackstopJS 6.3.25 specifies Node.js 16 or later and npm 8 or later. Check the version in your own lockfile and its package requirements before selecting a CI image.
npm install --save-dev backstopjs
npx backstop initCommit the generated configuration and the approved reference screenshots. A minimal configuration needs at least one viewport and one scenario with a label and URL. For example, adapt the generated backstop.json:
{
"id": "webapp",
"viewports": [
{ "label": "desktop", "width": 1365, "height": 900 }
],
"scenarios": [
{
"label": "Home page",
"url": "http://app-under-test:3000/"
}
],
"paths": {
"bitmaps_reference": "backstop_data/bitmaps_reference",
"bitmaps_test": "backstop_data/bitmaps_test",
"html_report": "backstop_data/html_report",
"ci_report": "backstop_data/ci_report"
},
"report": ["CI"]
}Use the structure produced by backstop init for your installed version if it differs. Set scenario URLs to addresses that resolve from the process doing the browser capture. The example hostname is illustrative: the correct address depends on your GitLab runner, job containers, and app server.
2. Create and review the baseline
BackstopJS’s workflow is init, test, then approve. Create the initial reference screenshots intentionally, inspect them, and commit or otherwise provide the approved references to the CI job. When an intentional design change causes differences, review the new captures before running backstop approve and updating the baseline. Automatically approving every failed run would erase the comparison that detects unexpected changes.
npx backstop test
# After reviewing an intentional change locally:
npx backstop approve3. Add a GitLab CI job
The following is a starting pattern. Replace the image with one compatible with your pinned BackstopJS and application, and provide your actual build and app startup commands. Set paths.ci_report in BackstopJS configuration to match the report directory below.
visual_regression:
stage: test
image: node:20
script:
- npm ci
- npm run build
# Start the app or connect to the environment under test here.
- npx backstop test
artifacts:
when: always
paths:
- backstop_data/ci_report/
- backstop_data/bitmaps_test/
reports:
junit: backstop_data/ci_report/xunit.xmlBackstopJS’s CI report produces JUnit XML by default, with a documented default filename of xunit.xml; its report directory and filename can be configured. Point GitLab to the exact file your configuration produces. GitLab accepts an XML filename, glob, or array of paths; a directory alone is not a valid JUnit report path. Including report and capture files under artifacts:paths makes them downloadable, while artifacts:when: always asks GitLab to upload artifacts after failure too.
Keep npx backstop test in the job’s script so its exit status determines success. GitLab states that JUnit report ingestion does not affect job status. Verify the pinned BackstopJS version returns non-zero for a visual failure before treating this job as a merge gate.
4. Make the application reachable
The screenshot browser must be able to load every scenario URL before the test runs. Choose an arrangement suited to the runner:
- Build and serve the app in the same job, then wait for the server to become ready before invoking BackstopJS.
- Use a separate job or deployed test environment, and ensure the visual-test job runs after it and can route to its URL.
- If using GitLab services or containers, use the hostname and port visible from the job’s network. Do not assume
localhostrefers to another container or to the runner host.
The exact service networking and startup command depend on your runner and project. A scenario URL that works on a developer’s machine may fail in CI if its hostname is private to that machine or the application is not ready yet.
5. Choose a rendering environment
BackstopJS can run its rendering browser directly or use its documented --docker option. Docker can reduce differences between rendering environments by using a versioned BackstopJS image, but it requires the GitLab runner to have Docker access and introduces container networking and filesystem-permission considerations.
npx backstop test --dockerConfirm the exact command options for the version pinned in your project. BackstopJS documents removing -t from its default Docker command template for CI-like output where output is piped. Ensure the runner can start the container, reach the application from inside it, and write report and screenshot artifacts where GitLab can collect them. The hostname host.docker.internal is mentioned for particular Mac/Windows Docker setups; runner networking differs, so do not copy it without checking the route available in your GitLab environment.
6. Make failures and reports useful
- Use the process status as the gate. JUnit reports improve GitLab test visibility; they do not replace the command’s exit code.
- Keep report paths aligned. The configured CI report directory and filename must match
artifacts:reports:junit. - Retain failure evidence. Upload the report and test screenshots with
when: alwaysso failed comparisons are diagnosable. - Keep names unique. GitLab ignores duplicate test names after the first occurrence in JUnit reports.
- Watch report limits. GitLab documents a limit below 30 MB per JUnit file and below 100 MB total per job.
GitLab supports screenshot attachments through JUnit system-out attachment tags when the corresponding screenshot files are also uploaded as artifacts. Use this if you want test details to link to capture evidence in GitLab’s test view.
7. Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Scenario navigation fails or times out | The URL is not reachable from the runner or capture container, or the app is not ready. | Use a runner-reachable hostname, check container routing, and wait for the app’s readiness before running the test. |
| Works locally, differs in CI | Browser, fonts, OS, viewport, or rendering environment differs. | Pin dependencies and viewport settings; consider --docker if the runner supports it, then compare artifacts from the same environment. |
| JUnit report is missing | CI reporting is not enabled, or the configured output path differs from GitLab’s report path. | Enable "report": ["CI"], configure the CI report directory if needed, and point reports:junit at the produced XML file. |
| Pipeline passes despite visual failures | The report was ingested, but the test process returned zero or its status was masked. | Ensure the test command’s failure status reaches the job script and confirm behavior with the pinned version. Do not rely on JUnit ingestion to fail the job. |
| Artifacts disappear on failure | Artifacts are configured to upload only on success. | Set artifacts:when: always and include the actual report and screenshot directories under paths. |
| Docker command hangs or cannot start | The runner lacks Docker access or a TTY option is unsuitable for piped CI output. | Configure runner container access, use the documented CI-compatible command template without -t where applicable, or run without Docker. |
| GitLab shows no test cases or rejects the report | The XML path or format is wrong, the extension is not .xml, or size limits are exceeded. |
Inspect the generated file, use the correct path with an XML extension, and keep each file below 30 MB and total JUnit reports below 100 MB per job. |
| Old visual differences appear after a design update | The approved reference set still represents the previous design. | Review the test captures, then deliberately run backstop approve and commit the reviewed baseline update. |
8. Performance, reliability, and cost
BackstopJS runs browser captures, so pipeline time depends on scenario count, viewport count, page load behavior, and runner resources. Keep scenarios focused on pages and states where visual regressions matter; avoid starting overlapping test jobs against an unstable shared deployment. Ensure the app is ready before capture and retain failure artifacts so intermittent load problems can be distinguished from genuine visual changes.
For consistent comparisons, keep approved references in version control or another reproducible artifact store, pin the dependency through the lockfile, and use a stable rendering setup. Docker may help standardize rendering but adds runner setup and routing requirements. The CI cost is the time and resources consumed by browser jobs and retained artifacts; no external pricing or benchmark is implied here.
9. Or skip the browser setup
If you need a clean capture in a script without maintaining a browser environment, ScreenshotNeo is a website screenshot API and MCP server for developers. Its one-call API returns an image or PDF, and its documentation lists the request options.
curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webpimport requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);- Cookie banners, popups, and chat widgets are removed before the shot; each cleanup step can be turned off.
- Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed; response headers identify the page verdict and billing status.
- An MCP server lets AI agents use screenshot, page information, and PDF capture tools.
- 1,000 screenshots a month are free with no card; paid plans start at $5 for 3,000 screenshots.
Sign up for 1,000 free screenshots a month, with no card required.
FAQ
Does GitLab fail a pipeline when a JUnit test fails?
No. JUnit reports provide test-result visibility. The job script must exit non-zero to fail the pipeline.
Should reference screenshots be generated during every pipeline run?
No. Treat references as approved baselines. Review visual changes and update them intentionally with backstop approve.
Is Docker required for BackstopJS in GitLab CI?
No. It is an option for rendering consistency, subject to runner support and container networking.
Can the scenario use localhost?
Only if localhost from the capture process reaches the intended app. In containerized jobs, that may not be the host or another service; verify the runner’s network context.
