How to Analyze Tests and Integrate Them with CI/CD

To analyze tests in CI/CD, make three separate outputs part of the workflow: run the test suite with an exit status that gates the change, publish a machine-readable test report such as JUnit XML, and publish coverage data in the format your CI platform accepts. Preserve reports and diagnostic artifacts even when tests fail. A dashboard can display a failure without making the pipeline fail, so verify that the test command or pipeline step controls status.

This guide covers a practical workflow and configurations for GitLab CI/CD and Jenkins, plus the documented GitHub Actions coverage path. The examples use the official platform documentation as their reference; adapt commands and report paths to your test runner.

1. Separate test execution, test results, and coverage

These outputs answer different questions:

Output	Question it answers	Typical input
Test execution and exit status	Did the suite pass, and should the pipeline gate the change?	Test command exit code or CI step status
Test result report	Which tests passed, failed, or were skipped?	JUnit XML consumed by GitLab or Jenkins
Coverage report	Which code was exercised under this measurement?	Coverage summary, Cobertura XML, or JaCoCo XML, depending on the platform
Diagnostic artifacts	What evidence helps reproduce or investigate a failure?	Logs, screenshots, traces, and other files produced by the runner

Do not treat coverage as a verdict on test quality. It describes what code ran under a particular measurement; it does not establish that assertions check the intended behavior. Review coverage alongside failing tests and the changes under review.

2. Build a useful CI test workflow

Run the relevant suite. Start with the tests appropriate for the change, such as a unit suite, and keep its failure status meaningful.
Generate machine-readable test results. Configure the runner or formatter to write JUnit XML if your CI platform consumes it.
Publish the report. Point the platform configuration at the file the runner actually creates.
Preserve results after failure. Upload the report and useful logs or screenshots even if the test command exits non-zero.
Publish coverage separately. Confirm whether the platform expects a log percentage, a line-coverage report, or both.
Review the evidence. Inspect test names, failure details, logs, and artifacts before changing tests or loosening a gate.

When available, compare source-branch results with the target branch. GitLab’s merge request test report can compare test results between those branches and show failure details and screenshots. Its report display does not itself control job status: the test script must exit non-zero for a failed test to fail the job. See the GitLab unit test reports documentation.

3. Publish JUnit results in GitLab CI/CD

GitLab accepts JUnit XML through artifacts:reports:junit. The following job is adapted from the official RSpec example. It assumes the project has RSpec and the RspecJunitFormatter dependency configured.

ruby:
  stage: test
  script:
    - bundle install
    - bundle exec rspec --format progress --format RspecJunitFormatter --out rspec.xml
  artifacts:
    when: always
    paths:
      - rspec.xml
    reports:
      junit: rspec.xml

when: always makes the report artifact upload after a failed job, which helps keep failure details available. The report path must match the file written by the test command. GitLab accepts a file path, filename pattern, or array of report paths; it does not accept a directory as the report path. Check the GitLab report configuration details when adapting this example.

Make the job status reflect test failures

The test command’s exit status must remain non-zero on test failure. Avoid shell patterns or wrapper scripts that swallow that status. The report is for readable test results; it does not replace the job’s pass/fail signal.

4. Configure GitLab coverage as a separate output

GitLab has two distinct coverage reporting paths:

Percentage in the job or merge request view: configure a coverage regular expression that extracts a percentage from job log output.
Changed-line annotations: upload a Cobertura or JaCoCo XML report with artifacts:reports:coverage_report. Annotations appear for files changed in the merge request.

Configure both if you need both experiences. A percentage extraction setting does not automatically create line annotations. Test the regular expression against actual job output; ANSI color codes or output format changes can prevent a match. See GitLab’s documentation for code coverage and coverage reporting.

The exact coverage command and report-generation configuration depend on the language and coverage tool. Keep that tool-specific setup separate from the GitLab upload settings, and verify the generated report file exists at the configured path.

5. Consume test results in Jenkins

Jenkins’ JUnit Pipeline step consumes test-result XML. The documented format is also used by TestNG. A typical Pipeline stage looks like this; replace the placeholder command and glob with the project’s test command and generated XML path:

pipeline {
  agent any
  stages {
    stage('Test') {
      steps {
        sh './run-tests'
      }
      post {
        always {
          junit testResults: '**/build/test-results/**/*.xml'
          archiveArtifacts artifacts: 'logs/**, screenshots/**', allowEmptyArchive: true
        }
      }
    }
  }
}

Ensure the glob matches only the intended test-result XML files. The test command’s status should still determine whether the stage or build passes. Jenkins’ JUnit step has configuration that affects whether failures mark a build or stage unstable; choose behavior deliberately in the JUnit Pipeline step documentation.

Jenkins warns that including every passing-test log message can substantially increase memory consumption. Keep report output useful and bounded. Jenkins also documents recording and aggregating test results and retrieving build artifacts to investigate failures in its tests and artifacts guide.

6. Publish coverage in GitHub Actions

The cited GitHub instructions describe a coverage workflow that generates Cobertura XML from tests run in GitHub Actions and uploads it for pull-request coverage results. Keep coverage generation and upload as explicit workflow steps, and confirm the generated file format and path match the upload configuration. Consult GitHub’s official code coverage setup for the current supported setup.

The referenced GitHub source covers code coverage; it does not establish a general test-result publishing or test-failure gating configuration. Treat those as separate workflow requirements and follow the documentation for your test runner and any result-reporting integration you use.

7. Analyze a failing pipeline

Find the failed test name and its error or assertion details in the report.
Read the test job log around the first failure; later errors can be consequences of that failure.
Open saved logs, screenshots, or other artifacts from the failed run.
Compare the source branch with the target branch where the platform provides that view.
Decide whether the evidence points to a product regression, a test defect, an environment problem, or a flaky dependency.
Make a focused correction, then confirm that the report and status signal agree on the result.

A useful failure report tells a reviewer what failed and leaves enough evidence to investigate without rerunning blindly. Preserve only artifacts that help diagnosis, and keep their paths stable so developers know where to find them.

8. Common errors and fixes

Symptom	Likely cause	Fix
The job fails but the report is missing	Artifacts are uploaded only on success, or the report was never written.	Use an always-upload setting where supported and confirm the runner creates the configured file before the job ends.
The test report view is empty	The report path is wrong, a directory was specified where a file or pattern is required, or the XML is not in the expected format.	Inspect the workspace after the test step, correct the exact path or glob, and validate the report format against the CI documentation.
Tests fail but the pipeline passes	A script or wrapper swallowed the test command’s non-zero exit status, or the reporting integration only displays failures.	Propagate the test runner’s exit status and configure the pipeline step’s failure behavior explicitly.
Coverage percentage is absent	The regular expression does not match actual log output, perhaps because output changed or includes ANSI color codes.	Test the expression against the raw job log and adjust it to the real output format.
Coverage percentage appears but changed lines have no annotations	Only log extraction was configured; no Cobertura or JaCoCo report was uploaded, or no changed file has annotations to show.	Configure the coverage report artifact separately and verify the generated XML path and format.
Jenkins becomes memory-heavy while recording results	Every passing test log message is being included in the JUnit result processing.	Reduce unnecessary passing-test log output and consult the JUnit step options.
Jenkins finds unexpected XML files	The result glob is too broad and matches unrelated XML.	Narrow the glob to the test runner’s result directory and expected report names.

9. Performance, reliability, and cost considerations

Keep feedback focused. Run the relevant suite for the change and make broader suites an intentional pipeline stage when appropriate.
Preserve the useful evidence. Reports, logs, and screenshots support diagnosis after the original test process has ended; keep them available on failure.
Control report size. Jenkins documents that retaining every passing-test log message can substantially increase memory use.
Keep status and reports aligned. A readable report is valuable only if the actual job status still gates failed tests as intended.
Verify configuration after changes. Runner output, file paths, coverage formats, and CI platform behavior can change. Recheck the official documentation and a real pipeline run when updating the workflow.
Account for CI usage. Pipeline runtime and artifact storage have costs determined by your runner and platform plan. The cited sources do not provide comparable pricing figures, so review the terms for the services you use.

10. Or skip the browser setup

If your test or review workflow also needs a screenshot of a page, ScreenshotNeo provides a one-call website screenshot API. The ScreenshotNeo API docs cover its request options.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

import requests

r = requests.get(
    "https://api.screenshotneo.com/v1/shot",
    params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"},
    timeout=90,
)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

ScreenshotNeo removes cookie and consent banners, newsletter popups, and chat widgets before the screenshot; each cleanup step can be turned off. Bot checks, blank pages, failed loads, timeouts, and cache hits are not billed, and response headers say which page verdict and billing outcome applied. Its MCP server lets AI agents use take_screenshot, get_page_info, and capture_pdf. The free plan includes 1,000 screenshots a month with no card; paid plans start at $5 for 3,000 screenshots. Every feature is on every plan.

Frequently asked questions

Does a JUnit report make a failing pipeline fail?

Not by itself in GitLab. The test command must return a non-zero status for the failed test to fail the job. In Jenkins, configure the JUnit step and pipeline status behavior deliberately.

Is a coverage percentage enough?

No. It summarizes measured execution but does not show whether tests assert the right behavior. It also may not provide line annotations; those require a supported coverage report and separate configuration.

Which report should I start with?

Start with the machine-readable test-result format supported by your CI platform and test runner. JUnit XML is the integration path documented in the cited GitLab and Jenkins examples.

Should every CI run keep screenshots and logs?

Preserve artifacts that help investigate failures, especially when a job can fail before a developer sees its live output. Choose retention and artifact scope according to your team’s debugging needs and platform configuration.