How to Automate Tests in a CI Pipeline

To automate tests in a CI pipeline, connect your repository to a CI service, trigger a workflow when a change is proposed, install the project’s declared dependencies, run fast checks and tests, and show the outcome to reviewers. Start with a small, reliable set of checks. Add integration and end-to-end tests where they cover risks that lower-level tests cannot.

This guide uses GitHub Actions for a complete Python example. The same sequence applies to GitLab CI/CD, Jenkins, and other systems, though their event syntax, runners, report formats, and configuration differ. GitHub describes Actions as a platform for automating build, test, and deployment pipelines, with workflows triggered by repository events and run as jobs on hosted or self-hosted runners (GitHub Actions overview, continuous integration).

1. Decide what the pipeline should prove

Before writing configuration, list the checks that protect a change and the command that runs each check locally. A useful starting order is:

Fast static checks: formatting, linting, type checks, or validation, when the project uses them.
Unit tests: focused tests of individual functions or modules.
Integration tests: checks that cover important interactions, such as a service and its database.
End-to-end tests: a small number of checks for complete user journeys that need the whole application.

Fast failures are easier to diagnose, so run quick checks early. Avoid duplicating the same behavior at every test level without a reason. GitLab’s testing guidance recommends progressive testing and fast feedback; its end-to-end guidance describes tests across the whole stack and cautions against adding end-to-end coverage when a lower-level test already covers the feature (GitLab testing standards and guidance, end-to-end testing).

2. Add a GitHub Actions workflow

The following example runs a Python project’s tests for pull requests and pushes to the default branch. It assumes the repository has requirements.txt listing its dependencies and pytest, plus tests discoverable by python -m pytest. Change the Python version and install/test commands to match your project. Save it as .github/workflows/tests.yml.

name: Tests

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  tests:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - name: Check out source
        uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
      - name: Run tests
        run: python -m pytest

Commit the file to the repository. GitHub Actions will then create runs for the configured events, and the result appears with the pull request checks. Configure branch protection or the repository’s merge rules to require the relevant check if a passing run should be a merge condition. The exact settings depend on the repository’s policy.

What each part controls

on selects triggering events. This example checks proposed pull requests and pushes to main. Add other branches or scheduled runs only when they serve a purpose.
permissions limits the workflow token to reading repository contents. Grant additional permissions only if a job needs them.
runs-on selects the runner. GitHub offers hosted virtual machines and supports self-hosted runners when a team needs its own machine or environment (GitHub runner concepts).
timeout-minutes prevents a stuck job from consuming runner time indefinitely. Set it based on the expected duration of this job.
The steps check out code, set up the runtime, install dependencies, and run tests. Replace dependency setup and test commands with the project’s documented local equivalents.

3. Adapt the pattern to your stack

CI does not require a special test command. It runs the same commands developers use locally, inside a runner with the required runtime and services. For example, a Java project might run mvn test; a JavaScript project might install from its lockfile and run its test script; a compiled project might build before testing. Keep these commands explicit and make failures return a nonzero exit code so CI can mark the job as failed.

For projects requiring a database, queue, or other service, provision the dependency for the job or connect to an approved test instance. Use disposable test data and isolate parallel runs so one job cannot corrupt another. Do not rely on undocumented state left behind by an earlier run: GitHub-hosted jobs run on fresh virtual machines, while self-hosted environments need their own cleanup and isolation practices.

GitLab CI/CD and Jenkins examples

In GitLab, define jobs in the repository’s CI configuration and run them for merge-request or branch pipelines. Configure the test runner to emit a supported report when you want structured test details and coverage in the merge request; exact report formats depend on the framework and CI configuration. GitLab documents feature-branch testing and test reports in its testing documentation.

In Jenkins, put pipeline stages in a Jenkinsfile, run the project’s test command, and publish the test report in a post-build step. For example, if Maven writes JUnit XML under target/surefire-reports:

pipeline {
    agent any
    stages {
        stage('Test') {
            steps {
                sh 'mvn test'
            }
        }
    }
    post {
        always {
            junit 'target/surefire-reports/*.xml'
        }
    }
}

The junit publisher records results when the command succeeds or fails, provided report files were created. Jenkins documents this reporting pattern in its tests and artifacts guide.

4. Make test results useful in code review

A red or green status is the minimum. Preserve enough output for a developer to understand a failure without rerunning the job blindly:

Keep test-runner output in the job log.
Publish structured test reports when the platform and test runner support them.
Preserve relevant logs or diagnostic artifacts for failed integration and browser tests, while avoiding credentials and sensitive user data.
Use distinct job names for distinct responsibilities, such as unit tests and browser tests.
Require the stable checks that match the risk of the change. If a suite is flaky, investigate and fix its instability before treating it as a dependable gate.

GitHub reports CI outcomes in pull requests. GitLab supports test and coverage reports, and Jenkins can collect JUnit XML reports and show test results in its interface (GitHub CI, GitLab testing, Jenkins reports).

5. Choose what blocks a merge

Make a check required when its result is stable, fast enough for the team’s workflow, and relevant to the change risk. Unit tests and essential build checks are common initial gates. Integration tests can block when they reliably validate critical interactions. End-to-end tests are most useful for a limited set of journeys that require the full application; their runtime and setup often make them less suitable as the only or earliest feedback.

There is no universal rule that every end-to-end test must block a merge. Decide based on impact, reliability, and how quickly failures can be diagnosed. A scheduled broader suite can find issues beyond the pull-request checks, but it does not replace fast feedback on a proposed change.

6. Keep the pipeline reliable and affordable

Keep setup reproducible. Install from lockfiles or pinned dependency declarations where the ecosystem supports them, and make required services explicit.
Match the environment to production where it matters. Differences in runtime, operating system, database, or configuration can hide defects. GitLab’s best-practice guidance emphasizes environment similarity and simple pipelines (GitLab pipeline efficiency).
Use caching carefully. Cache downloaded dependencies to reduce repeated setup, but do not use a cache as the source of truth for build outputs or test state. A stale cache should be safe to remove.
Parallelize only independent work. Splitting a suite can shorten elapsed time, but shared resources, fixed ports, or shared test data can create collisions. Measure the effect in your own pipeline rather than assuming more jobs cost less.
Control expensive suites. Run focused checks on every proposal; reserve broad cross-platform or long-running suites for changes that need them or for scheduled runs.
Review runner and service usage. Hosted and self-hosted runner costs and limits vary by provider and plan. This guide does not make a price comparison; check the CI provider’s current terms for your account.
Assign ownership. A team should know who investigates flaky or repeatedly failing tests. Unowned failures erode confidence in the pipeline.

7. Troubleshoot common CI failures

Symptom	Likely cause	Fix
Workflow does not run on a pull request	The event or branch filter does not match, the file is not in the platform’s expected location, or the workflow has a YAML error.	Check the CI configuration’s location and event filters, inspect the workflow validation/run details, and open a pull request that matches the configured event.
Dependency installation fails	The runner runtime differs from local development, a dependency is unavailable, or the project’s install command is incomplete.	Declare the runtime and dependencies in the repository, use the same install process as local development, and inspect the first package-manager error in the log.
Tests pass locally but fail in CI	Environment differences, missing environment variables, timezone/locale assumptions, test-order dependence, or shared external state.	Compare runtime and service versions, declare required configuration, isolate test data, and make each test independent of execution order.
Job times out	A process is hanging, a test waits indefinitely, or the configured timeout is too short for legitimate work.	Find the last log line, add bounded waits to tests, clean up child processes, and adjust the timeout only after confirming expected duration.
Report is missing even though tests ran	The report path or format does not match the publisher configuration, or the test process stopped before writing it.	Confirm the runner generated report files at the expected path and configure the platform’s report collector for that format.
Results are inconsistent between runs	Flaky assertions, race conditions, external dependencies, shared state, or insufficient cleanup.	Capture diagnostics, remove timing assumptions, isolate resources, and assign the test for repair. Avoid hiding repeated failures with automatic retries alone.
A forked contribution cannot access a secret	CI platforms commonly restrict secrets for untrusted contributions to protect the repository.	Do not expose privileged credentials to untrusted code. Separate secret-dependent checks into a trusted process and follow the platform’s security guidance.

Or skip the browser setup

If your CI job needs a screenshot of a page—for a visual review artifact, for example—you can drive a browser yourself, or use ScreenshotNeo, a website screenshot API and MCP server from Yorker Media. Its API accepts one GET request and returns an image or PDF; see the ScreenshotNeo API documentation.

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

import requests

r = requests.get(
    "https://api.screenshotneo.com/v1/shot",
    params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"},
    timeout=90,
)
open("shot.webp", "wb").write(r.content)

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);
if (!res.ok) throw new Error(`Screenshot request failed: ${res.status}`);
await Bun.write('shot.webp', res);

Use the API key as a CI secret, not a literal committed to the workflow. ScreenshotNeo removes cookie and consent banners, newsletter popups, and chat widgets before capture. Bot checks, blank pages, failed loads, timeouts, and cache hits are not billed; response headers identify the page verdict and billing status. Its MCP server lets AI agents use screenshot and PDF capture tools. The free plan includes 1,000 shots per month with no card; paid plans start at $5 for 3,000 shots. The same features are available on every plan.

Create a free ScreenshotNeo account for 1,000 screenshots a month, with no card required.

Frequently asked questions

Do I need a dedicated computer to run CI?

No. GitHub provides hosted virtual machines, and self-hosted runners are an option when a team needs its own environment or hardware.

Should CI run on every commit or only pull requests?

Choose events that provide useful feedback without running redundant work. Pull-request checks catch issues before merge; branch pushes and scheduled runs can serve additional workflows.

Can CI run tests that need a browser?

Yes. Provide the browser and application environment the test requires, then publish useful logs and reports. Keep browser journeys focused on behavior that needs a real browser.

What should happen when a test is flaky?

Capture evidence and repair the test or its environment. A check that fails unpredictably cannot give reviewers a dependable signal.

Which CI platform should I choose?

Start with the service already connected to your repository, then compare runner needs, review integration, reporting, framework compatibility, and operational ownership. The capabilities described here do not establish a universal platform winner.