ScreenshotNeo

BlogHow-to

How to Run Playwright Tests with Azure Test Plans

Run Playwright in Azure Pipelines, publish results, and understand the current limits of Azure Test Plans automation.

By the ScreenshotNeo team30 September 202610 min read

How to Run Playwright Tests with Azure Test Plans

Short answer: run Playwright from an Azure Pipeline job, configure it to emit a supported test-result format, and publish that file with PublishTestResults@2. The results then appear in the pipeline’s Tests view. Associating a Playwright method with an Azure Test Plans test case is a separate workflow, and Microsoft’s currently documented framework lists name JavaScript Jest but do not name JavaScript Playwright. Check your Azure DevOps tenant and task version before relying on plan-triggered Playwright execution.

This guide covers both goals: reliable CI reporting and test-case traceability. It also shows where Azure Test Plans support ends, how to troubleshoot failures, and when hosted browser execution may make sense.

1. Understand the two Azure DevOps workflows

“Run Playwright tests with Azure Test Plans” can mean two different things:

  1. Run tests in Azure Pipelines and report results. Your pipeline executes npx playwright test. A reporter writes a result file, and PublishTestResults@2 uploads it. The run is visible on the pipeline’s Tests tab.
  2. Associate or execute an automated test from Test Plans. A test method is linked to an Azure Test Plans test case. Supported task and framework combinations may then allow a plan or test run to select that automation.

The first workflow is the normal route for Playwright CI. The second depends on Azure DevOps framework support and your project’s current task experience. Microsoft’s automated-testing overview describes pipeline execution, test-result review, and association as related but distinct activities.

Before changing YAML, decide which outcome you need. If the requirement is “show Playwright pass/fail results in Azure DevOps,” use the pipeline route below. If the requirement is “start a Playwright method from a Test Plans test case,” treat framework compatibility as a gate.

2. Prepare a Playwright project

Use the Playwright version and package manager already defined by your repository. A minimal JavaScript project can be created with:

npm init playwright@latest

Commit the generated configuration and tests. A basic test might look like this:

import { test, expect } from '@playwright/test';

test('home page has a title', async ({ page }) => {
  await page.goto('https://example.com');
  await expect(page).toHaveTitle(/Example Domain/);
});

For CI, make the test command explicit in package.json:

{
  "scripts": {
    "test:e2e": "playwright test"
  }
}

Install browser binaries during the pipeline rather than assuming the hosted agent has the exact browser revision your lockfile expects:

npx playwright install --with-deps

The --with-deps option is useful on Linux agents because it installs system packages required by the browsers. If your organization uses a custom agent image with those packages already installed, npx playwright install may be sufficient.

3. Configure a result format Azure Pipelines can publish

PublishTestResults@2 documents these result formats: CTest, JUnit (including PHPUnit), NUnit 2/3, Visual Studio Test/TRX, and xUnit 2. Playwright’s built-in reporters can produce several formats, but your selected reporter must match the format named in the publishing task. The generic approach for a non-Microsoft runner is to generate a supported file and publish it after the test command.

A Playwright run produces a supported result file that Azure Pipelines can publish.
A Playwright run produces a supported result file that Azure Pipelines can publish.

JUnit is a practical interchange format. Configure it in playwright.config.ts:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  reporter: [
    ['list'],
    ['junit', { outputFile: 'test-results/playwright-junit.xml' }]
  ],
  use: {
    trace: 'retain-on-failure'
  },
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : undefined
});

Keep the output directory predictable and create it as part of the build if your reporter does not do so automatically. Do not publish a path that differs from the path Playwright actually writes.

4. Add an Azure Pipelines job

The following YAML installs Node.js, restores dependencies, installs browsers, runs Playwright, and publishes JUnit results:

trigger:
- main

pool:
  vmImage: ubuntu-latest

steps:
- task: NodeTool@0
  inputs:
    versionSpec: '20.x'
  displayName: Use Node.js

- script: npm ci
  displayName: Install dependencies

- script: npx playwright install --with-deps
  displayName: Install Playwright browsers

- script: npm run test:e2e
  displayName: Run Playwright tests
  continueOnError: true

- task: PublishTestResults@2
  condition: succeededOrFailed()
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: 'test-results/playwright-junit.xml'
    searchFolder: '$(System.DefaultWorkingDirectory)'
    mergeTestResults: true
    failTaskOnFailedTests: true
    testRunTitle: 'Playwright browser tests'
  displayName: Publish Playwright results

continueOnError: true lets the publishing step run when tests fail. The publishing task uses condition: succeededOrFailed() for the same reason. With failTaskOnFailedTests: true, the task reports the failures and still leaves the job failed. If you want a reporting-only step that does not fail the task, set that input to false.

For a monorepo, replace the commands and result path with the package directory. For multiple projects, either publish each file separately or configure mergeTestResults: true with a glob such as **/test-results/*.xml. Keep result filenames unique when parallel jobs upload artifacts.

5. Confirm results in Azure DevOps

  1. Queue the pipeline and wait for the Playwright step and publishing step to finish.
  2. Open the completed pipeline run.
  3. Select the Tests tab.
  4. Check that the expected test names, totals, duration, and pass/fail outcomes are present.
  5. Open a failed test and review the error, attachments, and any trace or screenshot artifacts you separately published.

A result file can be valid XML yet still produce an unhelpful run if its test-suite names or paths are unexpected. Inspect the generated file as a build artifact when diagnosing missing cases.

6. Publish traces, screenshots, and videos

Test-result publishing and browser-artifact publishing are separate concerns. Playwright can retain traces on failure, and you can upload the result directory as a pipeline artifact:

- task: PublishPipelineArtifact@1
  condition: succeededOrFailed()
  inputs:
    targetPath: '$(System.DefaultWorkingDirectory)/test-results'
    artifact: 'playwright-results'
  displayName: Upload Playwright artifacts

If your Playwright configuration writes screenshots, videos, or traces outside test-results, upload that directory too. Keep artifacts on failed runs by using succeededOrFailed(). Avoid uploading browser caches or the entire workspace; they increase storage and make diagnosis slower.

7. Associate automation with an Azure Test Plans case

For supported frameworks, Microsoft’s association workflow requires a test to run at least once before it becomes available for association from pipeline results. The general sequence is:

  1. Run the pipeline and publish results successfully.
  2. Open the pipeline run’s Tests tab.
  3. Select the relevant automated test.
  4. Choose the action to associate it with an existing test case.
  5. Open the test case and verify its associated automation details.

This association adds work-item traceability. A test method may be associated with multiple test cases, while a test case can have only one associated test method according to Microsoft’s automated-testing overview. If the test case is linked to a requirement, the association can help connect requirement coverage to automated outcomes.

8. What Playwright can and cannot do through Test Plans

Microsoft’s Azure Test Plan task reference labels AzureTestPlan@0 as Public Preview. Its documented language choices include Java Maven, Java Gradle, Python PyTest, and JavaScript Jest. The association guide and FAQ likewise name JavaScript Jest, but the reviewed documentation does not name JavaScript Playwright.

A separate Microsoft roadmap entry says the service planned to expand supported frameworks to allow linking and executing JavaScript Playwright tests. That roadmap page was last updated April 22, 2025. A dated roadmap statement is not proof that the capability is available in your tenant today.

Therefore:

  • Use Azure Pipelines plus PublishTestResults@2 for dependable Playwright CI reporting.
  • Do not promise plan-triggered Playwright execution until your tenant exposes a supported option.
  • Check the current Azure Test Plan task documentation and your project’s task UI before designing a Test Plans workflow around Playwright.
  • If the task rejects the framework, keep the pipeline reporting workflow and use Test Plans for manual planning and traceability where appropriate.

9. Optional hosted browsers with Playwright Workspaces

Microsoft documents Playwright Workspaces as an optional hosted-browser route. It can run Playwright tests in cloud-hosted browsers through a CI workflow and publish results in the Azure portal. It is not required for ordinary Azure Pipelines execution or result publication.

Choose this route when your team specifically needs hosted browser infrastructure or wants to separate browser capacity from build agents. Microsoft’s quickstart warns that charges depend on total test minutes, so estimate suite duration, retry behavior, parallelism, and scheduled frequency before enabling it. Verify current service availability, limits, and pricing in Microsoft’s documentation because those details can change.

10. Troubleshooting

No tests appear on the Tests tab

Cause: the publish task did not run, the file path is wrong, or the file format does not match testResultsFormat.
Fix: use condition: succeededOrFailed(), print the result directory with ls -la test-results, and confirm the XML is JUnit if the task is configured for JUnit.

The pipeline stops before results are uploaded

Cause: the Playwright command fails the job immediately.
Fix: set the test step to continueOnError: true or use a job-level condition that permits the publishing step to run. Keep failTaskOnFailedTests aligned with whether the publish task should be the final failure point.

Browser executable is missing

Cause: the agent does not contain the browser revision required by your Playwright package.
Fix: run npx playwright install --with-deps after npm ci. If you use a private agent, confirm that its OS and system libraries are supported.

Tests pass locally but fail in Azure

Cause: differences in viewport, timezone, fonts, environment variables, network access, or browser dependencies.
Fix: log the effective configuration, set required secrets as pipeline variables, use deterministic test data, and retain traces on failure. Avoid relying on a developer’s logged-in browser profile.

Only some tests are reported

Cause: the reporter writes multiple files, a glob excludes files, or parallel jobs overwrite one output file.
Fix: give each project or shard a unique result filename, publish with a deliberate glob, and enable mergeTestResults only when combining compatible files.

Azure Test Plans will not accept the automation

Cause: Playwright is not exposed as a supported framework for the current association or preview task.
Fix: continue publishing pipeline results, confirm the current framework list, and do not substitute Jest labels for a Playwright runner unless Microsoft explicitly documents that configuration.

11. Reliability, performance, and cost practices

Concern Practical choice
Flaky browser state Use isolated contexts, stable test data, and trace retention on failure.
Pipeline duration Use Playwright workers carefully; excessive parallelism can overload the agent or application.
Retries Limit CI retries so real regressions remain visible and runtime does not grow silently.
Result reliability Always publish with a failure-tolerant condition and verify the output path.
Storage Upload traces and screenshots for failures first; avoid archiving browser caches.
Hosted execution For Playwright Workspaces, budget for total test minutes and confirm current pricing.

Viewing test work items or running test cases requires at least Basic access according to Microsoft’s overview. Full Test Plans features require Basic plus Test Plans or an eligible Visual Studio subscription. Verify current licensing for your organization before rollout.

ScreenshotNeo removes common consent and overlay elements before capture.
ScreenshotNeo removes common consent and overlay elements before capture.

Or skip the browser setup

If your goal is to capture a page image or PDF rather than maintain a browser-test pipeline, ScreenshotNeo provides a single website screenshot API call. Its capture flow accepts consent banners like a visitor and removes more than 60 known consent platforms, newsletter popups, and chat widgets before the shot; each step can be disabled. Bot checks, blank pages, timeouts, failed loads, and cache hits are not billed, and the response identifies the page verdict and billing status in X-Page-Verdict and X-Billed headers. It also includes an MCP server with take_screenshot, get_page_info, and capture_pdf tools for Claude, Cursor, and other MCP clients.

See the ScreenshotNeo API documentation for all options. A direct request looks like this:

curl -G "https://api.screenshotneo.com/v1/shot" -d access_key=YOUR_API_KEY --data-urlencode url=https://stripe.com -o shot.webp

Python:

import requests
r = requests.get("https://api.screenshotneo.com/v1/shot", params={"access_key": "YOUR_API_KEY", "url": "https://stripe.com"}, timeout=90)
open("shot.webp", "wb").write(r.content)

Node.js:

const q = new URLSearchParams({ access_key: 'YOUR_API_KEY', url: 'https://stripe.com' });
const res = await fetch(`https://api.screenshotneo.com/v1/shot?${q}`);

ScreenshotNeo supports full-page and element captures, device presets, custom viewports, dark mode, retina scale, PDF output, custom CSS and JavaScript, click and wait actions, request blocking, headers, cookies, user agents, authorization, timezone, geolocation, transparent backgrounds, resizing, caching, signed links, asynchronous jobs, webhooks, bulk capture, usage reporting, and an OpenAPI specification. The free plan includes 1,000 screenshots per month with no card; paid plans start at $5 for 3,000. Create a free ScreenshotNeo account.

FAQ

Does Azure Test Plans run Playwright tests directly?

Pipeline execution is supported through your own Playwright command. Direct plan-triggered execution depends on current framework support, and the reviewed Microsoft lists do not name JavaScript Playwright.

Which result format should I publish?

Use a format documented by PublishTestResults@2, such as JUnit, NUnit, TRX, xUnit 2, or CTest. JUnit is a common choice for Playwright integration.

Do I need Playwright Workspaces?

No. It is an optional hosted-browser service. A normal Azure Pipeline agent can install browsers and run Playwright locally on that agent.

Why associate a test with a Test Plans case?

Association adds work-item traceability and can connect automated outcomes to planned test cases and requirements when the framework and task support it.

Can I publish results if Playwright fails?

Yes. Let the test step continue long enough for the publishing task to run, use a failure-tolerant condition, and configure whether the publishing task itself fails on failed tests.