Lesson 22 / 25

Running in CI

Headless runs, retries and artifacts.

What a CI job needs

A CI job installs dependencies, installs browsers (npx playwright install --with-deps for Playwright; Cypress downloads its binary on install, and caching that binary speeds builds), starts or builds the app, runs tests headless, and uploads artifacts such as the Playwright HTML report and traces or Cypress screenshots and videos, even when tests fail. Configure retries only in CI: Playwright retries: process.env.CI ? 2 : 0; Cypress retries: { runMode: 2, openMode: 0 }. Use secrets for credentials, pin tool versions, and consider the official Docker images so browsers and fonts match between runs.

Fast feedback on every pull request

E2E suites earn their keep when they run headless in CI, in parallel, with artifacts that explain failures.

Four ideas: CI pipelines, parallelism and sharding, choosing a tool, a final checklist.
Figure 8.1 — A CI pipeline with shards, retries and uploaded reports.

A GitHub Actions workflow for Playwright

Action versions change; check the docs for current ones.

name: e2e
on: [push, pull_request]

jobs:
  playwright:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test
        env:
          E2E_USER: ${{ secrets.E2E_USER }}
          E2E_PASSWORD: ${{ secrets.E2E_PASSWORD }}
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 14

A GitHub Actions job for Cypress

The official cypress-io/github-action installs, caches and runs (check the docs for the current major version).

jobs:
  cypress:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: cypress-io/github-action@v6
        with:
          build: npm run build
          start: npm start
          wait-on: 'http://localhost:3000'
          browser: chrome
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: cypress-screenshots
          path: cypress/screenshots

Always upload the report

Use a condition such as if: ${{ !cancelled() }} or if: failure() so the report and traces are uploaded exactly when you need them most: after a failing run.

Quick check: Why enable retries only in CI?

  • CI cannot run a test only once
  • Retries make tests run faster locally
  • Local runs do not support assertions
  • Locally you want to see failures immediately; in CI retries reduce noise while flaky tests are reported
Answer

Locally you want to see failures immediately; in CI retries reduce noise while flaky tests are reported — Retries are a safety net, not a fix.