End-to-end testing that does not slow your team down

By Sven Lindqvist · 20 July 202685 views
End-to-end testing that does not slow your team down
#testing#e2e#playwright#quality#devops

An engineering team adds Playwright E2E tests for every feature. After three months, the E2E suite has 180 tests. CI takes 35 minutes. Test flakiness is at 8% — on any given pipeline run, about 14 tests fail for reasons unrelated to code changes (timing issues, test data collisions, element not visible yet). The team starts marking flaky tests as skipped. The skipped test count reaches 40. The remaining 140 tests are trusted; the 40 are not.

The suite that was meant to increase confidence has decreased it: engineers do not know which failures represent real bugs and which are noise.

This is not a Playwright problem or an E2E testing problem. It is a scope problem. The team tried to cover every feature with E2E tests, which is appropriate for unit tests but counterproductive for E2E tests. The result is a slow, flaky suite that the team has learned to distrust.

The path out is not to add more infrastructure — retry logic, better selectors, faster machines. It is to have fewer, better-scoped tests.

What E2E tests are for

End-to-end tests exist to verify that critical user journeys work from the browser's perspective. They test what unit and integration tests cannot: the interaction between the frontend, the backend, and the database, as experienced by a real user.

The test pyramid applies: many unit tests, fewer integration tests, and a small number of E2E tests for the most critical flows. An E2E suite of 15-30 tests covering the most critical user journeys is more valuable than a suite of 180 tests that are slow and flaky.

Critical user journeys worth E2E testing:

  • User registration and email verification
  • Login (including failure cases and password reset)
  • The core value transaction (purchase, booking, submission — whatever the product's primary action is)
  • Critical settings changes (email change, password change)

What is not worth E2E testing (covered more reliably by other test types):

  • Form validation error messages (unit test)
  • API response format (integration test)
  • Individual component rendering (component test)
  • Business logic edge cases (unit test)

The discipline is recognizing that "this should be tested" does not mean "this should be E2E tested." Most things should be tested at the cheapest level that provides confidence — which is usually a unit or integration test. E2E tests are reserved for the flows where the interaction between layers is the thing being tested.

Writing reliable E2E tests

Test IDs for stable element selection

CSS classes, text content, and element structure change. data-testid attributes are stable because they are explicitly added for testing purposes:

// Component with test IDs
function CheckoutButton({ disabled, loading }: { disabled: boolean; loading: boolean }) {
  return (
    <button
      data-testid="checkout-button"
      disabled={disabled || loading}
      className={/* styling changes don't break tests */}
    >
      {loading ? 'Processing...' : 'Complete purchase'}
    </button>
  );
}

// Playwright test — uses test ID, not text or class
test('user can complete checkout', async ({ page }) => {
  await page.getByTestId('checkout-button').click();
  await expect(page.getByTestId('order-confirmation')).toBeVisible();
});

Avoid selecting by text content for interactive elements. Text changes during copy revisions; test IDs do not. The exception is asserting that specific text appears — toHaveText, toContainText — which is appropriate because the text content is the thing being verified.

Waiting for state, not time

Fixed waits (await page.waitForTimeout(2000)) are the primary source of flakiness. They are either too short (test fails on slow CI) or too long (test is unnecessarily slow).

// FLAKY: Fixed wait
await page.waitForTimeout(2000);
await page.click('#submit');

// STABLE: Wait for a specific state
await page.waitForLoadState('networkidle');  // Wait for network to be quiet
await page.click('[data-testid="submit-button"]');

// STABLE: Wait for an element to be visible
await page.getByTestId('success-message').waitFor({ state: 'visible' });

// STABLE: Wait for a URL change
await page.waitForURL('/order-confirmation/**');

// STABLE: Wait for a specific network request
const responsePromise = page.waitForResponse('/api/orders');
await page.click('[data-testid="submit-button"]');
const response = await responsePromise;
expect(response.status()).toBe(201);

Playwright's built-in auto-waiting handles most cases automatically — click, fill, and check all wait for the element to be visible and actionable before proceeding. Explicit waits are needed for assertions about application state after an action completes.

Test isolation: each test creates its own data

Tests that share data or depend on execution order are flaky when test order changes or tests run in parallel. Each test should create its own state:

// playwright.config.ts — setup for isolated tests
import { defineConfig } from '@playwright/test';

export default defineConfig({
  use: {
    baseURL: 'http://localhost:3000',
    storageState: undefined,  // No shared auth state between tests
  },
  workers: 4,  // Parallel test execution
});

// Each test creates its own user
test.beforeEach(async ({ request }) => {
  // Create a fresh test user via API — not via UI (faster)
  const response = await request.post('/api/test/create-user', {
    data: {
      email: `test-${Date.now()}@example.com`,
      password: 'testpassword123',
    }
  });
  const { email, password } = await response.json();

  // Store credentials for use in the test
  test.info().annotations.push({ type: 'email', description: email });
  test.info().annotations.push({ type: 'password', description: password });
});

Test data created in beforeEach should be cleaned up in afterEach to prevent accumulation in test databases:

let testUserId: string;

test.beforeEach(async ({ request }) => {
  const response = await request.post('/api/test/create-user', {
    data: { email: `test-${Date.now()}@example.com`, password: 'testpassword123' }
  });
  const user = await response.json();
  testUserId = user.id;
});

test.afterEach(async ({ request }) => {
  if (testUserId) {
    await request.delete(`/api/test/users/${testUserId}`);
  }
});

Authentication once per test suite

Re-authenticating in every test is slow. Playwright's storageState allows saving authentication state and reusing it:

// Setup file — runs once, saves auth state
// tests/auth.setup.ts
import { test as setup } from '@playwright/test';

const authFile = 'playwright/.auth/user.json';

setup('authenticate', async ({ page }) => {
  await page.goto('/login');
  await page.getByTestId('email-input').fill('[email protected]');
  await page.getByTestId('password-input').fill('testpassword123');
  await page.getByTestId('login-button').click();
  await page.waitForURL('/dashboard');

  // Save authentication state
  await page.context().storageState({ path: authFile });
});

// playwright.config.ts — use saved auth for most tests
export default defineConfig({
  projects: [
    {
      name: 'setup',
      testMatch: /.*\.setup\.ts/,
    },
    {
      name: 'authenticated',
      use: { storageState: 'playwright/.auth/user.json' },
      dependencies: ['setup'],
    },
    {
      name: 'unauthenticated',
      testMatch: /.*unauth\.spec\.ts/,
      // No storageState — tests run unauthenticated
    },
  ],
});

For tests that require different user roles, create separate auth state files:

// tests/admin.setup.ts
setup('authenticate as admin', async ({ page }) => {
  await page.goto('/login');
  await page.getByTestId('email-input').fill('[email protected]');
  await page.getByTestId('password-input').fill('adminpassword');
  await page.getByTestId('login-button').click();
  await page.waitForURL('/admin/dashboard');
  await page.context().storageState({ path: 'playwright/.auth/admin.json' });
});

Diagnosing and fixing flaky tests

When a test is flaky, the cause is almost always one of:

  1. A timing assumption (the element is not yet visible when the assertion runs)
  2. A test data collision (two parallel tests modified the same shared resource)
  3. An environment dependency (the test assumes a specific network condition or response time)

The diagnostic approach:

# Run a test 10 times to measure flakiness
npx playwright test checkout.spec.ts --repeat-each=10

# Run with trace to capture what the browser was doing when it failed
npx playwright test checkout.spec.ts --trace=on

# Open the trace viewer for a specific test run
npx playwright show-trace test-results/checkout-spec-ts-checkout/trace.zip

The trace viewer shows a timeline of network requests, browser events, and screenshots at each step. Identifying the exact moment of failure — and what the browser state was — reveals whether the failure is a timing issue, a missing element, or a network problem.

For persistent flakiness in a specific test, increasing the assertion timeout for that test is a stopgap while the root cause is investigated:

// Increase timeout for a known-slow operation
await expect(page.getByTestId('report-generated')).toBeVisible({
  timeout: 30000,  // Reports take up to 25 seconds to generate
});

The right fix is to understand why the operation is slow and either optimize it or change what is being asserted. Long timeouts in tests are a smell that the application has a slow path that should be addressed.

Running E2E tests selectively in CI

Running all E2E tests on every PR is what makes CI slow. Running them selectively keeps CI fast while maintaining coverage:

# .github/workflows/ci.yml
jobs:
  e2e:
    runs-on: ubuntu-latest
    # Run E2E tests only on:
    # - Main branch push
    # - PRs that touch critical paths
    if: |
      github.ref == 'refs/heads/main' ||
      contains(github.event.pull_request.labels.*.name, 'needs-e2e')
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test
        env:
          CI: true

PRs that touch the checkout flow or authentication can be labeled needs-e2e to trigger the full suite. Most PRs run only unit and integration tests, with E2E reserved for changes that are likely to affect critical flows.

For always-on E2E coverage without slowing every PR, run the full E2E suite on a schedule after merging to main:

# .github/workflows/e2e-nightly.yml
on:
  schedule:
    - cron: '0 2 * * *'  # 2am UTC daily
  push:
    branches: [main]

jobs:
  e2e-full:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test --project=authenticated --project=unauthenticated
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 14

Uploading the Playwright report on failure makes it easy to diagnose failures without access to the CI machine.

The outcome

The team that had 180 flaky E2E tests reduced to 28 tests covering their five critical user journeys. All 28 pass consistently. CI runs in 8 minutes. The confidence level is higher with 28 reliable tests than with 180 unreliable ones — not despite the smaller number, but because of it.

The reduction was not achieved by deleting tests arbitrarily. Each deleted test was replaced by a question: "what is this testing that a unit test cannot test?" When the answer was "not much," the E2E test was replaced with a unit or integration test. When the answer was "the interaction between the frontend and backend at this specific critical flow," the test was kept and made reliable.

The 28 tests that remained are the tests that justify the overhead of E2E testing: they verify the complete user journeys that would cause real business impact if they broke and that cannot be verified with faster tests. The 152 tests that were removed tested things that could be verified faster, cheaper, and more reliably at a lower level of the test pyramid.

The discipline of E2E testing is not writing tests — it is deciding what not to test at that level.

Visual regression testing alongside functional E2E tests

Functional E2E tests verify that interactions work. Visual regression tests verify that the UI looks correct. The two are complementary: a functional test can pass while the UI is visually broken (misaligned layout, wrong colors, missing elements) because the test only checks for element presence and behavior, not visual appearance.

Playwright supports visual comparisons through toHaveScreenshot:

// tests/visual.spec.ts
import { test, expect } from '@playwright/test';

test('checkout page matches visual baseline', async ({ page }) => {
  await page.goto('/checkout');
  await page.getByTestId('checkout-summary').waitFor({ state: 'visible' });
  
  // Take a screenshot and compare to the stored baseline
  await expect(page).toHaveScreenshot('checkout-page.png', {
    maxDiffPixels: 50,  // Allow up to 50 pixels of difference (for anti-aliasing)
    threshold: 0.1,     // 10% color difference threshold per pixel
  });
});

On the first run, Playwright stores the screenshot as a baseline. On subsequent runs, it compares and fails if the difference exceeds the threshold. Visual baselines are committed to the repository and updated explicitly when UI changes are intentional:

# Update all visual baselines after an intentional UI change
npx playwright test --update-snapshots

# Update a specific test's baseline
npx playwright test visual.spec.ts --update-snapshots

Visual regression tests are most valuable for design system components (buttons, cards, form fields) and critical pages (pricing, checkout, login) where visual correctness directly affects user trust. They are less valuable for pages that change frequently or that have dynamic content that varies between runs.

Handling dynamic content in E2E tests

E2E tests fail when they assert on dynamic content: dates, IDs, counter values, or content that changes between runs. The strategies:

Mask dynamic regions in visual tests:

test('dashboard matches visual baseline', async ({ page }) => {
  await page.goto('/dashboard');
  
  await expect(page).toHaveScreenshot('dashboard.png', {
    mask: [
      page.getByTestId('current-date'),    // Mask the date display
      page.getByTestId('session-counter'), // Mask the session counter
      page.getByTestId('user-avatar'),     // Mask the avatar (varies by user)
    ]
  });
});

Use regex for assertions on dynamic text:

// FRAGILE: Exact match fails when ID changes
await expect(page.getByTestId('order-id')).toHaveText('order_abc123');

// STABLE: Regex match works for any valid order ID format
await expect(page.getByTestId('order-id')).toHaveText(/^order_[a-z0-9]+$/);

Intercept network requests to control response data:

test('order confirmation page with controlled data', async ({ page }) => {
  // Intercept the API call and return controlled data
  await page.route('/api/orders/*', async (route) => {
    await route.fulfill({
      status: 200,
      contentType: 'application/json',
      body: JSON.stringify({
        id: 'order_test123',
        status: 'confirmed',
        total: 2999,
        createdAt: '2026-01-01T10:00:00Z',  // Fixed date for visual stability
      }),
    });
  });

  await page.goto('/orders/order_test123');
  await expect(page.getByTestId('order-confirmation')).toBeVisible();
  await expect(page).toHaveScreenshot('order-confirmation.png');
});

Intercepting network requests allows E2E tests to run against controlled, predictable data — making visual tests stable and functional tests deterministic — without requiring a separate test environment with pre-seeded data.

E2E tests in pull request workflows

The most effective integration of E2E tests into PR workflows is to run a subset automatically and make the full suite available on demand:

# Fast smoke E2E — runs on every PR, covers critical flows only
jobs:
  e2e-smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test --grep @smoke
        # Tests tagged @smoke: login, checkout, core feature

Tag critical tests with @smoke in the test title:

test('@smoke user can complete checkout', async ({ page }) => {
  // This test runs on every PR
});

test('user can change their subscription plan', async ({ page }) => {
  // This test runs on main branch push and when labeled 'needs-e2e'
});

The @smoke tests take 2-3 minutes and catch regressions in the most critical flows without adding significant time to the PR pipeline. The full suite runs after merge and nightly.

This approach balances coverage (smoke tests on every PR) against speed (full suite only when needed) in a way that teams actually maintain over time. A suite that is too slow to run on PRs gets disabled; a suite that is disabled provides no coverage.

Comments

No comments yet. Be the first!

Sign in to leave a comment.