Files

T

nilesh.pandClaude Opus 4.8 45b5b34680 Add AI-assisted QA test workflow: scenario cards, runner, hooks, API capture

A complete loop for defining, generating, running and gating tests:

- qa/: scenario-card template + examples (API/UI/binding), the Claude
  generation prompt, and a QA runbook tying it together
- e2e/CLAUDE.md: conventions that keep AI-generated tests consistent
- run-tests.ps1: one runner for every layer (backend/api/web/ui/binding),
  logs per run under .testlogs/, JUnit output, evidence report, and a
  re-seed guard against stale demo data
- .githooks/ + scripts/install-hooks.ps1: pre-commit secret scan and
  pre-push unit tests (shift-left before CI)
- evidence report now captures each API call's request, payload and
  response (Authorization redacted); JUnit reporter added for CI
- scripts/db.mjs: db:check freshness guard

Azure DevOps pipeline deferred; the runner already emits JUnit + the
evidence report for any CI to publish.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-10-10 13:04:46 +05:30

3.1 KiB

Raw Permalink Blame History

Conventions for generating Playwright tests in this project

Read this before generating or editing any test under e2e/. These rules keep AI-generated tests consistent, reviewable, and trustworthy. A generated test that breaks them should be rejected in review.

Where things go

Specs live in e2e/tests/<area>.spec.ts, grouped by the areas defined in e2e/reporters/report-model.ts (auth, search, booking, tenant-isolation, dashboard, simulation, responsive). Add a scenario to the spec whose area it belongs to; do not create a new spec file per scenario.
Page objects live in e2e/pages/*.page.ts. One class per screen, exposing intent (search.searchRoute('JFK','LHR')), never raw selectors in the test.
Fixtures live in e2e/fixtures/. Import test and expect from ../fixtures/test, never from @playwright/test directly — the project fixtures add evidence capture, the console-error guard, and API-traffic capture automatically.
Accounts come from e2e/fixtures/accounts.ts. Never hardcode an email or password in a spec.

Hard rules

No raw selectors in specs. If a selector is missing, add a method or locator to the relevant page object and use that. Prefer role- and text-based locators; avoid brittle CSS/nth-child chains.
No hardcoded waits. Never waitForTimeout. Wait on state: await expect(locator).toBeVisible(), waitForURL, the page object's settled().
Never weaken an assertion to make a test pass. If the app is wrong, the test should fail. Report the discrepancy; do not loosen the check.
Never hardcode or print secrets. Credentials come from accounts.ts. The API-capture fixture already redacts the Authorization header — do not defeat it.
Capture evidence at each then. Use evidence.step('<what>', { ...data }) so the report shows a screenshot and the figures the test observed. Use evidence.note(...) for data compared across pages.
Leave the data as you found it. A test that creates a booking must cancel it (or the suite's cleanup must). Use uniqueSurname() so rows are findable and removable.
Assert on behaviour the user can observe, not on implementation details. expect(seatsAfter).toBe(seatsBefore - 2) survives a refactor; asserting an internal call does not.

Shape of a generated test

import { expect, test, uniqueSurname } from '../fixtures/test';
import { SKYLINE_AGENT } from '../fixtures/accounts';

test.describe('Booking and cancelling', () => {
  test('the last seat cannot be oversold', async ({ signInAs, search, checkout, evidence, page }) => {
    // given
    await signInAs(SKYLINE_AGENT);
    // when ... (drive through page objects)
    // then ...
    await expect(/* observable state */).toBeTruthy();
    await evidence.step('Seat refused as oversold', { status: 409 });
  });
});

After generating

Run it and read the evidence report before committing:

..\run-tests.ps1 -Layer ui -Grep "<part of the title>" -Open

The report must show the screenshots and API request/response the scenario card listed under evidence:. If they are missing, the test is not done.

3.1 KiB Raw Permalink Blame History