Add AI-assisted QA test workflow: scenario cards, runner, hooks, API capture
A complete loop for defining, generating, running and gating tests: - qa/: scenario-card template + examples (API/UI/binding), the Claude generation prompt, and a QA runbook tying it together - e2e/CLAUDE.md: conventions that keep AI-generated tests consistent - run-tests.ps1: one runner for every layer (backend/api/web/ui/binding), logs per run under .testlogs/, JUnit output, evidence report, and a re-seed guard against stale demo data - .githooks/ + scripts/install-hooks.ps1: pre-commit secret scan and pre-push unit tests (shift-left before CI) - evidence report now captures each API call's request, payload and response (Authorization redacted); JUnit reporter added for CI - scripts/db.mjs: db:check freshness guard Azure DevOps pipeline deferred; the runner already emits JUnit + the evidence report for any CI to publish. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
df9e534e42
commit
45b5b34680
@@ -0,0 +1,72 @@
|
||||
# Conventions for generating Playwright tests in this project
|
||||
|
||||
Read this before generating or editing any test under `e2e/`. These rules keep
|
||||
AI-generated tests consistent, reviewable, and trustworthy. A generated test
|
||||
that breaks them should be rejected in review.
|
||||
|
||||
## Where things go
|
||||
|
||||
- **Specs** live in `e2e/tests/<area>.spec.ts`, grouped by the areas defined in
|
||||
`e2e/reporters/report-model.ts` (auth, search, booking, tenant-isolation,
|
||||
dashboard, simulation, responsive). Add a scenario to the spec whose area it
|
||||
belongs to; do not create a new spec file per scenario.
|
||||
- **Page objects** live in `e2e/pages/*.page.ts`. One class per screen,
|
||||
exposing intent (`search.searchRoute('JFK','LHR')`), never raw selectors in
|
||||
the test.
|
||||
- **Fixtures** live in `e2e/fixtures/`. Import `test` and `expect` from
|
||||
`../fixtures/test`, never from `@playwright/test` directly — the project
|
||||
fixtures add evidence capture, the console-error guard, and API-traffic
|
||||
capture automatically.
|
||||
- **Accounts** come from `e2e/fixtures/accounts.ts`. Never hardcode an email or
|
||||
password in a spec.
|
||||
|
||||
## Hard rules
|
||||
|
||||
1. **No raw selectors in specs.** If a selector is missing, add a method or
|
||||
locator to the relevant page object and use that. Prefer role- and
|
||||
text-based locators; avoid brittle CSS/nth-child chains.
|
||||
2. **No hardcoded waits.** Never `waitForTimeout`. Wait on state: `await
|
||||
expect(locator).toBeVisible()`, `waitForURL`, the page object's `settled()`.
|
||||
3. **Never weaken an assertion to make a test pass.** If the app is wrong, the
|
||||
test should fail. Report the discrepancy; do not loosen the check.
|
||||
4. **Never hardcode or print secrets.** Credentials come from `accounts.ts`.
|
||||
The API-capture fixture already redacts the `Authorization` header — do not
|
||||
defeat it.
|
||||
5. **Capture evidence at each `then`.** Use `evidence.step('<what>', { ...data })`
|
||||
so the report shows a screenshot and the figures the test observed. Use
|
||||
`evidence.note(...)` for data compared across pages.
|
||||
6. **Leave the data as you found it.** A test that creates a booking must cancel
|
||||
it (or the suite's cleanup must). Use `uniqueSurname()` so rows are findable
|
||||
and removable.
|
||||
7. **Assert on behaviour the user can observe**, not on implementation details.
|
||||
`expect(seatsAfter).toBe(seatsBefore - 2)` survives a refactor; asserting an
|
||||
internal call does not.
|
||||
|
||||
## Shape of a generated test
|
||||
|
||||
```ts
|
||||
import { expect, test, uniqueSurname } from '../fixtures/test';
|
||||
import { SKYLINE_AGENT } from '../fixtures/accounts';
|
||||
|
||||
test.describe('Booking and cancelling', () => {
|
||||
test('the last seat cannot be oversold', async ({ signInAs, search, checkout, evidence, page }) => {
|
||||
// given
|
||||
await signInAs(SKYLINE_AGENT);
|
||||
// when ... (drive through page objects)
|
||||
// then ...
|
||||
await expect(/* observable state */).toBeTruthy();
|
||||
await evidence.step('Seat refused as oversold', { status: 409 });
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
## After generating
|
||||
|
||||
Run it and read the evidence report before committing:
|
||||
|
||||
```
|
||||
..\run-tests.ps1 -Layer ui -Grep "<part of the title>" -Open
|
||||
```
|
||||
|
||||
The report must show the screenshots and API request/response the scenario card
|
||||
listed under `evidence:`. If they are missing, the test is not done.
|
||||
Reference in New Issue
Block a user