Add AI-assisted QA test workflow: scenario cards, runner, hooks, API capture

A complete loop for defining, generating, running and gating tests:

- qa/: scenario-card template + examples (API/UI/binding), the Claude
  generation prompt, and a QA runbook tying it together
- e2e/CLAUDE.md: conventions that keep AI-generated tests consistent
- run-tests.ps1: one runner for every layer (backend/api/web/ui/binding),
  logs per run under .testlogs/, JUnit output, evidence report, and a
  re-seed guard against stale demo data
- .githooks/ + scripts/install-hooks.ps1: pre-commit secret scan and
  pre-push unit tests (shift-left before CI)
- evidence report now captures each API call's request, payload and
  response (Authorization redacted); JUnit reporter added for CI
- scripts/db.mjs: db:check freshness guard

Azure DevOps pipeline deferred; the runner already emits JUnit + the
evidence report for any CI to publish.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
nilesh.p
2026-10-10 13:04:46 +05:30
co-authored by Claude Opus 4.8
parent df9e534e42
commit 45b5b34680
20 changed files with 1054 additions and 11 deletions
+72
View File
@@ -0,0 +1,72 @@
# Conventions for generating Playwright tests in this project
Read this before generating or editing any test under `e2e/`. These rules keep
AI-generated tests consistent, reviewable, and trustworthy. A generated test
that breaks them should be rejected in review.
## Where things go
- **Specs** live in `e2e/tests/<area>.spec.ts`, grouped by the areas defined in
`e2e/reporters/report-model.ts` (auth, search, booking, tenant-isolation,
dashboard, simulation, responsive). Add a scenario to the spec whose area it
belongs to; do not create a new spec file per scenario.
- **Page objects** live in `e2e/pages/*.page.ts`. One class per screen,
exposing intent (`search.searchRoute('JFK','LHR')`), never raw selectors in
the test.
- **Fixtures** live in `e2e/fixtures/`. Import `test` and `expect` from
`../fixtures/test`, never from `@playwright/test` directly — the project
fixtures add evidence capture, the console-error guard, and API-traffic
capture automatically.
- **Accounts** come from `e2e/fixtures/accounts.ts`. Never hardcode an email or
password in a spec.
## Hard rules
1. **No raw selectors in specs.** If a selector is missing, add a method or
locator to the relevant page object and use that. Prefer role- and
text-based locators; avoid brittle CSS/nth-child chains.
2. **No hardcoded waits.** Never `waitForTimeout`. Wait on state: `await
expect(locator).toBeVisible()`, `waitForURL`, the page object's `settled()`.
3. **Never weaken an assertion to make a test pass.** If the app is wrong, the
test should fail. Report the discrepancy; do not loosen the check.
4. **Never hardcode or print secrets.** Credentials come from `accounts.ts`.
The API-capture fixture already redacts the `Authorization` header — do not
defeat it.
5. **Capture evidence at each `then`.** Use `evidence.step('<what>', { ...data })`
so the report shows a screenshot and the figures the test observed. Use
`evidence.note(...)` for data compared across pages.
6. **Leave the data as you found it.** A test that creates a booking must cancel
it (or the suite's cleanup must). Use `uniqueSurname()` so rows are findable
and removable.
7. **Assert on behaviour the user can observe**, not on implementation details.
`expect(seatsAfter).toBe(seatsBefore - 2)` survives a refactor; asserting an
internal call does not.
## Shape of a generated test
```ts
import { expect, test, uniqueSurname } from '../fixtures/test';
import { SKYLINE_AGENT } from '../fixtures/accounts';
test.describe('Booking and cancelling', () => {
test('the last seat cannot be oversold', async ({ signInAs, search, checkout, evidence, page }) => {
// given
await signInAs(SKYLINE_AGENT);
// when ... (drive through page objects)
// then ...
await expect(/* observable state */).toBeTruthy();
await evidence.step('Seat refused as oversold', { status: 409 });
});
});
```
## After generating
Run it and read the evidence report before committing:
```
..\run-tests.ps1 -Layer ui -Grep "<part of the title>" -Open
```
The report must show the screenshots and API request/response the scenario card
listed under `evidence:`. If they are missing, the test is not done.