Add AI-assisted QA test workflow: scenario cards, runner, hooks, API capture
A complete loop for defining, generating, running and gating tests: - qa/: scenario-card template + examples (API/UI/binding), the Claude generation prompt, and a QA runbook tying it together - e2e/CLAUDE.md: conventions that keep AI-generated tests consistent - run-tests.ps1: one runner for every layer (backend/api/web/ui/binding), logs per run under .testlogs/, JUnit output, evidence report, and a re-seed guard against stale demo data - .githooks/ + scripts/install-hooks.ps1: pre-commit secret scan and pre-push unit tests (shift-left before CI) - evidence report now captures each API call's request, payload and response (Authorization redacted); JUnit reporter added for CI - scripts/db.mjs: db:check freshness guard Azure DevOps pipeline deferred; the runner already emits JUnit + the evidence report for any CI to publish. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
nilesh.p
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
df9e534e42
commit
45b5b34680
+135
@@ -0,0 +1,135 @@
|
||||
# TravelHub QA workflow — AI-assisted testing
|
||||
|
||||
This is how we define, automate, run and continuously gate tests on TravelHub.
|
||||
It is the same loop for API, UI, backend and binding scenarios:
|
||||
|
||||
```
|
||||
scenario card -> Claude generates the test -> run-tests.ps1 runs + logs + evidence -> gates CI/CD
|
||||
(you) (AI, you review) (one runner, anywhere)
|
||||
```
|
||||
|
||||
The scenario card is the single source of truth: a human reviews it, Claude
|
||||
generates the test from it, and the evidence report traces each result back to
|
||||
it. **No card, no test.**
|
||||
|
||||
---
|
||||
|
||||
## 1. Define a scenario
|
||||
|
||||
Copy [`scenario-template.yml`](scenario-template.yml) to
|
||||
`qa/scenarios/<ID>.yml` and fill it in. Pick the layer — it decides the tool and
|
||||
where the test lands:
|
||||
|
||||
| Layer | Proves | Lands in |
|
||||
| ----- | ------ | -------- |
|
||||
| `backend` | logic & data (tenant scoping, seat accounting) | `backend/**/*_test.go` |
|
||||
| `api` | HTTP contract: status, schema, errors | Go handler tests / Playwright request |
|
||||
| `ui` | a user journey in the browser | `e2e/tests/*.spec.ts` |
|
||||
| `binding` | the UI↔API / API↔DB seam (payload sent + response rendered) | `e2e/tests/*.spec.ts` (with API capture) |
|
||||
|
||||
Worked examples to copy from:
|
||||
[BOOK-012](scenarios/BOOK-012-no-oversell.yml) (binding),
|
||||
[API-005](scenarios/API-005-cross-tenant-booking.yml) (api),
|
||||
[SRCH-003](scenarios/SRCH-003-sort-by-fare.yml) (ui).
|
||||
|
||||
The card's `evidence:` list is a contract: the generated test must capture
|
||||
exactly those screenshots and API calls, and the report must show them.
|
||||
|
||||
## 2. Automate it with Claude Code
|
||||
|
||||
From the repo root, use the prompt in
|
||||
[`generate-test.prompt.md`](generate-test.prompt.md) with your card id. Claude
|
||||
reads the card and the conventions in [`e2e/CLAUDE.md`](../e2e/CLAUDE.md) and
|
||||
generates the spec using our page objects and fixtures.
|
||||
|
||||
Then **you review** — the diff *and* the evidence:
|
||||
|
||||
- the test uses page objects, not raw selectors;
|
||||
- every `then` has an assertion on observable state;
|
||||
- the evidence report shows each screenshot and API request/response the card
|
||||
listed.
|
||||
|
||||
The report is the proof of work: the AI *claims* it tested the scenario; the
|
||||
captured screenshots and API traffic *prove* it. Review the evidence, not just
|
||||
the code. Full reviewer checklist is in the prompt file.
|
||||
|
||||
> Tip: `npx playwright codegen http://localhost:4200` records a rough draft by
|
||||
> clicking; then ask Claude to refactor it into page-object style. You never
|
||||
> write selectors by hand.
|
||||
|
||||
## 3. Run it — one ps1, logged, CI-ready
|
||||
|
||||
[`run-tests.ps1`](../run-tests.ps1) runs any layer, logs everything, and emits
|
||||
machine-readable results. The **same script runs locally, in a git hook, and in
|
||||
a pipeline**.
|
||||
|
||||
```powershell
|
||||
.\run-tests.ps1 -Layer all # everything
|
||||
.\run-tests.ps1 -Layer unit # backend + web, no browser (fast)
|
||||
.\run-tests.ps1 -Layer ui -Grep "oversell" -Open # one scenario + open the report
|
||||
.\run-tests.ps1 -Layer binding # UI<->API seam with traffic capture
|
||||
```
|
||||
|
||||
What you get per run, under `.testlogs/<timestamp>/`:
|
||||
|
||||
- a log per layer (`backend.log`, `ui.log`, …) — full output, tee'd live;
|
||||
- `junit.xml` — the result format Azure DevOps / Gitea publish;
|
||||
- `summary.json` — pass/fail per layer;
|
||||
- the **evidence report** (`e2e/evidence-report/`) — screenshots + the request,
|
||||
payload and response of every API call, with credentials redacted.
|
||||
|
||||
It also **re-seeds the demo data if it has gone stale** (the seeded flights are
|
||||
dated relative to now and expire after ~3 weeks), so you never chase a failure
|
||||
that is really just old data.
|
||||
|
||||
Exit code is 0 only if every layer passed — that is what CI gates on.
|
||||
|
||||
## 4. Run continuously, before CI and CD
|
||||
|
||||
Five cadences, fastest to slowest — the same `run-tests.ps1` drives them all:
|
||||
|
||||
| When | What | Enforced? |
|
||||
| ---- | ---- | --------- |
|
||||
| while writing | watch mode (`ng test`, `playwright --ui`) | no |
|
||||
| **pre-commit** (git hook) | secret scan | local |
|
||||
| **pre-push** (git hook) | `run-tests.ps1 -Layer unit` | local |
|
||||
| **pre-merge** (CI) | full suite + evidence artifact | **blocks merge** |
|
||||
| nightly (scheduled) | full suite | alerts on drift |
|
||||
|
||||
Install the local hooks once after cloning:
|
||||
|
||||
```powershell
|
||||
.\scripts\install-hooks.ps1
|
||||
```
|
||||
|
||||
- **pre-commit** blocks committed secrets (it is what would have caught the dev
|
||||
DB password leak). Fast, no toolchain needed.
|
||||
- **pre-push** runs the unit layers so a logic break never leaves your machine.
|
||||
|
||||
"Before CI" is those two hooks (shift-left). "Before CD" is the pre-merge gate
|
||||
plus a smoke run against staging before promote. The **nightly** run matters:
|
||||
the stale-data failure above would surface at 2 a.m. instead of in front of a
|
||||
person.
|
||||
|
||||
### CI / CD wiring
|
||||
|
||||
`run-tests.ps1` is the contract; the pipeline just calls it and publishes its
|
||||
outputs (`junit.xml` + the evidence report). **Azure DevOps integration is
|
||||
planned** — when we add it, the pipeline will run this same script and use
|
||||
`PublishTestResults@2` on `junit.xml` plus `PublishBuildArtifacts` on the
|
||||
evidence report. Nothing in steps 1–3 changes.
|
||||
|
||||
---
|
||||
|
||||
## The 15-minute demo
|
||||
|
||||
Walk one card end to end:
|
||||
|
||||
1. Open [BOOK-012](scenarios/BOOK-012-no-oversell.yml) — the scenario.
|
||||
2. Prompt Claude Code with [the template](generate-test.prompt.md) → watch it
|
||||
generate the Playwright spec using the page objects.
|
||||
3. `.\run-tests.ps1 -Layer binding -Grep "oversell" -Open` → logs stream, the
|
||||
evidence report opens showing the 409 and both bookings' request/response.
|
||||
4. (Later) show the same ps1 running in the pipeline, blocking a bad merge.
|
||||
|
||||
That one loop is the whole system.
|
||||
@@ -0,0 +1,45 @@
|
||||
# Prompt template — generate a Playwright test from a scenario card
|
||||
|
||||
Paste this into Claude Code (from the repo root), filling in the card id.
|
||||
Claude has the repo in context, so it will read the card and the conventions
|
||||
itself.
|
||||
|
||||
---
|
||||
|
||||
Generate the automated test for scenario card `qa/scenarios/<ID>.yml`.
|
||||
|
||||
Follow `e2e/CLAUDE.md` exactly. In particular:
|
||||
|
||||
- Put the test in the spec file for the card's `area`; add to the existing
|
||||
`test.describe` block, do not create a new spec.
|
||||
- Drive the UI only through the page objects in `e2e/pages/`. If a needed action
|
||||
or locator is missing, add a method to the right page object and use it.
|
||||
- Import `test`/`expect` from `../fixtures/test`; take accounts from
|
||||
`e2e/fixtures/accounts.ts`.
|
||||
- For every clause under the card's `then:` and every item under `evidence:`,
|
||||
add an `evidence.step(name, { ...observed data })` capturing a screenshot and
|
||||
the figures the card asks for.
|
||||
- No raw selectors in the spec, no `waitForTimeout`, no hardcoded secrets.
|
||||
- If the test creates data (a booking), clean it up at the end.
|
||||
|
||||
When done:
|
||||
|
||||
1. Show me the diff (the spec, and any page-object change).
|
||||
2. Run it: `.\run-tests.ps1 -Layer <layer> -Grep "<title fragment>" -Open`.
|
||||
3. Confirm the evidence report shows each screenshot and the API
|
||||
request/response listed in the card's `evidence:` section.
|
||||
|
||||
Do not loosen any assertion to make it pass. If the application behaves
|
||||
differently from the card, stop and tell me what differs — that is a finding,
|
||||
not something to code around.
|
||||
|
||||
---
|
||||
|
||||
## Reviewer checklist (the human reading the diff)
|
||||
|
||||
- [ ] Test lives in the right area spec; uses page objects, no raw selectors.
|
||||
- [ ] Every `then` has a matching assertion on observable state.
|
||||
- [ ] Evidence steps cover everything the card listed.
|
||||
- [ ] No `waitForTimeout`, no hardcoded credentials, no weakened assertions.
|
||||
- [ ] It actually ran green, and the evidence report proves the scenario.
|
||||
- [ ] Data created by the test is cleaned up.
|
||||
@@ -0,0 +1,49 @@
|
||||
# TravelHub test scenario card — TEMPLATE
|
||||
#
|
||||
# One card = one scenario. Author this BEFORE any test code exists: it is the
|
||||
# thing a human reviews, the thing Claude Code generates the test from, and the
|
||||
# thing the evidence report traces each result back to.
|
||||
#
|
||||
# Rule: no card, no test. Copy this file to qa/scenarios/<ID>.yml and fill it in.
|
||||
|
||||
id: AREA-000 # stable id, e.g. BOOK-012, SRCH-003, AUTH-001
|
||||
title: One sentence, present tense, describing the behaviour under test
|
||||
|
||||
# Which layer this scenario lives at. Picks the tool and where the test lands:
|
||||
# backend Go logic/data (repository, middleware, config) -> backend/**/*_test.go
|
||||
# api HTTP contract (status, schema, errors) -> Go handler tests / Playwright request
|
||||
# ui a user journey through the browser -> e2e/tests/*.spec.ts
|
||||
# binding the UI<->API or API<->DB seam (payload + render) -> e2e/tests/*.spec.ts (with API capture)
|
||||
layer: ui
|
||||
|
||||
area: Booking and cancelling # must match an area in e2e/reporters/report-model.ts
|
||||
risk: medium # low | medium | high -> drives how hard CI gates it
|
||||
owner: qa # qa | dev — who maintains this card
|
||||
|
||||
# The world the test assumes before it starts.
|
||||
preconditions:
|
||||
- signed in as agent@skyline.test / skyline
|
||||
- at least one bookable flight on JFK -> LHR
|
||||
|
||||
# Given / When / Then. Keep each a single clear clause.
|
||||
steps:
|
||||
given: the agent is on the flight search screen
|
||||
when: they search JFK -> LHR and select the cheapest flight
|
||||
then: the checkout screen shows that flight at the same fare
|
||||
|
||||
# What the evidence report must contain to prove this ran. The generated test
|
||||
# should capture exactly these with evidence.step(...).
|
||||
evidence:
|
||||
- screenshot of the checkout screen
|
||||
- the GET /api/flights request and response
|
||||
- the fare carried from search into checkout (recorded data)
|
||||
|
||||
# Inputs the test uses. Keep data here, not hardcoded in the spec.
|
||||
data:
|
||||
origin: JFK
|
||||
destination: LHR
|
||||
passengers: 1
|
||||
|
||||
# Optional: what must NOT happen (negative assertions).
|
||||
must_not:
|
||||
- no booking is created (this scenario stops at checkout)
|
||||
@@ -0,0 +1,27 @@
|
||||
id: API-005
|
||||
title: One agency cannot read another agency's booking by id
|
||||
layer: api
|
||||
area: Keeping agencies separate
|
||||
risk: high
|
||||
owner: qa
|
||||
|
||||
preconditions:
|
||||
- a confirmed booking exists under the Globetrotter tenant
|
||||
- valid credentials for the Skyline tenant
|
||||
|
||||
steps:
|
||||
given: Skyline holds a valid booking id that belongs to Globetrotter
|
||||
when: Skyline requests GET /api/bookings/{that-id}
|
||||
then: the API responds 404 Not Found (not 403), leaking nothing about the record
|
||||
|
||||
evidence:
|
||||
- the GET /api/bookings/{id} request (X-Tenant skyline) and the 404 response
|
||||
- the same id fetched successfully as Globetrotter, for contrast (recorded data)
|
||||
|
||||
data:
|
||||
owning_tenant: globetrotter
|
||||
requesting_tenant: skyline
|
||||
|
||||
must_not:
|
||||
- the response body must not contain the other tenant's passenger or contact data
|
||||
- the status must not be 200 or 403 (404 keeps existence hidden)
|
||||
@@ -0,0 +1,28 @@
|
||||
id: BOOK-012
|
||||
title: The last seat cannot be oversold to two agents at once
|
||||
layer: binding
|
||||
area: Booking and cancelling
|
||||
risk: high
|
||||
owner: qa
|
||||
|
||||
preconditions:
|
||||
- signed in as agent@skyline.test / skyline
|
||||
- a flight with exactly 1 seat remaining
|
||||
|
||||
steps:
|
||||
given: a flight showing "1 left"
|
||||
when: two bookings for that flight are confirmed at the same moment
|
||||
then: one succeeds (201 with a PNR), the other is refused (409), and seats reach 0
|
||||
|
||||
evidence:
|
||||
- the POST /api/bookings request and response for BOTH attempts
|
||||
- the seat count before and after (recorded data)
|
||||
- screenshot of the 409 "just sold" message on the losing attempt
|
||||
|
||||
data:
|
||||
route: JFK -> LHR
|
||||
passengers: 1
|
||||
|
||||
must_not:
|
||||
- the flight must never show negative seats
|
||||
- both bookings must never both succeed
|
||||
@@ -0,0 +1,25 @@
|
||||
id: SRCH-003
|
||||
title: Flight results sort by lowest fare
|
||||
layer: ui
|
||||
area: Searching for flights
|
||||
risk: low
|
||||
owner: qa
|
||||
|
||||
preconditions:
|
||||
- signed in as agent@skyline.test / skyline
|
||||
- more than one flight on JFK -> LHR
|
||||
|
||||
steps:
|
||||
given: the agent has searched JFK -> LHR
|
||||
when: they choose "Lowest fare" from the sort control
|
||||
then: the fares shown are in non-decreasing order down the page
|
||||
|
||||
evidence:
|
||||
- screenshot of the sorted results
|
||||
- the ordered list of fares read off the page (recorded data)
|
||||
- the GET /api/flights?sort=price request and response
|
||||
|
||||
data:
|
||||
origin: JFK
|
||||
destination: LHR
|
||||
sort: price
|
||||
Reference in New Issue
Block a user