161 lines
6.8 KiB
Markdown
161 lines
6.8 KiB
Markdown
# TravelHub QA workflow — AI-assisted testing
|
||
|
||
This is how we define, automate, run and continuously gate tests on TravelHub.
|
||
It is the same loop for API, UI, backend and binding scenarios:
|
||
|
||
```
|
||
scenario card -> Claude generates the test -> run-tests.ps1 runs + logs + evidence -> gates CI/CD
|
||
(you) (AI, you review) (one runner, anywhere)
|
||
```
|
||
|
||
The scenario card is the single source of truth: a human reviews it, Claude
|
||
generates the test from it, and the evidence report traces each result back to
|
||
it. **No card, no test.**
|
||
|
||
---
|
||
|
||
## 1. Define a scenario
|
||
|
||
Copy [`scenario-template.yml`](scenario-template.yml) to
|
||
`qa/scenarios/<ID>.yml` and fill it in. Pick the layer — it decides the tool and
|
||
where the test lands:
|
||
|
||
| Layer | Proves | Lands in |
|
||
| ----- | ------ | -------- |
|
||
| `backend` | logic & data (tenant scoping, seat accounting) | `backend/**/*_test.go` |
|
||
| `api` | HTTP contract: status, schema, errors | Go handler tests / Playwright request |
|
||
| `ui` | a user journey in the browser | `e2e/tests/*.spec.ts` |
|
||
| `binding` | the UI↔API / API↔DB seam (payload sent + response rendered) | `e2e/tests/*.spec.ts` (with API capture) |
|
||
|
||
Worked examples to copy from:
|
||
[BOOK-012](scenarios/BOOK-012-no-oversell.yml) (binding),
|
||
[API-005](scenarios/API-005-cross-tenant-booking.yml) (api),
|
||
[SRCH-003](scenarios/SRCH-003-sort-by-fare.yml) (ui).
|
||
|
||
The card's `evidence:` list is a contract: the generated test must capture
|
||
exactly those screenshots and API calls, and the report must show them.
|
||
|
||
## 2. Automate it with Claude Code
|
||
|
||
**The quick way — the `qa` tool.** It assembles a *grounded* generation prompt
|
||
(your card + the app model + the conventions) so Claude never guesses selectors,
|
||
routes, or endpoints:
|
||
|
||
```bash
|
||
npm run qa:index # build the app-model grounding index (do once / after UI changes)
|
||
node scripts/qa-tool.mjs list # list scenario cards
|
||
node scripts/qa-tool.mjs prompt BOOK-012 # print the grounded prompt — paste into Claude Code
|
||
node scripts/qa-tool.mjs run BOOK-012 # run the card's tests + open the evidence report
|
||
```
|
||
|
||
`npm run qa:index` reads the codebase and writes
|
||
[`app-model.md`](app-model.md) / `app-model.json`: the routes, page objects,
|
||
API endpoints, areas and accounts that actually exist. `prompt` embeds that plus
|
||
the card and [`e2e/CLAUDE.md`](../e2e/CLAUDE.md) into one ready-to-paste ask, so
|
||
the generated test is correct by construction. `run` wraps `run-tests.ps1` for
|
||
the card's layer and opens the evidence.
|
||
|
||
Generation stays human-in-the-loop: the tool prepares the grounded ask and runs
|
||
the result; you (with Claude Code) write and review the test.
|
||
|
||
**Prefer a screen?** `npm run qa:ui` opens a small local web UI (localhost:9500)
|
||
over the same commands: pick a card, copy its grounded prompt, hit **Run tests**,
|
||
and see the evidence report inline. Same engine, nicer front door.
|
||
|
||
**Or by hand:** use the template in
|
||
[`generate-test.prompt.md`](generate-test.prompt.md) with your card id. Claude
|
||
reads the card and the conventions in [`e2e/CLAUDE.md`](../e2e/CLAUDE.md) and
|
||
generates the spec using our page objects and fixtures.
|
||
|
||
Then **you review** — the diff *and* the evidence:
|
||
|
||
- the test uses page objects, not raw selectors;
|
||
- every `then` has an assertion on observable state;
|
||
- the evidence report shows each screenshot and API request/response the card
|
||
listed.
|
||
|
||
The report is the proof of work: the AI *claims* it tested the scenario; the
|
||
captured screenshots and API traffic *prove* it. Review the evidence, not just
|
||
the code. Full reviewer checklist is in the prompt file.
|
||
|
||
> Tip: `npx playwright codegen http://localhost:4200` records a rough draft by
|
||
> clicking; then ask Claude to refactor it into page-object style. You never
|
||
> write selectors by hand.
|
||
|
||
## 3. Run it — one ps1, logged, CI-ready
|
||
|
||
[`run-tests.ps1`](../run-tests.ps1) runs any layer, logs everything, and emits
|
||
machine-readable results. The **same script runs locally, in a git hook, and in
|
||
a pipeline**.
|
||
|
||
```powershell
|
||
.\run-tests.ps1 -Layer all # everything
|
||
.\run-tests.ps1 -Layer unit # backend + web, no browser (fast)
|
||
.\run-tests.ps1 -Layer ui -Grep "oversell" -Open # one scenario + open the report
|
||
.\run-tests.ps1 -Layer binding # UI<->API seam with traffic capture
|
||
```
|
||
|
||
What you get per run, under `.testlogs/<timestamp>/`:
|
||
|
||
- a log per layer (`backend.log`, `ui.log`, …) — full output, tee'd live;
|
||
- `junit.xml` — the result format Azure DevOps / Gitea publish;
|
||
- `summary.json` — pass/fail per layer;
|
||
- the **evidence report** (`e2e/evidence-report/`) — screenshots + the request,
|
||
payload and response of every API call, with credentials redacted.
|
||
|
||
It also **re-seeds the demo data if it has gone stale** (the seeded flights are
|
||
dated relative to now and expire after ~3 weeks), so you never chase a failure
|
||
that is really just old data.
|
||
|
||
Exit code is 0 only if every layer passed — that is what CI gates on.
|
||
|
||
## 4. Run continuously, before CI and CD
|
||
|
||
Five cadences, fastest to slowest — the same `run-tests.ps1` drives them all:
|
||
|
||
| When | What | Enforced? |
|
||
| ---- | ---- | --------- |
|
||
| while writing | watch mode (`ng test`, `playwright --ui`) | no |
|
||
| **pre-commit** (git hook) | secret scan | local |
|
||
| **pre-push** (git hook) | `run-tests.ps1 -Layer unit` | local |
|
||
| **pre-merge** (CI) | full suite + evidence artifact | **blocks merge** |
|
||
| nightly (scheduled) | full suite | alerts on drift |
|
||
|
||
Install the local hooks once after cloning:
|
||
|
||
```powershell
|
||
.\scripts\install-hooks.ps1
|
||
```
|
||
|
||
- **pre-commit** blocks committed secrets (it is what would have caught the dev
|
||
DB password leak). Fast, no toolchain needed.
|
||
- **pre-push** runs the unit layers so a logic break never leaves your machine.
|
||
|
||
"Before CI" is those two hooks (shift-left). "Before CD" is the pre-merge gate
|
||
plus a smoke run against staging before promote. The **nightly** run matters:
|
||
the stale-data failure above would surface at 2 a.m. instead of in front of a
|
||
person.
|
||
|
||
### CI / CD wiring
|
||
|
||
`run-tests.ps1` is the contract; the pipeline just calls it and publishes its
|
||
outputs (`junit.xml` + the evidence report). **Azure DevOps integration is
|
||
planned** — when we add it, the pipeline will run this same script and use
|
||
`PublishTestResults@2` on `junit.xml` plus `PublishBuildArtifacts` on the
|
||
evidence report. Nothing in steps 1–3 changes.
|
||
|
||
---
|
||
|
||
## The 15-minute demo
|
||
|
||
Walk one card end to end:
|
||
|
||
1. Open [BOOK-012](scenarios/BOOK-012-no-oversell.yml) — the scenario.
|
||
2. Prompt Claude Code with [the template](generate-test.prompt.md) → watch it
|
||
generate the Playwright spec using the page objects.
|
||
3. `.\run-tests.ps1 -Layer binding -Grep "oversell" -Open` → logs stream, the
|
||
evidence report opens showing the 409 and both bookings' request/response.
|
||
4. (Later) show the same ps1 running in the pipeline, blocking a bad merge.
|
||
|
||
That one loop is the whole system.
|