Add AI-assisted QA test workflow: scenario cards, runner, hooks, API capture
A complete loop for defining, generating, running and gating tests: - qa/: scenario-card template + examples (API/UI/binding), the Claude generation prompt, and a QA runbook tying it together - e2e/CLAUDE.md: conventions that keep AI-generated tests consistent - run-tests.ps1: one runner for every layer (backend/api/web/ui/binding), logs per run under .testlogs/, JUnit output, evidence report, and a re-seed guard against stale demo data - .githooks/ + scripts/install-hooks.ps1: pre-commit secret scan and pre-push unit tests (shift-left before CI) - evidence report now captures each API call's request, payload and response (Authorization redacted); JUnit reporter added for CI - scripts/db.mjs: db:check freshness guard Azure DevOps pipeline deferred; the runner already emits JUnit + the evidence report for any CI to publish. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
nilesh.p
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
df9e534e42
commit
45b5b34680
+135
@@ -0,0 +1,135 @@
|
||||
# TravelHub QA workflow — AI-assisted testing
|
||||
|
||||
This is how we define, automate, run and continuously gate tests on TravelHub.
|
||||
It is the same loop for API, UI, backend and binding scenarios:
|
||||
|
||||
```
|
||||
scenario card -> Claude generates the test -> run-tests.ps1 runs + logs + evidence -> gates CI/CD
|
||||
(you) (AI, you review) (one runner, anywhere)
|
||||
```
|
||||
|
||||
The scenario card is the single source of truth: a human reviews it, Claude
|
||||
generates the test from it, and the evidence report traces each result back to
|
||||
it. **No card, no test.**
|
||||
|
||||
---
|
||||
|
||||
## 1. Define a scenario
|
||||
|
||||
Copy [`scenario-template.yml`](scenario-template.yml) to
|
||||
`qa/scenarios/<ID>.yml` and fill it in. Pick the layer — it decides the tool and
|
||||
where the test lands:
|
||||
|
||||
| Layer | Proves | Lands in |
|
||||
| ----- | ------ | -------- |
|
||||
| `backend` | logic & data (tenant scoping, seat accounting) | `backend/**/*_test.go` |
|
||||
| `api` | HTTP contract: status, schema, errors | Go handler tests / Playwright request |
|
||||
| `ui` | a user journey in the browser | `e2e/tests/*.spec.ts` |
|
||||
| `binding` | the UI↔API / API↔DB seam (payload sent + response rendered) | `e2e/tests/*.spec.ts` (with API capture) |
|
||||
|
||||
Worked examples to copy from:
|
||||
[BOOK-012](scenarios/BOOK-012-no-oversell.yml) (binding),
|
||||
[API-005](scenarios/API-005-cross-tenant-booking.yml) (api),
|
||||
[SRCH-003](scenarios/SRCH-003-sort-by-fare.yml) (ui).
|
||||
|
||||
The card's `evidence:` list is a contract: the generated test must capture
|
||||
exactly those screenshots and API calls, and the report must show them.
|
||||
|
||||
## 2. Automate it with Claude Code
|
||||
|
||||
From the repo root, use the prompt in
|
||||
[`generate-test.prompt.md`](generate-test.prompt.md) with your card id. Claude
|
||||
reads the card and the conventions in [`e2e/CLAUDE.md`](../e2e/CLAUDE.md) and
|
||||
generates the spec using our page objects and fixtures.
|
||||
|
||||
Then **you review** — the diff *and* the evidence:
|
||||
|
||||
- the test uses page objects, not raw selectors;
|
||||
- every `then` has an assertion on observable state;
|
||||
- the evidence report shows each screenshot and API request/response the card
|
||||
listed.
|
||||
|
||||
The report is the proof of work: the AI *claims* it tested the scenario; the
|
||||
captured screenshots and API traffic *prove* it. Review the evidence, not just
|
||||
the code. Full reviewer checklist is in the prompt file.
|
||||
|
||||
> Tip: `npx playwright codegen http://localhost:4200` records a rough draft by
|
||||
> clicking; then ask Claude to refactor it into page-object style. You never
|
||||
> write selectors by hand.
|
||||
|
||||
## 3. Run it — one ps1, logged, CI-ready
|
||||
|
||||
[`run-tests.ps1`](../run-tests.ps1) runs any layer, logs everything, and emits
|
||||
machine-readable results. The **same script runs locally, in a git hook, and in
|
||||
a pipeline**.
|
||||
|
||||
```powershell
|
||||
.\run-tests.ps1 -Layer all # everything
|
||||
.\run-tests.ps1 -Layer unit # backend + web, no browser (fast)
|
||||
.\run-tests.ps1 -Layer ui -Grep "oversell" -Open # one scenario + open the report
|
||||
.\run-tests.ps1 -Layer binding # UI<->API seam with traffic capture
|
||||
```
|
||||
|
||||
What you get per run, under `.testlogs/<timestamp>/`:
|
||||
|
||||
- a log per layer (`backend.log`, `ui.log`, …) — full output, tee'd live;
|
||||
- `junit.xml` — the result format Azure DevOps / Gitea publish;
|
||||
- `summary.json` — pass/fail per layer;
|
||||
- the **evidence report** (`e2e/evidence-report/`) — screenshots + the request,
|
||||
payload and response of every API call, with credentials redacted.
|
||||
|
||||
It also **re-seeds the demo data if it has gone stale** (the seeded flights are
|
||||
dated relative to now and expire after ~3 weeks), so you never chase a failure
|
||||
that is really just old data.
|
||||
|
||||
Exit code is 0 only if every layer passed — that is what CI gates on.
|
||||
|
||||
## 4. Run continuously, before CI and CD
|
||||
|
||||
Five cadences, fastest to slowest — the same `run-tests.ps1` drives them all:
|
||||
|
||||
| When | What | Enforced? |
|
||||
| ---- | ---- | --------- |
|
||||
| while writing | watch mode (`ng test`, `playwright --ui`) | no |
|
||||
| **pre-commit** (git hook) | secret scan | local |
|
||||
| **pre-push** (git hook) | `run-tests.ps1 -Layer unit` | local |
|
||||
| **pre-merge** (CI) | full suite + evidence artifact | **blocks merge** |
|
||||
| nightly (scheduled) | full suite | alerts on drift |
|
||||
|
||||
Install the local hooks once after cloning:
|
||||
|
||||
```powershell
|
||||
.\scripts\install-hooks.ps1
|
||||
```
|
||||
|
||||
- **pre-commit** blocks committed secrets (it is what would have caught the dev
|
||||
DB password leak). Fast, no toolchain needed.
|
||||
- **pre-push** runs the unit layers so a logic break never leaves your machine.
|
||||
|
||||
"Before CI" is those two hooks (shift-left). "Before CD" is the pre-merge gate
|
||||
plus a smoke run against staging before promote. The **nightly** run matters:
|
||||
the stale-data failure above would surface at 2 a.m. instead of in front of a
|
||||
person.
|
||||
|
||||
### CI / CD wiring
|
||||
|
||||
`run-tests.ps1` is the contract; the pipeline just calls it and publishes its
|
||||
outputs (`junit.xml` + the evidence report). **Azure DevOps integration is
|
||||
planned** — when we add it, the pipeline will run this same script and use
|
||||
`PublishTestResults@2` on `junit.xml` plus `PublishBuildArtifacts` on the
|
||||
evidence report. Nothing in steps 1–3 changes.
|
||||
|
||||
---
|
||||
|
||||
## The 15-minute demo
|
||||
|
||||
Walk one card end to end:
|
||||
|
||||
1. Open [BOOK-012](scenarios/BOOK-012-no-oversell.yml) — the scenario.
|
||||
2. Prompt Claude Code with [the template](generate-test.prompt.md) → watch it
|
||||
generate the Playwright spec using the page objects.
|
||||
3. `.\run-tests.ps1 -Layer binding -Grep "oversell" -Open` → logs stream, the
|
||||
evidence report opens showing the 409 and both bookings' request/response.
|
||||
4. (Later) show the same ps1 running in the pipeline, blocking a bad merge.
|
||||
|
||||
That one loop is the whole system.
|
||||
Reference in New Issue
Block a user