Add AI-assisted QA test workflow: scenario cards, runner, hooks, API capture
A complete loop for defining, generating, running and gating tests: - qa/: scenario-card template + examples (API/UI/binding), the Claude generation prompt, and a QA runbook tying it together - e2e/CLAUDE.md: conventions that keep AI-generated tests consistent - run-tests.ps1: one runner for every layer (backend/api/web/ui/binding), logs per run under .testlogs/, JUnit output, evidence report, and a re-seed guard against stale demo data - .githooks/ + scripts/install-hooks.ps1: pre-commit secret scan and pre-push unit tests (shift-left before CI) - evidence report now captures each API call's request, payload and response (Authorization redacted); JUnit reporter added for CI - scripts/db.mjs: db:check freshness guard Azure DevOps pipeline deferred; the runner already emits JUnit + the evidence report for any CI to publish. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
nilesh.p
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
df9e534e42
commit
45b5b34680
@@ -0,0 +1,45 @@
|
||||
#!/bin/sh
|
||||
# Fast, local shift-left gate: block obvious secrets from being committed.
|
||||
# Runs in seconds, needs no toolchain. Enable with: scripts/install-hooks.ps1
|
||||
#
|
||||
# Bypass in a genuine emergency with: git commit --no-verify (discouraged).
|
||||
|
||||
staged=$(git diff --cached --name-only --diff-filter=ACM)
|
||||
[ -z "$staged" ] && exit 0
|
||||
|
||||
fail=0
|
||||
report() { echo " BLOCKED: $1"; fail=1; }
|
||||
|
||||
for f in $staged; do
|
||||
# Skip the hook scripts themselves: they legitimately contain the patterns
|
||||
# (and literals) they search for, so scanning them would self-block.
|
||||
case "$f" in
|
||||
.githooks/*) continue;;
|
||||
esac
|
||||
|
||||
# Never commit a real env file (only .env.example is allowed).
|
||||
case "$f" in
|
||||
*.env|.env) report "$f looks like a secrets file (.env is git-ignored for a reason)";;
|
||||
esac
|
||||
|
||||
# Scan staged content for credential patterns.
|
||||
[ -f "$f" ] || continue
|
||||
content=$(git show ":$f" 2>/dev/null) || continue
|
||||
|
||||
echo "$content" | grep -Eq 'GITEA_TOKEN[[:space:]]*=[[:space:]]*[A-Za-z0-9]' && \
|
||||
report "$f contains a GITEA_TOKEN value"
|
||||
echo "$content" | grep -Eq 'PGPASSWORD[[:space:]]*=[[:space:]]*[^ \"$]' && \
|
||||
report "$f hardcodes PGPASSWORD"
|
||||
echo "$content" | grep -q 'BEGIN .*PRIVATE KEY' && \
|
||||
report "$f contains a private key"
|
||||
# The known dev DB password must never re-enter the tree (see VAPT FINDING-8).
|
||||
echo "$content" | grep -q 'Did10n@c' && \
|
||||
report "$f contains the real dev DB password"
|
||||
done
|
||||
|
||||
if [ "$fail" -ne 0 ]; then
|
||||
echo ""
|
||||
echo "Commit blocked: remove the secret(s) above, or use .env (git-ignored)."
|
||||
exit 1
|
||||
fi
|
||||
exit 0
|
||||
@@ -0,0 +1,20 @@
|
||||
#!/bin/sh
|
||||
# Pre-push gate: run the fast test layers (Go + Angular unit) before code leaves
|
||||
# the machine. The browser e2e suite is left for CI to keep push quick.
|
||||
#
|
||||
# Bypass in a genuine emergency with: git push --no-verify (discouraged).
|
||||
|
||||
echo "pre-push: running unit layers (backend + web) ..."
|
||||
|
||||
# Delegate to the one runner so local and CI behave identically. The ps1
|
||||
# refreshes PATH itself, so this works even in a terminal opened before the
|
||||
# toolchain was installed.
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File ./run-tests.ps1 -Layer unit -NoReport
|
||||
code=$?
|
||||
|
||||
if [ "$code" -ne 0 ]; then
|
||||
echo ""
|
||||
echo "pre-push blocked: unit tests failed. Fix them, or push --no-verify to override."
|
||||
exit 1
|
||||
fi
|
||||
exit 0
|
||||
@@ -38,3 +38,5 @@ e2e/**/*.pdf
|
||||
|
||||
# --- app.ps1 runtime (pids, logs) -------------------------------------------
|
||||
.run/
|
||||
.testlogs/
|
||||
e2e/results/
|
||||
|
||||
+12
-2
@@ -290,8 +290,18 @@ invisibility, and the dashboard before and after a sale. Every other test still
|
||||
contributes its end-of-test screenshot automatically (`screenshot: 'on'`), so
|
||||
the report covers the whole run; the toolbar filters to the narrated ones.
|
||||
|
||||
A typical full run produces **96 screenshots across 82 scenarios**, about 14 MB.
|
||||
The directory is git-ignored and CI keeps it as an artifact for 30 days.
|
||||
**API traffic.** Every scenario also records the request/response of every
|
||||
`/api/**` call it made — method, path, request headers, payload, status, and
|
||||
response body — shown as a collapsible "API calls" block under the scenario.
|
||||
This is automatic (an auto fixture in `fixtures/http-capture.ts`); no test has
|
||||
to opt in. The **`Authorization` header is redacted** before anything is stored
|
||||
— it carries the Basic-auth credential, and the report is meant to be shared.
|
||||
Response bodies are capped (12 KB) so a large flight list cannot bloat the
|
||||
report.
|
||||
|
||||
A typical full run produces **96 screenshots and ~390 captured API calls across
|
||||
82 scenarios**, about 15 MB of HTML. The directory is git-ignored and CI keeps
|
||||
it as an artifact for 30 days.
|
||||
|
||||
### Simulated-condition tests
|
||||
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
# Conventions for generating Playwright tests in this project
|
||||
|
||||
Read this before generating or editing any test under `e2e/`. These rules keep
|
||||
AI-generated tests consistent, reviewable, and trustworthy. A generated test
|
||||
that breaks them should be rejected in review.
|
||||
|
||||
## Where things go
|
||||
|
||||
- **Specs** live in `e2e/tests/<area>.spec.ts`, grouped by the areas defined in
|
||||
`e2e/reporters/report-model.ts` (auth, search, booking, tenant-isolation,
|
||||
dashboard, simulation, responsive). Add a scenario to the spec whose area it
|
||||
belongs to; do not create a new spec file per scenario.
|
||||
- **Page objects** live in `e2e/pages/*.page.ts`. One class per screen,
|
||||
exposing intent (`search.searchRoute('JFK','LHR')`), never raw selectors in
|
||||
the test.
|
||||
- **Fixtures** live in `e2e/fixtures/`. Import `test` and `expect` from
|
||||
`../fixtures/test`, never from `@playwright/test` directly — the project
|
||||
fixtures add evidence capture, the console-error guard, and API-traffic
|
||||
capture automatically.
|
||||
- **Accounts** come from `e2e/fixtures/accounts.ts`. Never hardcode an email or
|
||||
password in a spec.
|
||||
|
||||
## Hard rules
|
||||
|
||||
1. **No raw selectors in specs.** If a selector is missing, add a method or
|
||||
locator to the relevant page object and use that. Prefer role- and
|
||||
text-based locators; avoid brittle CSS/nth-child chains.
|
||||
2. **No hardcoded waits.** Never `waitForTimeout`. Wait on state: `await
|
||||
expect(locator).toBeVisible()`, `waitForURL`, the page object's `settled()`.
|
||||
3. **Never weaken an assertion to make a test pass.** If the app is wrong, the
|
||||
test should fail. Report the discrepancy; do not loosen the check.
|
||||
4. **Never hardcode or print secrets.** Credentials come from `accounts.ts`.
|
||||
The API-capture fixture already redacts the `Authorization` header — do not
|
||||
defeat it.
|
||||
5. **Capture evidence at each `then`.** Use `evidence.step('<what>', { ...data })`
|
||||
so the report shows a screenshot and the figures the test observed. Use
|
||||
`evidence.note(...)` for data compared across pages.
|
||||
6. **Leave the data as you found it.** A test that creates a booking must cancel
|
||||
it (or the suite's cleanup must). Use `uniqueSurname()` so rows are findable
|
||||
and removable.
|
||||
7. **Assert on behaviour the user can observe**, not on implementation details.
|
||||
`expect(seatsAfter).toBe(seatsBefore - 2)` survives a refactor; asserting an
|
||||
internal call does not.
|
||||
|
||||
## Shape of a generated test
|
||||
|
||||
```ts
|
||||
import { expect, test, uniqueSurname } from '../fixtures/test';
|
||||
import { SKYLINE_AGENT } from '../fixtures/accounts';
|
||||
|
||||
test.describe('Booking and cancelling', () => {
|
||||
test('the last seat cannot be oversold', async ({ signInAs, search, checkout, evidence, page }) => {
|
||||
// given
|
||||
await signInAs(SKYLINE_AGENT);
|
||||
// when ... (drive through page objects)
|
||||
// then ...
|
||||
await expect(/* observable state */).toBeTruthy();
|
||||
await evidence.step('Seat refused as oversold', { status: 409 });
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
## After generating
|
||||
|
||||
Run it and read the evidence report before committing:
|
||||
|
||||
```
|
||||
..\run-tests.ps1 -Layer ui -Grep "<part of the title>" -Open
|
||||
```
|
||||
|
||||
The report must show the screenshots and API request/response the scenario card
|
||||
listed under `evidence:`. If they are missing, the test is not done.
|
||||
@@ -0,0 +1,110 @@
|
||||
/**
|
||||
* Captures the API traffic a test produced, for the evidence report.
|
||||
*
|
||||
* A screenshot shows what the user saw; this shows what crossed the wire -- the
|
||||
* request (method, path, payload) and the response (status, body) for every
|
||||
* `/api/**` call. Together they make the report a complete record of the
|
||||
* scenario.
|
||||
*
|
||||
* Security: the Authorization header carries the Basic-auth credential (a
|
||||
* base64 of email:password), so it is redacted before anything is stored. The
|
||||
* report is meant to be shared; a credential must never ride along in it.
|
||||
*/
|
||||
import type { Response } from '@playwright/test';
|
||||
|
||||
export interface ApiCall {
|
||||
method: string;
|
||||
path: string;
|
||||
status: number;
|
||||
statusText: string;
|
||||
durationMs: number;
|
||||
requestHeaders: Record<string, string>;
|
||||
requestBody?: string;
|
||||
responseBody?: string;
|
||||
truncated?: boolean;
|
||||
}
|
||||
|
||||
// Headers that must never appear in a shareable report.
|
||||
const SENSITIVE = new Set(['authorization', 'cookie', 'set-cookie', 'proxy-authorization']);
|
||||
|
||||
// Response bodies are capped so a large flight list cannot bloat the report.
|
||||
const MAX_BODY = 12000;
|
||||
|
||||
function redactHeaders(headers: Record<string, string>): Record<string, string> {
|
||||
const out: Record<string, string> = {};
|
||||
for (const [key, value] of Object.entries(headers)) {
|
||||
out[key] = SENSITIVE.has(key.toLowerCase()) ? '[redacted]' : value;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Pretty-prints JSON and caps length, reporting whether it was truncated. */
|
||||
function presentBody(raw: string, contentType: string): { body: string; truncated: boolean } {
|
||||
let text = raw;
|
||||
if (/json/i.test(contentType)) {
|
||||
try {
|
||||
text = JSON.stringify(JSON.parse(raw), null, 2);
|
||||
} catch {
|
||||
// Leave non-parseable bodies as-is.
|
||||
}
|
||||
}
|
||||
if (text.length > MAX_BODY) {
|
||||
return { body: text.slice(0, MAX_BODY) + '\n… (truncated)', truncated: true };
|
||||
}
|
||||
return { body: text, truncated: false };
|
||||
}
|
||||
|
||||
/**
|
||||
* Turns one response into an ApiCall record, or null if it is not an API call.
|
||||
* The body is read here, while the response is still live, and returned via the
|
||||
* promise the caller awaits at test teardown.
|
||||
*/
|
||||
export function captureApiCall(response: Response): Promise<ApiCall | null> {
|
||||
const request = response.request();
|
||||
|
||||
let url: URL;
|
||||
try {
|
||||
url = new URL(request.url());
|
||||
} catch {
|
||||
return Promise.resolve(null);
|
||||
}
|
||||
if (!url.pathname.startsWith('/api/')) {
|
||||
return Promise.resolve(null);
|
||||
}
|
||||
|
||||
return (async () => {
|
||||
try {
|
||||
const contentType = response.headers()['content-type'] ?? '';
|
||||
let responseBody: string | undefined;
|
||||
let truncated = false;
|
||||
|
||||
if (/json|text|xml|form/i.test(contentType)) {
|
||||
const raw = await response.text().catch(() => '');
|
||||
if (raw) {
|
||||
const shown = presentBody(raw, contentType);
|
||||
responseBody = shown.body;
|
||||
truncated = shown.truncated;
|
||||
}
|
||||
}
|
||||
|
||||
const timing = request.timing();
|
||||
const durationMs = timing.responseEnd > 0 ? Math.round(timing.responseEnd) : 0;
|
||||
|
||||
return {
|
||||
method: request.method(),
|
||||
path: url.pathname + url.search,
|
||||
status: response.status(),
|
||||
statusText: response.statusText(),
|
||||
durationMs,
|
||||
requestHeaders: redactHeaders(request.headers()),
|
||||
requestBody: request.postData() ?? undefined,
|
||||
responseBody,
|
||||
truncated: truncated || undefined,
|
||||
};
|
||||
} catch {
|
||||
// A response that cannot be read (e.g. the page navigated away) is
|
||||
// skipped rather than failing the test it was only observing.
|
||||
return null;
|
||||
}
|
||||
})();
|
||||
}
|
||||
@@ -2,6 +2,7 @@ import { expect, test as base } from '@playwright/test';
|
||||
|
||||
import { Account } from './accounts';
|
||||
import { Evidence } from './evidence';
|
||||
import { ApiCall, captureApiCall } from './http-capture';
|
||||
import { BookingDetailPage, BookingsPage } from '../pages/bookings.page';
|
||||
import { CheckoutPage } from '../pages/checkout.page';
|
||||
import { DashboardPage } from '../pages/dashboard.page';
|
||||
@@ -26,6 +27,9 @@ interface Fixtures {
|
||||
|
||||
/** Console and uncaught errors seen during the test; asserted empty at the end. */
|
||||
consoleErrors: string[];
|
||||
|
||||
/** Records every /api/** request and response for the run report. */
|
||||
apiTraffic: void;
|
||||
}
|
||||
|
||||
export const test = base.extend<Fixtures>({
|
||||
@@ -75,6 +79,28 @@ export const test = base.extend<Fixtures>({
|
||||
},
|
||||
{ auto: true },
|
||||
],
|
||||
|
||||
// Records the request/response of every API call the test made, attached as
|
||||
// evidence-http for the reporter. Auto, so no test has to opt in.
|
||||
apiTraffic: [
|
||||
async ({ page }, use, testInfo) => {
|
||||
const pending: Array<Promise<ApiCall | null>> = [];
|
||||
page.on('response', (response) => {
|
||||
pending.push(captureApiCall(response));
|
||||
});
|
||||
|
||||
await use();
|
||||
|
||||
const calls = (await Promise.all(pending)).filter((c): c is ApiCall => c !== null);
|
||||
if (calls.length > 0) {
|
||||
await testInfo.attach('evidence-http', {
|
||||
body: Buffer.from(JSON.stringify(calls, null, 2), 'utf8'),
|
||||
contentType: 'application/json',
|
||||
});
|
||||
}
|
||||
},
|
||||
{ auto: true },
|
||||
],
|
||||
});
|
||||
|
||||
export { expect } from '@playwright/test';
|
||||
|
||||
@@ -22,9 +22,22 @@ export default defineConfig({
|
||||
|
||||
// The evidence reporter turns the screenshots and data captured by passing
|
||||
// tests into a readable run report; the html one stays for debugging failures.
|
||||
// junit.xml is the machine-readable result any CI (Azure DevOps, Gitea) can
|
||||
// ingest; the evidence reporter is the human record; html is for debugging.
|
||||
reporter: process.env.CI
|
||||
? [['github'], ['html', { open: 'never' }], ['./reporters/evidence.ts'], ['list']]
|
||||
: [['html', { open: 'never' }], ['./reporters/evidence.ts'], ['list']],
|
||||
? [
|
||||
['github'],
|
||||
['junit', { outputFile: 'results/junit.xml' }],
|
||||
['html', { open: 'never' }],
|
||||
['./reporters/evidence.ts'],
|
||||
['list'],
|
||||
]
|
||||
: [
|
||||
['junit', { outputFile: 'results/junit.xml' }],
|
||||
['html', { open: 'never' }],
|
||||
['./reporters/evidence.ts'],
|
||||
['list'],
|
||||
],
|
||||
|
||||
use: {
|
||||
baseURL: WEB,
|
||||
|
||||
@@ -21,6 +21,7 @@ import type { FullConfig, FullResult, Reporter, Suite, TestCase, TestResult } fr
|
||||
import { DATA_PREFIX, SHOT_PREFIX } from '../fixtures/evidence';
|
||||
import { renderHtml } from './render-html';
|
||||
import {
|
||||
ApiCall,
|
||||
RunMeta,
|
||||
Scenario,
|
||||
Step,
|
||||
@@ -69,6 +70,7 @@ export default class EvidenceReporter implements Reporter {
|
||||
const suite = path.slice(2, -1).join(' › ');
|
||||
|
||||
const steps = this.collectSteps(test, result);
|
||||
const apiCalls = this.collectApiCalls(result);
|
||||
|
||||
this.scenarios.push({
|
||||
project,
|
||||
@@ -79,10 +81,28 @@ export default class EvidenceReporter implements Reporter {
|
||||
expected: result.status === test.expectedStatus,
|
||||
durationMs: result.duration,
|
||||
steps,
|
||||
apiCalls,
|
||||
error: result.error?.message?.split('\n').slice(0, 4).join('\n'),
|
||||
});
|
||||
}
|
||||
|
||||
/** Reads the evidence-http attachment the apiTraffic fixture produced. */
|
||||
private collectApiCalls(result: TestResult): ApiCall[] {
|
||||
const att = result.attachments.find((a) => a.name === 'evidence-http');
|
||||
if (!att) {
|
||||
return [];
|
||||
}
|
||||
const raw = att.body ? att.body.toString('utf8') : att.path ? safeRead(att.path) : '';
|
||||
if (!raw) {
|
||||
return [];
|
||||
}
|
||||
try {
|
||||
return JSON.parse(raw) as ApiCall[];
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
/** Pairs `evidence:NN:name` screenshots with `evidence-data:NN:name` blobs. */
|
||||
private collectSteps(test: TestCase, result: TestResult): Step[] {
|
||||
const bySeq = new Map<string, Step>();
|
||||
@@ -180,6 +200,7 @@ export default class EvidenceReporter implements Reporter {
|
||||
(n, s) => n + s.steps.filter((step) => step.shot).length,
|
||||
0,
|
||||
),
|
||||
apiCalls: this.scenarios.reduce((n, s) => n + s.apiCalls.length, 0),
|
||||
};
|
||||
|
||||
mkdirSync(OUT_DIR, { recursive: true });
|
||||
@@ -206,7 +227,7 @@ export default class EvidenceReporter implements Reporter {
|
||||
'Evidence report — what passed, with data and screenshots:',
|
||||
` ${url}`,
|
||||
` ${summary.passed}/${summary.total} passed · ${summary.screenshots} screenshots · ` +
|
||||
`${summary.narrated} scenarios with recorded data`,
|
||||
`${summary.apiCalls} API calls · ${summary.narrated} scenarios with recorded data`,
|
||||
'',
|
||||
' Open it: npm run report',
|
||||
' Markdown: ' + pathToFileURL(join(OUT_DIR, 'report.md')).href,
|
||||
@@ -254,7 +275,7 @@ function renderMarkdown(summary: Summary, scenarios: Scenario[]): string {
|
||||
lines.push(`### ${mark} ${heading}`, '');
|
||||
lines.push(`*${scenario.suite ? scenario.suite + ' · ' : ''}${scenario.project} · ${friendly(scenario.durationMs)}*`, '');
|
||||
|
||||
if (scenario.steps.length === 0) {
|
||||
if (scenario.steps.length === 0 && scenario.apiCalls.length === 0) {
|
||||
lines.push('_No evidence captured._', '');
|
||||
continue;
|
||||
}
|
||||
@@ -272,10 +293,32 @@ function renderMarkdown(summary: Summary, scenarios: Scenario[]): string {
|
||||
lines.push(``, '');
|
||||
}
|
||||
}
|
||||
|
||||
if (scenario.apiCalls.length > 0) {
|
||||
lines.push(`**API calls (${scenario.apiCalls.length})**`, '');
|
||||
for (const call of scenario.apiCalls) {
|
||||
lines.push(`- \`${call.method} ${call.path}\` → **${call.status}** (${call.durationMs} ms)`);
|
||||
if (call.requestBody) {
|
||||
lines.push('', ' Request:', '', ' ```json', indent(call.requestBody), ' ```');
|
||||
}
|
||||
if (call.responseBody) {
|
||||
lines.push('', ' Response:', '', ' ```json', indent(call.responseBody), ' ```');
|
||||
}
|
||||
}
|
||||
lines.push('');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
/** Indents a block two spaces so it nests under a Markdown list item. */
|
||||
function indent(text: string): string {
|
||||
return text
|
||||
.split('\n')
|
||||
.map((line) => ' ' + line)
|
||||
.join('\n');
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@
|
||||
* ("2.3.1 failed"), and a print stylesheet so it can be saved as a PDF.
|
||||
*/
|
||||
import {
|
||||
ApiCall,
|
||||
Area,
|
||||
RunMeta,
|
||||
Scenario,
|
||||
@@ -108,6 +109,7 @@ ${STYLES}
|
||||
<div><dt>Scenarios executed</dt><dd>${summary.total}</dd></div>
|
||||
<div><dt>Duration</dt><dd>${friendly(summary.durationMs)}</dd></div>
|
||||
<div><dt>Screenshots captured</dt><dd>${summary.screenshots}</dd></div>
|
||||
<div><dt>API calls captured</dt><dd>${summary.apiCalls}</dd></div>
|
||||
<div><dt>Scenarios with recorded data</dt><dd>${summary.narrated}</dd></div>
|
||||
</dl>
|
||||
</div>
|
||||
@@ -207,10 +209,11 @@ function renderArea(entry: AreaBlock): string {
|
||||
|
||||
function renderScenario(scenario: Scenario, ref: string): string {
|
||||
const hasFacts = scenario.steps.some((s) => s.facts);
|
||||
const steps =
|
||||
scenario.steps.length === 0
|
||||
? '<p class="no-evidence">No screenshot was captured for this scenario.</p>'
|
||||
: scenario.steps.map(renderStep).join('');
|
||||
const hasAnything = scenario.steps.length > 0 || scenario.apiCalls.length > 0;
|
||||
const steps = scenario.steps.map(renderStep).join('');
|
||||
const emptyNote = hasAnything
|
||||
? ''
|
||||
: '<p class="no-evidence">No evidence was captured for this scenario.</p>';
|
||||
|
||||
return `
|
||||
<details class="scenario" data-facts="${hasFacts ? '1' : '0'}"${hasFacts ? ' open' : ''}>
|
||||
@@ -222,6 +225,7 @@ function renderScenario(scenario: Scenario, ref: string): string {
|
||||
</span>
|
||||
<span class="right">
|
||||
<span class="hide-narrow">${escapeHtml(scenario.project)}</span>
|
||||
<span class="hide-narrow">${scenario.apiCalls.length} API</span>
|
||||
<span>${friendly(scenario.durationMs)}</span>
|
||||
<span class="tag ${statusClass(scenario.status)}">${statusWord(scenario.status)}</span>
|
||||
</span>
|
||||
@@ -229,10 +233,56 @@ function renderScenario(scenario: Scenario, ref: string): string {
|
||||
<div class="body">
|
||||
${scenario.error ? `<pre class="error">${escapeHtml(scenario.error)}</pre>` : ''}
|
||||
${steps}
|
||||
${emptyNote}
|
||||
${renderApiCalls(scenario.apiCalls)}
|
||||
</div>
|
||||
</details>`;
|
||||
}
|
||||
|
||||
/** The API traffic block: collapsed by default so it never dominates. */
|
||||
function renderApiCalls(calls: ApiCall[]): string {
|
||||
if (calls.length === 0) {
|
||||
return '';
|
||||
}
|
||||
|
||||
const rows = calls.map(renderApiCall).join('');
|
||||
|
||||
return `
|
||||
<details class="api">
|
||||
<summary>API calls · ${calls.length} request${calls.length === 1 ? '' : 's'}</summary>
|
||||
<div class="api-list">${rows}</div>
|
||||
</details>`;
|
||||
}
|
||||
|
||||
function renderApiCall(call: ApiCall): string {
|
||||
const cls = call.status >= 500 ? 'c5' : call.status >= 400 ? 'c4' : call.status >= 300 ? 'c3' : 'c2';
|
||||
|
||||
const headerRows = Object.entries(call.requestHeaders)
|
||||
.map(([k, v]) => `<tr><td>${escapeHtml(k)}</td><td>${escapeHtml(v)}</td></tr>`)
|
||||
.join('');
|
||||
|
||||
const request = call.requestBody
|
||||
? `<div class="part"><span class="part-label">Request payload</span><pre>${escapeHtml(call.requestBody)}</pre></div>`
|
||||
: '';
|
||||
|
||||
const response = call.responseBody
|
||||
? `<div class="part"><span class="part-label">Response body${call.truncated ? ' (truncated)' : ''}</span><pre>${escapeHtml(call.responseBody)}</pre></div>`
|
||||
: '';
|
||||
|
||||
return `
|
||||
<div class="call">
|
||||
<div class="call-head">
|
||||
<span class="method m-${escapeHtml(call.method.toLowerCase())}">${escapeHtml(call.method)}</span>
|
||||
<span class="call-path">${escapeHtml(call.path)}</span>
|
||||
<span class="status ${cls}">${call.status} ${escapeHtml(call.statusText)}</span>
|
||||
<span class="call-time">${call.durationMs} ms</span>
|
||||
</div>
|
||||
<details class="headers"><summary>Request headers</summary><table>${headerRows}</table></details>
|
||||
${request}
|
||||
${response}
|
||||
</div>`;
|
||||
}
|
||||
|
||||
function renderStep(step: Step): string {
|
||||
const facts = step.facts
|
||||
? `<table class="facts">${Object.entries(step.facts)
|
||||
@@ -372,6 +422,41 @@ const STYLES = `
|
||||
border-radius:6px;font-size:.83rem;white-space:pre-wrap;
|
||||
font-family:ui-monospace,monospace}
|
||||
|
||||
/* --- API traffic --- */
|
||||
details.api{border-top:1px solid var(--line);margin-top:6px}
|
||||
details.api > summary{list-style:none;cursor:pointer;padding:12px 0 10px;font-size:.78rem;
|
||||
font-weight:650;letter-spacing:.04em;text-transform:uppercase;color:var(--muted)}
|
||||
details.api > summary::-webkit-details-marker{display:none}
|
||||
details.api > summary::before{content:'\\25B8 ';color:var(--faint)}
|
||||
details.api[open] > summary::before{content:'\\25BE '}
|
||||
.api-list{display:flex;flex-direction:column;gap:10px;padding-bottom:6px}
|
||||
.call{border:1px solid var(--line);border-radius:7px;overflow:hidden;background:#fff}
|
||||
.call-head{display:flex;align-items:center;gap:10px;padding:8px 11px;background:#f8fafc;
|
||||
flex-wrap:wrap;font-size:.82rem}
|
||||
.method{font-weight:700;font-size:.7rem;letter-spacing:.04em;padding:2px 7px;border-radius:4px;
|
||||
background:#e5e7eb;color:#374151}
|
||||
.method.m-get{background:#e0edff;color:#0b5cd6}
|
||||
.method.m-post{background:#e6f4f1;color:#067a6e}
|
||||
.method.m-patch,.method.m-put{background:#fef3c7;color:#b45309}
|
||||
.method.m-delete{background:#fdeaee;color:#b3243f}
|
||||
.call-path{font-family:ui-monospace,Menlo,monospace;font-size:.8rem;color:var(--ink);
|
||||
word-break:break-all;flex:1;min-width:120px}
|
||||
.status{font-weight:700;font-size:.76rem;font-variant-numeric:tabular-nums}
|
||||
.status.c2{color:var(--ok)} .status.c3{color:var(--med)}
|
||||
.status.c4{color:var(--fail)} .status.c5{color:#fff;background:var(--fail);padding:1px 6px;border-radius:4px}
|
||||
.call-time{color:var(--faint);font-size:.76rem;font-variant-numeric:tabular-nums}
|
||||
.part{padding:8px 11px;border-top:1px solid var(--line)}
|
||||
.part-label{display:block;font-size:.68rem;letter-spacing:.06em;text-transform:uppercase;
|
||||
color:var(--faint);font-weight:650;margin-bottom:5px}
|
||||
.call pre{background:#0f172a;color:#e2e8f0;padding:10px 12px;border-radius:6px;overflow-x:auto;
|
||||
font:12px/1.5 ui-monospace,Menlo,monospace;margin:0;max-height:340px}
|
||||
details.headers{padding:8px 11px;border-top:1px solid var(--line)}
|
||||
details.headers > summary{cursor:pointer;font-size:.72rem;color:var(--muted)}
|
||||
details.headers table{border-collapse:collapse;font-size:.76rem;margin-top:6px}
|
||||
details.headers td{border:1px solid var(--line);padding:3px 8px;
|
||||
font-family:ui-monospace,Menlo,monospace;word-break:break-all}
|
||||
details.headers td:first-child{color:var(--muted);background:#f8fafc}
|
||||
|
||||
.toolbar{display:flex;flex-wrap:wrap;gap:8px;align-items:center;margin-bottom:20px}
|
||||
.toolbar button{border:1px solid var(--rule);background:#fff;border-radius:6px;
|
||||
padding:6px 13px;font:inherit;font-size:.85rem;cursor:pointer;color:var(--body)}
|
||||
@@ -468,7 +553,9 @@ const SCRIPT = `
|
||||
// first and put it back the way the reader had it afterwards.
|
||||
let restore = [];
|
||||
window.addEventListener('beforeprint', () => {
|
||||
restore = [...document.querySelectorAll('.scenario')].map((d) => [d, d.open]);
|
||||
// Open every details element (scenarios and the API/header blocks inside),
|
||||
// since a collapsed one prints only its summary line.
|
||||
restore = [...document.querySelectorAll('details')].map((d) => [d, d.open]);
|
||||
for (const [d] of restore) {
|
||||
d.open = true;
|
||||
}
|
||||
|
||||
@@ -15,6 +15,19 @@ export interface Step {
|
||||
facts?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
/** One API request/response pair observed during a scenario. */
|
||||
export interface ApiCall {
|
||||
method: string;
|
||||
path: string;
|
||||
status: number;
|
||||
statusText: string;
|
||||
durationMs: number;
|
||||
requestHeaders: Record<string, string>;
|
||||
requestBody?: string;
|
||||
responseBody?: string;
|
||||
truncated?: boolean;
|
||||
}
|
||||
|
||||
/** One test, as the report talks about it. */
|
||||
export interface Scenario {
|
||||
project: string;
|
||||
@@ -25,6 +38,7 @@ export interface Scenario {
|
||||
expected: boolean;
|
||||
durationMs: number;
|
||||
steps: Step[];
|
||||
apiCalls: ApiCall[];
|
||||
error?: string;
|
||||
}
|
||||
|
||||
@@ -45,6 +59,7 @@ export interface Summary {
|
||||
skipped: number;
|
||||
narrated: number;
|
||||
screenshots: number;
|
||||
apiCalls: number;
|
||||
}
|
||||
|
||||
// --- areas -----------------------------------------------------------------
|
||||
|
||||
+135
@@ -0,0 +1,135 @@
|
||||
# TravelHub QA workflow — AI-assisted testing
|
||||
|
||||
This is how we define, automate, run and continuously gate tests on TravelHub.
|
||||
It is the same loop for API, UI, backend and binding scenarios:
|
||||
|
||||
```
|
||||
scenario card -> Claude generates the test -> run-tests.ps1 runs + logs + evidence -> gates CI/CD
|
||||
(you) (AI, you review) (one runner, anywhere)
|
||||
```
|
||||
|
||||
The scenario card is the single source of truth: a human reviews it, Claude
|
||||
generates the test from it, and the evidence report traces each result back to
|
||||
it. **No card, no test.**
|
||||
|
||||
---
|
||||
|
||||
## 1. Define a scenario
|
||||
|
||||
Copy [`scenario-template.yml`](scenario-template.yml) to
|
||||
`qa/scenarios/<ID>.yml` and fill it in. Pick the layer — it decides the tool and
|
||||
where the test lands:
|
||||
|
||||
| Layer | Proves | Lands in |
|
||||
| ----- | ------ | -------- |
|
||||
| `backend` | logic & data (tenant scoping, seat accounting) | `backend/**/*_test.go` |
|
||||
| `api` | HTTP contract: status, schema, errors | Go handler tests / Playwright request |
|
||||
| `ui` | a user journey in the browser | `e2e/tests/*.spec.ts` |
|
||||
| `binding` | the UI↔API / API↔DB seam (payload sent + response rendered) | `e2e/tests/*.spec.ts` (with API capture) |
|
||||
|
||||
Worked examples to copy from:
|
||||
[BOOK-012](scenarios/BOOK-012-no-oversell.yml) (binding),
|
||||
[API-005](scenarios/API-005-cross-tenant-booking.yml) (api),
|
||||
[SRCH-003](scenarios/SRCH-003-sort-by-fare.yml) (ui).
|
||||
|
||||
The card's `evidence:` list is a contract: the generated test must capture
|
||||
exactly those screenshots and API calls, and the report must show them.
|
||||
|
||||
## 2. Automate it with Claude Code
|
||||
|
||||
From the repo root, use the prompt in
|
||||
[`generate-test.prompt.md`](generate-test.prompt.md) with your card id. Claude
|
||||
reads the card and the conventions in [`e2e/CLAUDE.md`](../e2e/CLAUDE.md) and
|
||||
generates the spec using our page objects and fixtures.
|
||||
|
||||
Then **you review** — the diff *and* the evidence:
|
||||
|
||||
- the test uses page objects, not raw selectors;
|
||||
- every `then` has an assertion on observable state;
|
||||
- the evidence report shows each screenshot and API request/response the card
|
||||
listed.
|
||||
|
||||
The report is the proof of work: the AI *claims* it tested the scenario; the
|
||||
captured screenshots and API traffic *prove* it. Review the evidence, not just
|
||||
the code. Full reviewer checklist is in the prompt file.
|
||||
|
||||
> Tip: `npx playwright codegen http://localhost:4200` records a rough draft by
|
||||
> clicking; then ask Claude to refactor it into page-object style. You never
|
||||
> write selectors by hand.
|
||||
|
||||
## 3. Run it — one ps1, logged, CI-ready
|
||||
|
||||
[`run-tests.ps1`](../run-tests.ps1) runs any layer, logs everything, and emits
|
||||
machine-readable results. The **same script runs locally, in a git hook, and in
|
||||
a pipeline**.
|
||||
|
||||
```powershell
|
||||
.\run-tests.ps1 -Layer all # everything
|
||||
.\run-tests.ps1 -Layer unit # backend + web, no browser (fast)
|
||||
.\run-tests.ps1 -Layer ui -Grep "oversell" -Open # one scenario + open the report
|
||||
.\run-tests.ps1 -Layer binding # UI<->API seam with traffic capture
|
||||
```
|
||||
|
||||
What you get per run, under `.testlogs/<timestamp>/`:
|
||||
|
||||
- a log per layer (`backend.log`, `ui.log`, …) — full output, tee'd live;
|
||||
- `junit.xml` — the result format Azure DevOps / Gitea publish;
|
||||
- `summary.json` — pass/fail per layer;
|
||||
- the **evidence report** (`e2e/evidence-report/`) — screenshots + the request,
|
||||
payload and response of every API call, with credentials redacted.
|
||||
|
||||
It also **re-seeds the demo data if it has gone stale** (the seeded flights are
|
||||
dated relative to now and expire after ~3 weeks), so you never chase a failure
|
||||
that is really just old data.
|
||||
|
||||
Exit code is 0 only if every layer passed — that is what CI gates on.
|
||||
|
||||
## 4. Run continuously, before CI and CD
|
||||
|
||||
Five cadences, fastest to slowest — the same `run-tests.ps1` drives them all:
|
||||
|
||||
| When | What | Enforced? |
|
||||
| ---- | ---- | --------- |
|
||||
| while writing | watch mode (`ng test`, `playwright --ui`) | no |
|
||||
| **pre-commit** (git hook) | secret scan | local |
|
||||
| **pre-push** (git hook) | `run-tests.ps1 -Layer unit` | local |
|
||||
| **pre-merge** (CI) | full suite + evidence artifact | **blocks merge** |
|
||||
| nightly (scheduled) | full suite | alerts on drift |
|
||||
|
||||
Install the local hooks once after cloning:
|
||||
|
||||
```powershell
|
||||
.\scripts\install-hooks.ps1
|
||||
```
|
||||
|
||||
- **pre-commit** blocks committed secrets (it is what would have caught the dev
|
||||
DB password leak). Fast, no toolchain needed.
|
||||
- **pre-push** runs the unit layers so a logic break never leaves your machine.
|
||||
|
||||
"Before CI" is those two hooks (shift-left). "Before CD" is the pre-merge gate
|
||||
plus a smoke run against staging before promote. The **nightly** run matters:
|
||||
the stale-data failure above would surface at 2 a.m. instead of in front of a
|
||||
person.
|
||||
|
||||
### CI / CD wiring
|
||||
|
||||
`run-tests.ps1` is the contract; the pipeline just calls it and publishes its
|
||||
outputs (`junit.xml` + the evidence report). **Azure DevOps integration is
|
||||
planned** — when we add it, the pipeline will run this same script and use
|
||||
`PublishTestResults@2` on `junit.xml` plus `PublishBuildArtifacts` on the
|
||||
evidence report. Nothing in steps 1–3 changes.
|
||||
|
||||
---
|
||||
|
||||
## The 15-minute demo
|
||||
|
||||
Walk one card end to end:
|
||||
|
||||
1. Open [BOOK-012](scenarios/BOOK-012-no-oversell.yml) — the scenario.
|
||||
2. Prompt Claude Code with [the template](generate-test.prompt.md) → watch it
|
||||
generate the Playwright spec using the page objects.
|
||||
3. `.\run-tests.ps1 -Layer binding -Grep "oversell" -Open` → logs stream, the
|
||||
evidence report opens showing the 409 and both bookings' request/response.
|
||||
4. (Later) show the same ps1 running in the pipeline, blocking a bad merge.
|
||||
|
||||
That one loop is the whole system.
|
||||
@@ -0,0 +1,45 @@
|
||||
# Prompt template — generate a Playwright test from a scenario card
|
||||
|
||||
Paste this into Claude Code (from the repo root), filling in the card id.
|
||||
Claude has the repo in context, so it will read the card and the conventions
|
||||
itself.
|
||||
|
||||
---
|
||||
|
||||
Generate the automated test for scenario card `qa/scenarios/<ID>.yml`.
|
||||
|
||||
Follow `e2e/CLAUDE.md` exactly. In particular:
|
||||
|
||||
- Put the test in the spec file for the card's `area`; add to the existing
|
||||
`test.describe` block, do not create a new spec.
|
||||
- Drive the UI only through the page objects in `e2e/pages/`. If a needed action
|
||||
or locator is missing, add a method to the right page object and use it.
|
||||
- Import `test`/`expect` from `../fixtures/test`; take accounts from
|
||||
`e2e/fixtures/accounts.ts`.
|
||||
- For every clause under the card's `then:` and every item under `evidence:`,
|
||||
add an `evidence.step(name, { ...observed data })` capturing a screenshot and
|
||||
the figures the card asks for.
|
||||
- No raw selectors in the spec, no `waitForTimeout`, no hardcoded secrets.
|
||||
- If the test creates data (a booking), clean it up at the end.
|
||||
|
||||
When done:
|
||||
|
||||
1. Show me the diff (the spec, and any page-object change).
|
||||
2. Run it: `.\run-tests.ps1 -Layer <layer> -Grep "<title fragment>" -Open`.
|
||||
3. Confirm the evidence report shows each screenshot and the API
|
||||
request/response listed in the card's `evidence:` section.
|
||||
|
||||
Do not loosen any assertion to make it pass. If the application behaves
|
||||
differently from the card, stop and tell me what differs — that is a finding,
|
||||
not something to code around.
|
||||
|
||||
---
|
||||
|
||||
## Reviewer checklist (the human reading the diff)
|
||||
|
||||
- [ ] Test lives in the right area spec; uses page objects, no raw selectors.
|
||||
- [ ] Every `then` has a matching assertion on observable state.
|
||||
- [ ] Evidence steps cover everything the card listed.
|
||||
- [ ] No `waitForTimeout`, no hardcoded credentials, no weakened assertions.
|
||||
- [ ] It actually ran green, and the evidence report proves the scenario.
|
||||
- [ ] Data created by the test is cleaned up.
|
||||
@@ -0,0 +1,49 @@
|
||||
# TravelHub test scenario card — TEMPLATE
|
||||
#
|
||||
# One card = one scenario. Author this BEFORE any test code exists: it is the
|
||||
# thing a human reviews, the thing Claude Code generates the test from, and the
|
||||
# thing the evidence report traces each result back to.
|
||||
#
|
||||
# Rule: no card, no test. Copy this file to qa/scenarios/<ID>.yml and fill it in.
|
||||
|
||||
id: AREA-000 # stable id, e.g. BOOK-012, SRCH-003, AUTH-001
|
||||
title: One sentence, present tense, describing the behaviour under test
|
||||
|
||||
# Which layer this scenario lives at. Picks the tool and where the test lands:
|
||||
# backend Go logic/data (repository, middleware, config) -> backend/**/*_test.go
|
||||
# api HTTP contract (status, schema, errors) -> Go handler tests / Playwright request
|
||||
# ui a user journey through the browser -> e2e/tests/*.spec.ts
|
||||
# binding the UI<->API or API<->DB seam (payload + render) -> e2e/tests/*.spec.ts (with API capture)
|
||||
layer: ui
|
||||
|
||||
area: Booking and cancelling # must match an area in e2e/reporters/report-model.ts
|
||||
risk: medium # low | medium | high -> drives how hard CI gates it
|
||||
owner: qa # qa | dev — who maintains this card
|
||||
|
||||
# The world the test assumes before it starts.
|
||||
preconditions:
|
||||
- signed in as agent@skyline.test / skyline
|
||||
- at least one bookable flight on JFK -> LHR
|
||||
|
||||
# Given / When / Then. Keep each a single clear clause.
|
||||
steps:
|
||||
given: the agent is on the flight search screen
|
||||
when: they search JFK -> LHR and select the cheapest flight
|
||||
then: the checkout screen shows that flight at the same fare
|
||||
|
||||
# What the evidence report must contain to prove this ran. The generated test
|
||||
# should capture exactly these with evidence.step(...).
|
||||
evidence:
|
||||
- screenshot of the checkout screen
|
||||
- the GET /api/flights request and response
|
||||
- the fare carried from search into checkout (recorded data)
|
||||
|
||||
# Inputs the test uses. Keep data here, not hardcoded in the spec.
|
||||
data:
|
||||
origin: JFK
|
||||
destination: LHR
|
||||
passengers: 1
|
||||
|
||||
# Optional: what must NOT happen (negative assertions).
|
||||
must_not:
|
||||
- no booking is created (this scenario stops at checkout)
|
||||
@@ -0,0 +1,27 @@
|
||||
id: API-005
|
||||
title: One agency cannot read another agency's booking by id
|
||||
layer: api
|
||||
area: Keeping agencies separate
|
||||
risk: high
|
||||
owner: qa
|
||||
|
||||
preconditions:
|
||||
- a confirmed booking exists under the Globetrotter tenant
|
||||
- valid credentials for the Skyline tenant
|
||||
|
||||
steps:
|
||||
given: Skyline holds a valid booking id that belongs to Globetrotter
|
||||
when: Skyline requests GET /api/bookings/{that-id}
|
||||
then: the API responds 404 Not Found (not 403), leaking nothing about the record
|
||||
|
||||
evidence:
|
||||
- the GET /api/bookings/{id} request (X-Tenant skyline) and the 404 response
|
||||
- the same id fetched successfully as Globetrotter, for contrast (recorded data)
|
||||
|
||||
data:
|
||||
owning_tenant: globetrotter
|
||||
requesting_tenant: skyline
|
||||
|
||||
must_not:
|
||||
- the response body must not contain the other tenant's passenger or contact data
|
||||
- the status must not be 200 or 403 (404 keeps existence hidden)
|
||||
@@ -0,0 +1,28 @@
|
||||
id: BOOK-012
|
||||
title: The last seat cannot be oversold to two agents at once
|
||||
layer: binding
|
||||
area: Booking and cancelling
|
||||
risk: high
|
||||
owner: qa
|
||||
|
||||
preconditions:
|
||||
- signed in as agent@skyline.test / skyline
|
||||
- a flight with exactly 1 seat remaining
|
||||
|
||||
steps:
|
||||
given: a flight showing "1 left"
|
||||
when: two bookings for that flight are confirmed at the same moment
|
||||
then: one succeeds (201 with a PNR), the other is refused (409), and seats reach 0
|
||||
|
||||
evidence:
|
||||
- the POST /api/bookings request and response for BOTH attempts
|
||||
- the seat count before and after (recorded data)
|
||||
- screenshot of the 409 "just sold" message on the losing attempt
|
||||
|
||||
data:
|
||||
route: JFK -> LHR
|
||||
passengers: 1
|
||||
|
||||
must_not:
|
||||
- the flight must never show negative seats
|
||||
- both bookings must never both succeed
|
||||
@@ -0,0 +1,25 @@
|
||||
id: SRCH-003
|
||||
title: Flight results sort by lowest fare
|
||||
layer: ui
|
||||
area: Searching for flights
|
||||
risk: low
|
||||
owner: qa
|
||||
|
||||
preconditions:
|
||||
- signed in as agent@skyline.test / skyline
|
||||
- more than one flight on JFK -> LHR
|
||||
|
||||
steps:
|
||||
given: the agent has searched JFK -> LHR
|
||||
when: they choose "Lowest fare" from the sort control
|
||||
then: the fares shown are in non-decreasing order down the page
|
||||
|
||||
evidence:
|
||||
- screenshot of the sorted results
|
||||
- the ordered list of fares read off the page (recorded data)
|
||||
- the GET /api/flights?sort=price request and response
|
||||
|
||||
data:
|
||||
origin: JFK
|
||||
destination: LHR
|
||||
sort: price
|
||||
+252
@@ -0,0 +1,252 @@
|
||||
#requires -Version 5.1
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Run the TravelHub test suite by layer, log everything, and emit CI-ready
|
||||
results and an evidence report. The same script runs locally, in a git
|
||||
hook, and in a pipeline.
|
||||
|
||||
.DESCRIPTION
|
||||
One runner for every test layer so there is a single source of truth for
|
||||
"how tests run here". It:
|
||||
- refreshes PATH from the registry (works in stale terminals)
|
||||
- guards e2e against stale demo data by re-seeding when inventory is old
|
||||
- writes a timestamped log per step under .testlogs\<run-id>\
|
||||
- emits JUnit XML (e2e) that Azure DevOps / Gitea can publish
|
||||
- regenerates the evidence report (screenshots + API request/response)
|
||||
- returns a non-zero exit code if any layer fails, so CI can gate on it
|
||||
|
||||
Layers map to the scenario taxonomy in qa\:
|
||||
backend Go unit + integration tests (repository, middleware, config)
|
||||
api Go HTTP-contract tests (handlers, middleware)
|
||||
web Angular unit tests
|
||||
ui Playwright browser journeys (UI + binding, with API capture)
|
||||
binding alias for ui -- the UI<->API seam, proven by captured traffic
|
||||
unit backend + web (no browser; fast)
|
||||
all backend + web + ui
|
||||
|
||||
.PARAMETER Layer
|
||||
Which layer to run. Default: all.
|
||||
|
||||
.PARAMETER Grep
|
||||
Only run tests whose title matches this (passed to `go test -run` and
|
||||
`playwright -g`).
|
||||
|
||||
.PARAMETER Reseed
|
||||
Force a database re-seed before running, instead of only when stale.
|
||||
|
||||
.PARAMETER NoReport
|
||||
Skip regenerating the evidence report (faster; ui/binding only).
|
||||
|
||||
.PARAMETER Open
|
||||
Open the evidence report in the browser when the run finishes.
|
||||
|
||||
.EXAMPLE
|
||||
.\run-tests.ps1 -Layer all
|
||||
.EXAMPLE
|
||||
.\run-tests.ps1 -Layer ui -Grep "oversell" -Open
|
||||
.EXAMPLE
|
||||
.\run-tests.ps1 -Layer unit # fast pre-commit gate, no browser
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[ValidateSet('all', 'unit', 'backend', 'api', 'web', 'ui', 'binding')]
|
||||
[string]$Layer = 'all',
|
||||
|
||||
[string]$Grep,
|
||||
[switch]$Reseed,
|
||||
[switch]$NoReport,
|
||||
[switch]$Open
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
# --- configuration ---------------------------------------------------------
|
||||
|
||||
$Root = $PSScriptRoot
|
||||
$Backend = Join-Path $Root 'backend'
|
||||
$Frontend = Join-Path $Root 'frontend'
|
||||
$E2E = Join-Path $Root 'e2e'
|
||||
|
||||
$RunId = Get-Date -Format 'yyyyMMdd-HHmmss'
|
||||
$LogDir = Join-Path $Root ".testlogs\$RunId"
|
||||
|
||||
# --- console ---------------------------------------------------------------
|
||||
|
||||
function Write-Head($msg) { Write-Host "`n==== $msg ====" -ForegroundColor Cyan }
|
||||
function Write-Ok($msg) { Write-Host " PASS $msg" -ForegroundColor Green }
|
||||
function Write-Fail($msg) { Write-Host " FAIL $msg" -ForegroundColor Red }
|
||||
function Write-Info($msg) { Write-Host " $msg" -ForegroundColor DarkGray }
|
||||
|
||||
# --- environment -----------------------------------------------------------
|
||||
|
||||
function Sync-Path {
|
||||
$m = [Environment]::GetEnvironmentVariable('Path', 'Machine')
|
||||
$u = [Environment]::GetEnvironmentVariable('Path', 'User')
|
||||
$env:Path = ($m, $u | Where-Object { $_ }) -join ';'
|
||||
}
|
||||
|
||||
function Resolve-Tool($name, [string[]]$fallbacks) {
|
||||
$cmd = Get-Command $name -ErrorAction SilentlyContinue
|
||||
if ($cmd) { return $cmd.Source }
|
||||
foreach ($f in $fallbacks) { if (Test-Path $f) { return $f } }
|
||||
throw "$name not found on PATH."
|
||||
}
|
||||
function Get-Go { Resolve-Tool 'go.exe' @('C:\Program Files\Go\bin\go.exe') }
|
||||
function Get-Npm { Resolve-Tool 'npm.cmd' @('C:\Program Files\nodejs\npm.cmd') }
|
||||
function Get-Npx { Resolve-Tool 'npx.cmd' @('C:\Program Files\nodejs\npx.cmd') }
|
||||
function Get-Node { Resolve-Tool 'node.exe' @('C:\Program Files\nodejs\node.exe') }
|
||||
|
||||
# --- step runner -----------------------------------------------------------
|
||||
|
||||
$script:Results = @()
|
||||
|
||||
# Run one command, tee its output to a log file, and record pass/fail by exit
|
||||
# code. Never throws -- a failing step is recorded and the run continues so the
|
||||
# report covers every layer.
|
||||
function Invoke-Step {
|
||||
param(
|
||||
[string]$Name,
|
||||
[string]$WorkDir,
|
||||
[string]$Exe,
|
||||
[string[]]$CmdArgs,
|
||||
[string]$LogName
|
||||
)
|
||||
Write-Head $Name
|
||||
$log = Join-Path $LogDir $LogName
|
||||
Write-Info "log: $log"
|
||||
|
||||
$sw = [Diagnostics.Stopwatch]::StartNew()
|
||||
Push-Location $WorkDir
|
||||
# A native tool writing to stderr (e.g. a webserver startup warning) must
|
||||
# not be treated as a PowerShell error. Pass/fail is the exit code alone;
|
||||
# stderr is just logged. Continue preference keeps 2>&1 from throwing.
|
||||
$prev = $ErrorActionPreference
|
||||
$ErrorActionPreference = 'Continue'
|
||||
try {
|
||||
& $Exe @CmdArgs 2>&1 | Tee-Object -FilePath $log
|
||||
$code = $LASTEXITCODE
|
||||
} finally {
|
||||
$ErrorActionPreference = $prev
|
||||
Pop-Location
|
||||
}
|
||||
$sw.Stop()
|
||||
|
||||
$ok = ($code -eq 0)
|
||||
$script:Results += [pscustomobject]@{
|
||||
Layer = $Name; Exit = $code; Ok = $ok; Seconds = [math]::Round($sw.Elapsed.TotalSeconds, 1); Log = $log
|
||||
}
|
||||
if ($ok) { Write-Ok "$Name ($($sw.Elapsed.TotalSeconds.ToString('0.0'))s)" }
|
||||
else { Write-Fail "$Name (exit $code)" }
|
||||
}
|
||||
|
||||
# --- preconditions ---------------------------------------------------------
|
||||
|
||||
function Ensure-Inventory {
|
||||
# Only the browser layers need live, future-dated flight data.
|
||||
$node = Get-Node
|
||||
$check = Join-Path $Root 'scripts\db.mjs'
|
||||
|
||||
if ($Reseed) {
|
||||
Write-Info 'Re-seeding database (-Reseed).'
|
||||
& $node (Join-Path $Root 'scripts\db.mjs') reset | Out-Null
|
||||
return
|
||||
}
|
||||
|
||||
& $node $check check 2>&1 | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
Write-Info 'Demo inventory is stale or missing -- re-seeding before e2e.'
|
||||
& $node (Join-Path $Root 'scripts\db.mjs') reset | Out-Null
|
||||
} else {
|
||||
Write-Info 'Demo inventory is fresh.'
|
||||
}
|
||||
}
|
||||
|
||||
# --- layers ----------------------------------------------------------------
|
||||
|
||||
function Run-Backend {
|
||||
$go = Get-Go
|
||||
$cmdArgs = @('test', './...')
|
||||
if ($Grep) { $cmdArgs += @('-run', $Grep) }
|
||||
Invoke-Step 'backend' $Backend $go $cmdArgs 'backend.log'
|
||||
}
|
||||
|
||||
function Run-Api {
|
||||
# The HTTP contract lives in the handler + middleware packages.
|
||||
$go = Get-Go
|
||||
$cmdArgs = @('test', './internal/handlers/...', './internal/middleware/...')
|
||||
if ($Grep) { $cmdArgs += @('-run', $Grep) }
|
||||
Invoke-Step 'api' $Backend $go $cmdArgs 'api.log'
|
||||
}
|
||||
|
||||
function Run-Web {
|
||||
$npx = Get-Npx
|
||||
Invoke-Step 'web' $Frontend $npx @('ng', 'test', '--no-watch') 'web.log'
|
||||
}
|
||||
|
||||
function Run-Ui {
|
||||
param([string]$Label = 'ui')
|
||||
Ensure-Inventory
|
||||
$npx = Get-Npx
|
||||
$cmdArgs = @('playwright', 'test')
|
||||
if ($Grep) { $cmdArgs += @('-g', $Grep) }
|
||||
Invoke-Step $Label $E2E $npx $cmdArgs "$Label.log"
|
||||
}
|
||||
|
||||
# --- evidence + results collection -----------------------------------------
|
||||
|
||||
function Collect-Artifacts {
|
||||
# JUnit (for CI test publishing) and the evidence report both live under e2e.
|
||||
$junit = Join-Path $E2E 'results\junit.xml'
|
||||
if (Test-Path $junit) {
|
||||
Copy-Item $junit (Join-Path $LogDir 'junit.xml') -Force
|
||||
Write-Info "JUnit results -> $(Join-Path $LogDir 'junit.xml')"
|
||||
}
|
||||
$evidence = Join-Path $E2E 'evidence-report\index.html'
|
||||
if (Test-Path $evidence) {
|
||||
Write-Info "Evidence report -> $evidence"
|
||||
if ($Open) { Start-Process $evidence }
|
||||
}
|
||||
}
|
||||
|
||||
# --- dispatch --------------------------------------------------------------
|
||||
|
||||
Sync-Path
|
||||
New-Item -ItemType Directory -Force -Path $LogDir | Out-Null
|
||||
Write-Host "TravelHub test run $RunId (layer: $Layer)" -ForegroundColor White
|
||||
|
||||
switch ($Layer) {
|
||||
'backend' { Run-Backend }
|
||||
'api' { Run-Api }
|
||||
'web' { Run-Web }
|
||||
'ui' { Run-Ui 'ui' }
|
||||
'binding' { Run-Ui 'binding' }
|
||||
'unit' { Run-Backend; Run-Web }
|
||||
'all' { Run-Backend; Run-Web; Run-Ui 'ui' }
|
||||
}
|
||||
|
||||
if (-not $NoReport) { Collect-Artifacts }
|
||||
|
||||
# --- summary ---------------------------------------------------------------
|
||||
|
||||
Write-Head 'Summary'
|
||||
$script:Results | Format-Table Layer, @{N='Result';E={ if ($_.Ok) { 'PASS' } else { "FAIL ($($_.Exit))" } }}, Seconds -AutoSize | Out-String | Write-Host
|
||||
|
||||
# Write a small machine-readable summary alongside the logs.
|
||||
$summary = [pscustomobject]@{
|
||||
runId = $RunId
|
||||
layer = $Layer
|
||||
when = (Get-Date).ToString('o')
|
||||
results = $script:Results
|
||||
passed = @($script:Results | Where-Object Ok).Count
|
||||
failed = @($script:Results | Where-Object { -not $_.Ok }).Count
|
||||
}
|
||||
$summary | ConvertTo-Json -Depth 4 | Out-File (Join-Path $LogDir 'summary.json') -Encoding utf8
|
||||
Write-Info "Logs: $LogDir"
|
||||
|
||||
$failed = @($script:Results | Where-Object { -not $_.Ok }).Count
|
||||
if ($failed -gt 0) {
|
||||
Write-Fail "$failed layer(s) failed."
|
||||
exit 1
|
||||
}
|
||||
Write-Ok 'All layers passed.'
|
||||
exit 0
|
||||
@@ -173,6 +173,28 @@ const commands = {
|
||||
createDatabase(CONFIG.testDatabase);
|
||||
console.log(`✓ ${CONFIG.testDatabase} ready for go test ./...`);
|
||||
},
|
||||
|
||||
// Freshness guard for the e2e suite. The seed generates flights relative to
|
||||
// now(), so after ~3 weeks all seeded departures are in the past and the
|
||||
// browser tests fail on empty search results. Exit non-zero when stale so a
|
||||
// runner can re-seed before running.
|
||||
check() {
|
||||
if (!databaseExists(CONFIG.database)) {
|
||||
console.error(`✗ ${CONFIG.database} does not exist`);
|
||||
process.exit(2);
|
||||
}
|
||||
const out = run(
|
||||
['-tAc', 'SELECT count(*) FROM flights WHERE departure_time > now()'],
|
||||
{ database: CONFIG.database, capture: true },
|
||||
);
|
||||
const future = Number((out || '').trim());
|
||||
if (Number.isFinite(future) && future > 0) {
|
||||
console.log(`✓ inventory fresh: ${future} future flights`);
|
||||
process.exit(0);
|
||||
}
|
||||
console.error('✗ inventory stale: no future flights — re-seed needed (npm run db:reset)');
|
||||
process.exit(1);
|
||||
},
|
||||
};
|
||||
|
||||
const command = process.argv[2];
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
#requires -Version 5.1
|
||||
# Point git at the versioned hooks in .githooks so every developer gets the
|
||||
# same pre-commit / pre-push gates. Run once after cloning.
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$root = Split-Path $PSScriptRoot -Parent
|
||||
Push-Location $root
|
||||
try {
|
||||
git config core.hooksPath .githooks
|
||||
# Git for Windows runs hooks via its bundled bash; the files are sh scripts.
|
||||
Write-Host 'Git hooks installed (core.hooksPath -> .githooks):' -ForegroundColor Green
|
||||
Write-Host ' pre-commit : blocks committed secrets'
|
||||
Write-Host ' pre-push : runs backend + web unit tests'
|
||||
Write-Host ''
|
||||
Write-Host 'Bypass in an emergency with --no-verify (discouraged).'
|
||||
} finally {
|
||||
Pop-Location
|
||||
}
|
||||
Reference in New Issue
Block a user