# Testing Guide ## Test Strategy The project uses `bun test` (Bun's built-in test runner) with three test tiers: | Tier | Location | Database | What it tests | |------|----------|----------|---------------| | Unit | `tests/unit/` | None | Pure logic: permission registry, context guards, error classes, schema validation | | Integration | `tests/integration/` | Real PostgreSQL | Database operations: seed, RBAC joins, audit writes/queries | | E2E | `tests/e2e/` | Real PostgreSQL + real Typesense | Full lifecycle: invitation → activation → RBAC → audit, plus request-scoped search sync. Server starts automatically. | ## Running Tests Backend tests are only supported through the Nx targets below, which resolve to `bun run tests/run-profiles.ts ` (`apps/backend/tests/run-profiles.ts`). That launcher is the sole supported entrypoint: it applies the correct, fully-isolated environment profile for each tier, forwards only an explicit allowlist of host environment variables (never a blind copy of your shell), bootstraps and health-checks the dedicated E2E infrastructure before the "e2e" tier runs, and validates its own arguments before spawning anything. ### Locally: the unit tier, and individual files ```bash # Unit tests (no database needed) bun nx run supervin-core-backend:test # One file from the integration or e2e tier — pass the path to the launcher. # Running a handful of files this way is the normal loop while writing a test. bun run tests/run-profiles.ts integration ./tests/integration/visma-product-cases.test.ts bun run tests/run-profiles.ts e2e ./tests/e2e/orders.test.ts ``` The launcher routes each requested path to its matching profile, so a file only runs under the tier it belongs to. ### In CI: the whole integration and e2e tiers ```bash # Integration tests (requires TEST_DATABASE_URL against a dedicated test database) bun nx run supervin-core-backend:test:integration # E2E tests — bootstraps the isolated E2E infrastructure (docker-compose.e2e.yml) # automatically, then requires a dedicated test database and a running Typesense # instance bun nx run supervin-core-backend:e2e # All tiers, run sequentially through the same launcher bun run test:all ``` **These three are CI commands, not local ones.** CI shards integration 2× and e2e 3× across their own runners and finishes the whole pipeline in about four and a half minutes. A developer machine runs several worktrees at once and cannot; a full local tier is slower than CI *and* produces failures caused by contention rather than by the change under test. Trigger a run with `gh workflow run ci.yml --ref ` and read it with `gh run view --log-failed` — see [Validation](../../../../AGENTS.md#validation). A full local run is still the right tool for debugging the E2E infrastructure itself. Stop this worktree's dev servers first (`bun run dev:stop`); the tier allows each test 90 seconds, and with dev servers competing for the machine that budget goes on waiting instead. Do **not** invoke `bun test tests/e2e` (or any other test directory) directly. It is not a supported entrypoint: it skips the environment profile, the host environment allowlist, the E2E infrastructure preflight, and queue provisioning that `run-profiles.ts` performs. As defense in depth, `tests/e2e/server.ts` independently refuses to start unless `ENABLE_E2E_TEST_MODE=true` and `TEST_DATABASE_URL` resolves to a verified, dedicated test database (see `tests/config/e2e-db-guard.ts`), so a direct invocation fails fast instead of quietly touching the wrong database — but it is still an unsupported way to run these suites. ### AI assistant evals against a real model ```bash set -a && source .env && set +a # the provider key must be in the shell bun run ai:eval # every case in tests/ai-evals/cases/ AI_EVAL_CASE= bun run ai:eval # one case AI_EVAL_REPEAT=3 bun run ai:eval # measure every case three times AI_EVAL_ATTEMPTS=1 bun run ai:eval # no retries: the raw pass rate bun run ai:eval:smoke # the ~4-minute tier a pull request runs bun run ai:eval:fast # the whole suite across 8 slices, ~3 min AI_EVAL_CASE=a,b bun run ai:eval # a named few ``` **The suite is almost entirely waiting for the model.** One measurement: 32 cases, 123 model round trips, 15,9 minutes — and the fastest case still took 9,9 s while making three of those calls. `--ai-shards=N` splits the cases round-robin across N processes, each of which gets its own worker database and queues exactly like any other parallel file, so the isolation the cases need (several assert a *sum*, which another case's seeded rows would change) costs almost nothing. Eight slices took the full suite from 15,9 minutes to 3m22s. Round-robin rather than contiguous: the dashboard cases take four times the median and sit next to each other, so blocks would load one slice with all of them. Each case is a question that once got a bad answer, usually from AI-feedback marked "Klar til udvikling", plus what a good answer does. It goes through the real `/ai/chat`, so the system prompt, the tools and the schema registry are measured together. Case files are validated in the unit tier (`tests/unit/ai-evals.test.ts`). The workflow that produces cases is the `ai-feedback-triage` skill. CI runs the evals with the production model, as often as the change needs, because every run spends tokens: - **every night on develop**, the full suite twice (`release-evals.yml`) — skipped when AI code has not changed since the last green run. This is what gates releases: the release PR into `main` cannot be merged until `supervin/release-gate` is green, and that needs this run green for the release's code. After AI code changed during the day, run it by hand: `gh workflow run release-evals.yml --ref develop`. - a pull request into `develop` that touches the assistant (the files listed in `tools/ci/eval-coverage.ts`), not while it is a draft: the **smoke tier** — one case per prompt section; a unit test fails if a tool loses its smoke coverage — **plus the cases its changes touch**: changed case files, a changed skill's or prompt rule's `provenCases`, and the cases that name a tool whose definition changed (`tools/ai-eval-select.ts`). Each once. - a manual `gh workflow run ci.yml --ref `: the same selection; `-f evals=full` for every case twice. A newer push cancels the run in flight, so a pull request pays for its latest commit only. Each run ends with one line, `pris i alt: $x`, in its log and summary. **A failing case is retried, and is red only when all three tries fail** (`AI_EVAL_ATTEMPTS=3` in CI) **and its baseline says it normally passes** (`AI_EVAL_GATE_RATE`, 90 % of first tries, in `tests/ai-evals/evaluate.ts`; a case without a baseline counts as one that should pass). About 2% of case-runs fail for reasons unrelated to the change under test, and demanding that every try pass made a 26-case suite red 41% of the time — a gate that goes redder the more cases you add is a gate that stops meaning anything. A case that needed a retry is named as unstable in the job summary; a shaky case that failed every try is named too, as a warning — unless it failed every try of every repetition of a nightly run, which is a break, not bad luck, and turns the run red (`tools/ai-eval-baseline.ts shaky`). Retries alone would hide a case that is genuinely bad rather than unlucky, so every run is also read against `tests/ai-evals/baseline.json` — the measured pass rate per case, produced by the `ai-eval-baseline.yml` workflow (ten runs with retries off) and committed by a human. Both sides count the **first** attempt, so a case that only scraped through on a retry reads below its rate and is called out. See `tools/ai-eval-baseline.ts`. ```bash gh workflow run ai-eval-baseline.yml --ref develop # ~10 parallel jobs, ~1 hour, ~$5 ``` `workflow_dispatch` only works once the workflow file is on the default branch — before that `gh workflow run` answers 404 whatever `--ref` says. Locally the same measurement is one sharded run: ```bash AI_EVAL_REPEAT=10 AI_EVAL_ATTEMPTS=1 AI_EVAL_RESULTS_JSON=/tmp/b.json bun run ai:eval:fast bun run tools/ai-eval-baseline.ts aggregate /tmp/b.shard-*.json > tests/ai-evals/baseline.json ``` The committed baseline is 31 cases × 10 runs, mean 95%, lowest 80%. A unit test requires an entry for every case, so adding a case without measuring it fails CI for free rather than showing "⚠️ ingen baseline" forever. Tracking questions need no BigQuery: `ai:eval` sets `AI_EVAL_TRACKING_FIXTURE=true`, which serves the `tracking_*` tables from Postgres tables a case seeds in its `sql` (`tests/ai-evals/tracking-fixture.ts`). `env.ts` refuses the flag outside e2e test mode. Every run ends with what it cost: a table of tokens and USD per case run, the assistant's model calls and the rubric judge's together, printed to the log and in CI to the job summary. The route sends one usage entry per model call with the finished answer (`calls` in the message metadata), because a price depends on each request's size. Prices live in `src/modules/ai/model-pricing.ts`, with the page each was read from and the date; a model that is not listed is named in the report as unpriced rather than guessed at. Add it there when the eval or production model changes. ### AI-udfylder evals (the AI competitor setup) ```bash set -a && source .env && set +a bun run competitor-eval # every case in tests/competitor-setup-evals/cases/ COMPETITOR_EVAL_CASE=jyskvin-dk bun run competitor-eval # one bun run competitor-eval:corpus pull # new confirmed competitors from production ``` Each case is a competitor that marketing set up. It holds the frozen reference pages and the values a human confirmed. A case passes only when three stages pass: the reading matches the confirmed values, the generated code reproduces them on its first attempt, and that code also works on a held-out product it never saw. The goal is 95 %. The suite calls the modules directly, with no server and no database. Its CI job is `competitor-setup-evals`. The job runs the smoke tier on pull requests that touch the setup, and the full tier every night (`release-evals.yml`), which gates releases like the assistant's. It only goes red for a case whose baseline says the case should pass. The full description is in [tests/competitor-setup-evals/README.md](../../tests/competitor-setup-evals/README.md). ### Translation evals (the translation engine) ```bash set -a && source .env && set +a bun run translation-eval # every case in tests/translation-evals/cases/ TRANSLATION_EVAL_CASE=sv-da-kronor-become-sek bun run translation-eval # one ``` Each case is a text an editor would write — a review in Danish, Swedish or English — translated into another storefront's language by `translateEntityFields`, the function a translation job calls, with the entity's real prompt, fixed rules and output normaliser. Prices (DKK for Danish, SEK for Swedish), names, scores and vintages are checked deterministically; whether the result is natural target language and leaves nothing out is judged by the model, one criterion at a time. Every case holds a `reference` translation the unit tier checks against its own checks, so a case no translation could pass fails there. Its CI job is `translation-evals`: the smoke tier on pull requests that touch the engine, the full tier twice on `klar-til-merge` and on develop (`release-evals.yml`), which gates releases like the other two. The full description is in [tests/translation-evals/README.md](../../tests/translation-evals/README.md). If you use the local Docker Compose setup, start dependencies first: ```bash bun run dev:start ``` `bun run dev:start` provisions the standard local development stack and a dedicated Playwright/test sidecar stack backed by the physically isolated E2E infrastructure (`docker-compose.e2e.yml`, started via `bun run e2e:up`): | Purpose | Frontend | Backend | Database | |---------|----------|---------|----------| | Normal development | `http://localhost:43173` | `http://localhost:34045` | `supervin` (dev PostgreSQL) | | Playwright/runtime e2e | `http://localhost:43174` | `http://localhost:34046` | Dedicated E2E PostgreSQL instance (see `docker-compose.e2e.yml`) | It also runs migrations against the dedicated E2E database. You should not need to export `TEST_DATABASE_URL`, `PLAYWRIGHT_API_URL`, or `PLAYWRIGHT_BASE_URL` for the standard local workflow, and Playwright itself now refuses to start against any target other than the isolated E2E sidecar above (see `apps/frontend/playwright.config.cts`). ## Database Selection Invariant The runtime database is chosen by `ENABLE_E2E_TEST_MODE`, **not** by whether `TEST_DATABASE_URL` happens to be set. This matters because the shared `apps/backend/.env` defines *both* `DATABASE_URL` and `TEST_DATABASE_URL`, so a naive `TEST_DATABASE_URL ?? DATABASE_URL` fallback would silently point the dev backend at the test database that e2e suites truncate. | Mode | When | Database used | |------|------|---------------| | `ENABLE_E2E_TEST_MODE=false` | Dev backend (port `34045`), prod | `DATABASE_URL` (`supervin`) | | `ENABLE_E2E_TEST_MODE=true` | E2e sidecar (port `34046`), Bun e2e/integration suites | `TEST_DATABASE_URL` (dedicated E2E database `supervin_e2e_test`, see `docker-compose.e2e.yml`) | Rules: - `src/db/client.ts` (`getBaseDatabaseUrl`) and `drizzle.config.ts` both key off `ENABLE_E2E_TEST_MODE`. They are the source of truth — keep them in sync and never reintroduce a `TEST_DATABASE_URL ?? DATABASE_URL` fallback. - `ENABLE_E2E_TEST_MODE=true` requires `TEST_DATABASE_URL`; the backend fails fast otherwise instead of falling back to the dev database. - `TEST_DATABASE_URL` must target the dedicated E2E PostgreSQL stack exactly: loopback host, port `5434` (never the shared dev PostgreSQL's port `5432`, and never the fixed port `5433` of an unrelated Compose project), role `supervin_e2e`, and a database name starting with `supervin_e2e` (see `tests/config/e2e-db-guard.ts`). A `localhost:5432`/`supervin`-role target, or a merely "test"-named database on the dev instance, is rejected — not just accepted-then-warned-about. - Integration tests preload `tests/integration/setup-env.ts`, which forces `ENABLE_E2E_TEST_MODE=true` so the eager `db` export resolves to the test database at import time. This is why e2e runs no longer wipe dev data or sessions: dev (`supervin`, port `5432`) and the dedicated E2E database (`supervin_e2e_test`, port `5434`, `docker-compose.e2e.yml`) are physically separate PostgreSQL instances and can run side by side. ## Test Database Setup Integration tests and Bun-driven e2e suites that use `tests/helpers/db.ts` need the dedicated E2E PostgreSQL stack (`docker-compose.e2e.yml`) running, with the schema applied. `bun run e2e:up` (`tools/scripts/e2e-up.sh`) does both — it starts the stack and migrates it using an explicit, minimal environment (no inherited shell variables, no auto-loaded `.env` file), and is the only supported way to provision it: ```bash bun run e2e:up ``` Do not rely on `DATABASE_URL` fallback for Bun test suites. The helper in `tests/helpers/db.ts` refuses to run unless `TEST_DATABASE_URL` is set, and requires it to be the exact dedicated E2E identity described above. Several suites call `cleanAllTables()`, so this is a hard guard against truncating the primary development database — or any other database that merely looks test-like. If the variable is missing or malformed, the helper prints a concrete error explaining the exact expected identity and reminds you to use `bun nx run supervin-core-backend:test:integration` (which resolves the dedicated E2E profile automatically via `tests/config/environment-profiles.ts`) rather than setting `TEST_DATABASE_URL` by hand. E2E tests start and stop the application server automatically in each test suite — no need to start it manually. When running through the local dev workflow, the Playwright backend is already running on port `34046` with `ENABLE_E2E_TEST_MODE=true` and a dedicated `TEST_DATABASE_URL`. This is separate from request-scoped frontend e2e against a dedicated test server: - Bun test suites use `tests/helpers/db.ts` and now require `TEST_DATABASE_URL`. - Runtime request-scoped e2e uses `X-Test-run-id` and provisions isolated PostgreSQL schemas inside the configured test database. - That runtime flow is enabled by `ENABLE_E2E_TEST_MODE=true`, which now requires `TEST_DATABASE_URL`; it must not run against the normal development backend. ## Typesense Test Setup The search-related e2e tests use a real Typesense instance. Nothing is mocked. Expected local settings (see `docker-compose.e2e.yml`, started via `bun run e2e:up`): ```dotenv TYPESENSE_HOST=localhost TYPESENSE_PORT=8119 TYPESENSE_PROTOCOL=http TYPESENSE_API_KEY=supervin-typesense-e2e-key ``` These defaults match the dedicated E2E stack (`docker-compose.e2e.yml`), where Typesense is published on `localhost:8119` — a separate instance from the dev stack's Typesense on `localhost:8108` (`docker-compose.yml`). The backend itself runs in-process during e2e tests, but it talks to real PostgreSQL and real Typesense using the configured environment variables. ## Test Helpers Located in `tests/helpers/db.ts`: - `getTestDb()` — returns a Drizzle database instance for test use (requires `TEST_DATABASE_URL`) - `cleanAllTables(db)` — truncates all tables in FK-safe order - `closeTestDb()` — closes the connection pool E2E tests also use `tests/e2e/server.ts`: - `setupE2eServer()` — starts the Elysia app in-process on a random port, returns the base URL - `teardownE2eServer()` — stops the server - `getBaseUrl()` — returns the base URL of the running test server Search-related e2e tests also use `tests/e2e/helpers.ts`: - `gql()` — shared GraphQL request helper with optional bearer token and `X-Test-run-id` - `typesenseGet()` — direct Typesense HTTP helper for asserting indexed state ## What the Tests Cover ### Unit Tests - Permission keys follow `domain.action` format - All permissions have descriptions - No duplicate permission keys - GraphQL schema contains all required types and fields - `requireAuth` / `requirePerm` guards behave correctly - Custom error classes have correct properties ### Integration Tests - Seed creates all permissions and Superadmin role - Superadmin gets all permissions bound - Custom roles can be created with permission bindings - Users inherit permissions through role assignment - Audit log entries can be written and queried - JSONB before/after diffs work correctly ### E2E Tests - Admin can create an invitation - Invited user is inactive before first login - OAuth activation flow works correctly - Activated user gets the correct permissions - Security operations create audit entries - Superadmin role cannot be deleted - Product create, update, and delete sync to Typesense - `searchConfig(type: "products")` returns frontend Typesense configuration - Scoped Typesense keys can query the indexed collection directly - `X-Test-run-id` isolates both PostgreSQL data and Typesense collection names - Bun test helpers refuse `DATABASE_URL` fallback and same-DB `TEST_DATABASE_URL` values to avoid truncating the primary database ## Frontend E2E Login Shortcut Frontend e2e tests do not need to go through Google or Microsoft OAuth. Use the `devLogin` GraphQL mutation to mint a session token directly for an existing active user. This is intended for local development and automated tests only. It is blocked in production. ### Typical flow 1. Ensure the test user exists. 2. Ensure the test user is active. 3. Call `devLogin(email: String!)`. 4. Store the returned token in the frontend test runner. 5. Send it as `Authorization: Bearer ` on subsequent GraphQL requests. ### Example mutation ```graphql mutation DevLogin($email: String!) { devLogin(email: $email) { token user { id email } } } ``` Example variables: ```json { "email": "admin@supervin.dk" } ``` ### Example response ```json { "data": { "devLogin": { "token": "", "user": { "id": "...", "email": "admin@supervin.dk" } } } } ``` ### Important constraints - `devLogin` only works outside production. - The user must already exist. - The user must already be active. - Invited users created via `createInvitation` start as inactive, so activate them before using `devLogin` if your test depends on that user. - `bootstrapFirstUser` is useful for the first admin in a clean environment, while `devLogin` is the normal shortcut for repeated e2e logins. ## Request-Scoped Frontend E2E Runs The backend supports request-scoped frontend e2e isolation via the `X-Test-run-id` header when `ENABLE_E2E_TEST_MODE=true` and `TEST_DATABASE_URL` points at a dedicated test database. ### Purpose - Each unique `X-Test-run-id` value maps to its own isolated PostgreSQL schema. - Each unique `X-Test-run-id` value also maps to its own isolated Typesense collection. - The first request for a new test run provisions the schema automatically. - Provisioning runs migrations, seeds permissions and the Superadmin role, and ensures the active baseline superadmin user `cra@supervin.dk` exists. ### Safety guard - If `ENABLE_E2E_TEST_MODE` is not set to `true`, any request carrying `X-Test-run-id` is rejected with a 400 error. - If `ENABLE_E2E_TEST_MODE=true` is set without `TEST_DATABASE_URL`, backend startup fails so test schemas cannot be created inside the development database. - Frontend Playwright defaults use a separate frontend port (`43174`) and API port (`34046`) and reject the normal development ports (`43173` and `34045`). - This applies before GraphQL execution, so browser and test-runner requests fail fast instead of silently hitting shared data. ### Typical frontend flow 1. Choose a unique `X-Test-run-id`, for example `e2e__`. 2. Send GraphQL requests with that header on every request in the run. 3. Call `devLogin(email: "cra@supervin.dk")` to get a bearer token for the baseline superadmin in that isolated scope. 4. Run the suite using the same header and bearer token. 5. Call `cleanupTestRun` with the same header to drop the isolated schema and delete the isolated Typesense collection when the suite is done. ### Example request headers ```http Content-Type: application/json Authorization: Bearer X-Test-run-id: frontend_run_alpha ``` ### Example cleanup mutation ```graphql mutation CleanupCurrentTestRun { cleanupTestRun } ``` ### Notes - `X-Test-run-id` is included in CORS allow-headers so browser-based e2e tests can send it. - The isolated schema is created lazily on first use, so there is no separate setup endpoint required for baseline provisioning. - Search writes go to `products_e2e_` during isolation, so test traffic does not pollute the shared `products` index. - `cleanupTestRun` requires the `X-Test-run-id` header and only removes the current isolated test scope.