Milestone 3

CHANGELOG.md

@@ -0,0 +1,150 @@
+# Changelog
+
+All notable changes to this project will be documented here. Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) · [Conventional Commits](https://www.conventionalcommits.org/).
+
+## [Unreleased]
+
+### Added — M3 (RBAC: groups, permissions, users)
+- **Permission catalogue** (`app/services/permissions_seed.py`): 31 atomic codes across 10 families (`user`, `group`, `invitation`, `test_template`, `scenario_template`, `mission`, `detection_level`, `setting`, `mitre.sync`). Seeded at boot **and** after `/setup` to handle a freshly truncated DB. Idempotent + additive on system groups (never removes a perm).
+- **Default group bindings**: `admin` = all 31 codes; `redteam` = 8 (catalogue read + mission.{read,create,update,archive,write_red_fields} + detection_level.read); `blueteam` = 5 (catalogue read + mission.{read,write_blue_fields} + detection_level.read).
+- **Users admin service + API** (`app/services/users.py`, `app/api/users.py`): list (q + is_active filter + pagination), get, patch (display_name/locale/is_active), soft-delete, set groups. Last-admin protection on update/delete/group-strip.
+- **Groups admin service + API** (`app/services/groups.py`, `app/api/groups.py`): full CRUD with system-group protection (no rename, no delete), `PUT /groups/{id}/permissions` for the bindings. Admin system group's perm set is locked to "every perm" (preserves the bypass invariant).
+- **Permissions read-only API** (`app/api/permissions.py`): `GET /permissions` returns the catalogue (admin or `group.read` holders).
+- **Frontend admin pages** (`frontend/src/pages/Admin{Users,Groups,Invitations}Page.tsx`): list + edit modals using TanStack Query mutations, multi-select for perms grouped by family, copy-once invitation URL display.
+- **Frontend chrome** (`Layout.tsx` + `RequireAdmin.tsx`): admin nav links shown only when `is_admin === true`; direct navigation to `/admin/*` by non-admins redirects to `/`. Server remains the arbiter.
+- **`/diag/reset` now clears the rate-limit counters** so the Playwright suite can iterate without hitting `10/min/IP` budgets across spec files. Gated to non-prod environments only.
+- **Testing**:
+  - `tests/test_rbac.py` — **15 pytest integration tests** (39 backend total).
+  - `e2e/tests/m3-rbac.spec.ts` — **8 Playwright tests** covering DoD §10 #2/#3 (28 e2e total).
+  - `tasks/testing-m3.md` — manual + automated procedure.
+- **Frontend api helpers**: `apiPatch`, `apiPut`, `apiDelete` added to `frontend/src/lib/api.ts`.
+
+### Fixed (post-M3 spec-review pass)
+- **Rate-limit scope clarified**: `app/core/rate_limit.py` now enables the limiter for `APP_ENV in ("prod", "staging")` instead of `prod` only — a public staging deployment without auth limits would be surprising. Dev/test stay unthrottled for Playwright ergonomics. Spec §6 NF-security applies to operator-facing deployments.
+- **Admin perm invariant**: `set_group_permissions` refuses to alter the admin system group's perm set to anything other than the full catalogue (`SystemGroupProtected` → 409). The decorator bypass relies on `is_admin = "admin" in group_names`, but a future refactor could move to a perm-based check, so we keep the invariant.
+- **LogRecord field collision**: `log.info("...", extra={"name": g.name})` raised `KeyError: "Attempt to overwrite 'name' in LogRecord"` because Python's logger reserves `name`. Renamed to `group_name`. Audited all other `extra=` payloads in `app/api/`+`app/services/` for the same trap.
+
+### Validated end-to-end (M3 DoD)
+- `make clean && make up && make migrate` → boot logs show `metamorph.permissions.seeded {perms_created: 31, perms_total: 31, bindings: {admin: 31, redteam: 8, blueteam: 5}}`.
+- `make test-api` → **39 pytest pass** (1 health + 8 schema + 15 auth + 15 RBAC) in ~4 s.
+- `make e2e` → **28 Playwright pass** (8 M0 + 4 M1 + 8 M2 + 8 M3) in ~16 s.
+- Spec-reviewer pass: PASS verdict, 2 minor fixes applied (above), 2 anticipations noted for M12/M14 (no current action).
+
+### Added — M2 (Auth, bootstrap, invitations)
+- **Crypto plumbing**: `app.core.security` (Argon2id `time_cost=2 memory_cost=64MiB parallelism=2`, opaque-token SHA-256 helpers), `app.core.jwt_tokens` (HS256, claims `iss/sub/type/jti/iat/exp`, access 1h / refresh 30d).
+- **Auth services** (`app.services.auth`): login, refresh **with token rotation + reuse-detection chain revoke**, logout (idempotent), change_password (forces logout-all).
+- **Invitation services** (`app.services.invitations`): create, preview, accept, revoke. Token persisted only as SHA-256, default 7-day TTL.
+- **Bootstrap** (`app.services.bootstrap` + `app.core.install_token`): seeds 3 system groups (`admin`/`redteam`/`blueteam`), mints a one-shot install token at first boot when `users` is empty, logs a banner with the raw token. CLI `flask --app app.cli metamorph print-install-token [--force]`.
+- **Auth middleware** (`app.core.auth_decorators`): `@require_auth` populates `g.current_user`; `@require_perm("...")` checks atomic permissions; admin group bypasses the check (atomic perms land in M3).
+- **API endpoints**:
+  - `POST /api/v1/setup` (consume install token, create 1st admin) + `GET /api/v1/setup` (status).
+  - `POST /api/v1/auth/login` + `POST /auth/refresh` + `POST /auth/logout` + `GET /auth/me` + `POST /auth/change-password`.
+  - `POST /api/v1/invitations` (admin) + `GET /invitations` + `GET /invitations/preview/<token>` + `POST /invitations/accept/<token>` + `POST /invitations/<id>/revoke`.
+  - `POST /api/v1/diag/reset` (test-only kill switch — wipes auth tables + mints fresh install token; only available in `dev`/`test`).
+- **Rate limiting** (`flask-limiter`): 10/min/IP on `/auth/login`, `/auth/refresh`; 5/min on `/auth/change-password` and `/setup`; 10–20/min on invitation endpoints. Globally disabled when `APP_ENV=test`.
+- **Refresh cookie** `metamorph_refresh`: HttpOnly + Secure + SameSite=Strict + Path=`/api/v1/auth/`.
+- **Frontend auth state** (`frontend/src/lib/{api,auth}.ts`): access token in module memory, refresh in cookie, automatic 401-retry via `/auth/refresh` with reentrancy guard. `useAuth()` hook + `<RequireAuth>` route guard.
+- **Frontend pages**: `/login`, `/setup`, `/register?token=…`, `/profile` (with change-password form), all in RTOps design. Protected layout: nav shows email + Logout when authenticated, Login + Setup links when not.
+- **Frontend deps**: `@tanstack/react-query`, `react-router-dom`. Tanstack provider in `App.tsx` (will carry actual queries from M3+).
+- **Email validation** (`app.api._validation.Email`): permissive RFC-shape regex that accepts internal TLDs (`.local`, `.corp`) — `pydantic.EmailStr` was too strict for red-team labs.
+- **Testing**:
+  - `tests/test_auth_flow.py` — **15 pytest integration tests** (24 backend total with M0/M1).
+  - `e2e/tests/m2-auth.spec.ts` — **8 Playwright tests** covering setup → login → me → invitation → register → 2nd login → RBAC 403 → refresh rotation → logout (20 e2e total).
+  - `tasks/testing-m2.md` — manual + automated procedure.
+
+### Fixed (post-M2 spec-review pass)
+- **Refresh cookie `Secure=True`** unconditionally (`backend/app/api/auth.py`). Modern browsers treat `localhost` as a secure context, so dev/test still works. Closes the silent-degradation found by the reviewer.
+- **`/auth/refresh` rate-limit lowered to 10/min/IP** (`backend/app/api/auth.py`) to match spec §M2 ("10 req/min/IP on `/auth/*`").
+- `/diag/reset` kept allowed in `dev` *and* `test` (a `make e2e` against a `make up` dev stack must be able to reset). Added a WARNING log when triggered in `dev` and a clear docstring; production envs (`prod`/`staging`) remain locked out.
+
+### Known scope-creep (intentional, not retracted)
+- Rate-limits on `/setup` (5/min), `/invitations/preview` (20/min), `/invitations/accept` (10/min) and `/auth/change-password` (5/min) were added in M2 even though §M2 only mandated `/auth/*`. Defensible (these are abuse-attractor endpoints), and noted here so M14 doesn't double-spec them.
+
+### Added — M1 (DB schema & migrations)
+- **23 tables** + `alembic_version` covering auth/RBAC (8), MITRE (4), templates (4), missions (6), evidence (1), settings/detection-levels (2), notifications (1).
+- SQLAlchemy 2.x declarative models with Mapped[]/mapped_column(), grouped under `backend/app/models/{auth,mitre,template,mission,evidence,setting,notification}.py`.
+- Alembic init: `alembic.ini`, `alembic/env.py` reading `app.core.config.settings.database_url`, `alembic/script.py.mako`, naming convention `pk_/fk_/ck_/uq_/ix_` enforced via `MetaData(naming_convention=...)` on `app.db.base.Base`.
+- Reusable mixins in `app.db.mixins`: `UuidPkMixin` (uuid4 server-side), `TimestampMixin` (created_at/updated_at, server-default + onupdate), `SoftDeleteMixin` (deleted_at, no auto-injected index — declared explicitly per table to avoid mixin-vs-class `__table_args__` clobbering).
+- Postgres-specific features used: `JSONB` for `settings.value` and `notifications.payload`; native `Uuid` columns; partial indexes (`WHERE deleted_at IS NULL` on 9 tables; `WHERE read_at IS NULL` on `notifications`); CHECK constraints for status/state/opsec_level/mitre_kind enums; `exactly_one_mitre_fk` CHECK on `test_template_mitre_tags`.
+- **`mission_test_mitre_tags` deliberately denormalised** (no FK to `mitre_*` tables): copies `mitre_external_id`, `mitre_name`, `mitre_url` at tag time so a later MITRE re-sync that drops an entry cannot purge a mission's tags. Companion `test_template_mitre_tags` keeps FKs since templates are editable. (Spec §11 risk addressed.)
+- Backend `pyproject.toml` deps: SQLAlchemy ≥2, Alembic ≥1.13, psycopg[binary] ≥3.1.
+- New Makefile targets: `migrate`, `migrate-down`, `migrate-revision MSG=…`, `migrate-status`. The Dockerfile now ships `alembic.ini` + `alembic/` so the api container can run migrations directly.
+- **Test stage in `backend/Dockerfile`** (`--target test`): runtime image + dev extras + `tests/` dir. New `make test-api` target spins an ephemeral container against the live DB on the compose network. Backend tests no longer require any local Python toolchain.
+- `tests/test_schema.py` (8 integration tests + the existing M0 health test = 9 total): expected tables, expected timestamp/soft-delete columns, partial-index presence, expected FK pairs, expected CHECK constraints, alembic-at-head, and a negative INSERT proving the `exactly_one_mitre_fk` CHECK fires.
+- `tasks/testing-m1.md` — manual + automated verification procedure.
+
+### Fixed (post-M1 spec-review pass)
+- Soft delete now consistent across snapshot-bearing tables: `mission_scenarios`, `mission_tests`, `mission_categories` gained `SoftDeleteMixin` + their `ix_<table>_active` partial index (M12 trash bin depends on this).
+- `evidence_files` gained `TimestampMixin` (`created_at`/`updated_at`) on top of the domain `uploaded_at` (audit minimal everywhere, per M1 brief).
+- `mission_members` gained `TimestampMixin`, replacing the bespoke `added_at` column.
+- `scenario_template_tests` PK refactored to a UUID + `UNIQUE(scenario_template_id, position)` so the same test can appear at multiple positions in a scenario (chained operations).
+- `SoftDeleteMixin.__table_args__` removed (silently clobbered by class `__table_args__`); each soft-delete table now declares `ix_<table>_active` explicitly. Documented in the mixin's docstring.
+- `mission_test_mitre_tags` schema redesigned to denormalise MITRE labels (see "Added" entry above).
+- Migration 0001 regenerated end-to-end after these fixes — `24765a5014b6` is the new HEAD.
+
+### Validated end-to-end (M1 DoD)
+- `make clean && make up && make migrate` from a vide DB → 27 tables, 32 FK, 9 CHECK, 14 UQ, 12 partial indexes.
+- `make test-api` → **9 pytest pass** (1 health + 8 schema integration) in <1 s.
+- `make e2e` → **12 Playwright pass** (8 M0 smoke + 4 M1 db visibility) in 3 s.
+
+### Added (M1 visibility)
+- New API endpoint `GET /api/v1/diag/db` exposes `alembic_revision` (short-hashable) and the public-schema `table_count`. Returns 503 with `{"reachable": false}` when Postgres is down.
+- New `Database` card on the SPA home page consumes that endpoint, renders the revision short-hash and the count next to the existing `API` and `Roadmap` cards.
+- Footer updated to `M0 bootstrap · M1 db schema`. Roadmap card now points to `M2 — Auth + JWT`.
+- New e2e suite `e2e/tests/m1-db.spec.ts` (4 tests) covers the diag endpoint contract, the Database card rendering, and the footer/roadmap labels.
+
+### Added — M0 (bootstrap)
+- Repo scaffolding: `.gitignore`, `.env.example`, `Makefile`, `docker-compose.yml`, `README.md`, `CHANGELOG.md`.
+- `docker-compose.yml` with three services: `db` (postgres:16-alpine, no host port), `api` (Flask 3, port 8000), `front` (nginx serving the Vite bundle, port 80).
+- Named volumes `metamorph_db` and `metamorph_evidence` for data persistence.
+- Backend skeleton: Flask app factory, JSON structured logging on stdout, `GET /api/v1/health` endpoint, multi-stage Dockerfile, `pyproject.toml` driven by `uv`.
+- Frontend skeleton: Vite + React 18 + TypeScript strict + TailwindCSS, RTOps design tokens (`tasks/design.md`) translated into `tailwind.config.ts`, base UI primitives (`Card`, `Tag`, `SectionHeader`, `FlowNode`, `Button`), home page wired to `/api/v1/health`.
+- Multi-stage frontend Dockerfile that builds the bundle and serves it via nginx, proxying `/api/*` to the api container.
+- Pre-commit hook config: `ruff` for backend, `eslint` + `tsc --noEmit` for frontend.
+
+### Validated
+- `docker compose config` parses (validated via `pyyaml` since Docker is not installed in the dev shell).
+- Every env var referenced by the compose file is documented in `.env.example`.
+- All Python source files parse cleanly (`ast.parse`).
+- All TS/JSON config files parse cleanly.
+
+### Notes
+- TLS termination is delegated to an external reverse proxy (per spec §6 NF-network). The compose stack exposes plain HTTP on `HOST_FRONT_PORT` (8080) and `HOST_API_PORT` (8000).
+- The first-admin bootstrap token (M2) will be printed to the api container's stdout on first boot when the `users` table is empty.
+- `tasks/spec.md` and `tasks/todo.md` remain authoritative; update them before changing scope.
+
+### Fixed (M0 DoD validation pass on real podman)
+- **FQDN image references** in `docker-compose.yml`, `backend/Dockerfile`, `frontend/Dockerfile`. Podman on Fedora enforces `short-name-mode=enforcing` for pulls (no TTY ⇒ no prompt ⇒ failure). Replaced `postgres:16-alpine` / `python:3.12-slim` / `node:20-alpine` / `nginx:1.27-alpine` with their `docker.io/library/…` qualified equivalents. Docker accepts the same prefix transparently.
+- **`*.md` removed from `backend/.dockerignore` and `frontend/.dockerignore`**: `pyproject.toml` declared `readme = "README.md"`, but the file was being filtered out of the build context, so `hatchling.build.build_wheel` raised `OSError: Readme file does not exist: README.md`. Also removed the `readme` field itself from `pyproject.toml` to decouple the build from the doc.
+- **`Card.tsx` type clash**: `CardProps extends HTMLAttributes<HTMLDivElement>` redefined `title` as `ReactNode`, but the native `title` is `string`. `tsc -b` failed with TS2430 during `vite build`. Switched to `Omit<HTMLAttributes<HTMLDivElement>, 'title'>`.
+- **Explicit healthchecks added to compose `api` and `front`**: podman-compose 1.x doesn't surface healthchecks declared only in the `Dockerfile` via `inspect`. Mirroring them in `docker-compose.yml` makes `make inspect-health` actually see `healthy/unhealthy/starting` on every engine.
+- **Suppressed `podman compose` external-provider banner** via `PODMAN_COMPOSE_WARNING_LOGS=false` exported from the Makefile.
+
+### Validated end-to-end on podman 5.x (Fedora 43)
+- `make up` → 3 containers, all 3 healthy after start_period.
+- `make health` → `{"status":"ok","version":"0.1.0"}` via the front nginx proxy (port 8080) and direct API (port 8000).
+- `make logs-api` → JSON-structured lines on stdout (`ts`, `level`, `logger`, `message`, custom fields).
+- `make e2e` → **8/8 Playwright tests pass** in 2.5 s. Reports: `e2e/playwright-report/index.html` (529 KB, autoportant) + `junit.xml` (`tests=8 failures=0 skipped=0 errors=0`).
+
+### Added (engine portability)
+- Makefile auto-detects **docker** or **podman** at runtime and selects the matching compose driver (`docker compose`, `podman compose`, or legacy `podman-compose`). Override via `ENGINE=…` and/or `COMPOSE="…"`.
+- New targets: `engine` (print detected runtime), `volumes` (list project-named volumes), `inspect-health` (health status of all 3 containers), `logs-api` (tail just the api), `health` (single curl probe). All engine-agnostic.
+- `make help` now prints the active engine + compose driver in its footer.
+- `tasks/testing-m0.md` and `README.md` rewritten to be engine-agnostic — raw `docker logs` / `docker volume ls` / `docker inspect` calls replaced with the new make targets.
+
+### Added (M0 testing)
+- `e2e/` Playwright project with chromium, HTML + JUnit XML reporters, traces / screenshots / videos kept on retry. Reports land in `e2e/playwright-report/`.
+- `e2e/tests/m0-smoke.spec.ts` — 8 smoke tests covering the front rendering, the API proxy, the design tokens, the absence of any runtime CDN traffic (spec §7), and the CORS contract.
+- Makefile targets `e2e-install`, `e2e`, `e2e-report`, `e2e-up`, `wait-healthy`.
+- `tasks/testing-m0.md` — step-by-step manual + automated verification procedure for M0.
+- Convention added to `tasks/todo.md`: every milestone N delivers `tasks/testing-m<N>.md` + at least one `e2e/tests/m<N>-*.spec.ts`, and the spec-reviewer subagent runs before marking the milestone done.
+
+### Fixed (post-M0 spec-review pass)
+- `.pre-commit-config.yaml` added at repo root: ruff + ruff-format on backend, eslint + tsc --noEmit + prettier --check on frontend, plus baseline whitespace/JSON/private-key checks. Documented `pre-commit install` in `README.md`.
+- Self-hosted webfonts via `@fontsource/jetbrains-mono` and `@fontsource/ibm-plex-sans` (imported in `frontend/src/index.css`); dropped the Google Fonts `<link>` from `frontend/index.html` to honor spec §7 ("no runtime CDN").
+- Refuse-to-boot guard in `backend/app/core/config.py`: when `APP_ENV != "dev"`, defaults / placeholders for `JWT_SECRET` and `POSTGRES_PASSWORD` raise at startup. New `APP_ENV` env var documented in `.env.example`, `README.md`, and `docker-compose.yml`.
+- `make dev` now runs `dev-api` and `dev-front` in parallel via `make -j2` instead of just printing a hint.
+- Removed dead `database_url` property from `Settings` (will be reintroduced in M1 with the SQLAlchemy/Alembic stack).
+- Pinned Node engines to `>=20` in `frontend/package.json`.
+- Reconciled M0 DoD wording in `tasks/todo.md` (HTTP via `HOST_FRONT_PORT`, with explicit note that prod TLS is external).
+- Documented the `2xs/3xs/4xs` font-size aliases in `frontend/tailwind.config.ts` against the design.md §3 scale.