Metamorph/CHANGELOG.md

# Changelog

All notable changes to this project will be documented here. Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) · [Conventional Commits](https://www.conventionalcommits.org/).

## [Unreleased]

### Added — M4 (MITRE ATT&CK Enterprise)
- **STIX 2.1 parser + upsert** (`app/services/mitre_seed.py`): stdlib-only (`urllib.request` + `hashlib`), pinned to Enterprise v19.0 (`enterprise-attack-19.0.json`, sha256 `df520ea0…`). Parses 25k+ STIX objects → 15 tactics, 222 techniques, 475 sub-techniques in ~1.1 s. Skips revoked + deprecated, resolves sub-technique parents via `relationship[subtechnique-of]` with a `T1003.001 → T1003` dotted-id fallback, copies kill-chain phases into the `mitre_technique_tactics` M2M.
- **CLI**: `flask metamorph seed-mitre [--source <path|url>] [--checksum-sha256 <hex>] [--skip-checksum]` (`app/cli.py`). `make seed-mitre` wraps it.
- **REST endpoints** (`app/api/mitre.py`):
  - `GET /api/v1/mitre/tactics`, `/mitre/techniques?tactic=…&q=…`, `/mitre/subtechniques?technique=…&q=…` (paginated, search on name/external_id).
  - `GET /api/v1/mitre/status` (last_sync, version, source_url, defaults).
  - `POST /api/v1/mitre/sync` (perm `mitre.sync`) — re-pull on demand.
- **Persisted metadata** in `settings`: `mitre_last_sync`, `mitre_version`, `mitre_source_url`.
- **Compose volume `metamorph_mitre`** mounted at `/data/mitre/` in the api container — caches the downloaded bundle across restarts. Owned by `metamorph:metamorph`.
- **Frontend**:
  - `<MitreTagPicker>` component: 3-column tactic → technique → sub-technique with multi-select chips, autocomplete on each column. Returns `MitreTag[]` (`kind`, `id`, `external_id`, `name`), ready for M5 templates.
  - `/mitre` showcase page with status card, admin-gated **Trigger sync** button, picker preview, and `<pre>` payload preview.
  - Nav adds **MITRE** link for any logged-in user.
- **Testing**:
  - `backend/tests/test_mitre.py` — **12 pytest** (parser, idempotence, checksum mismatch, persisted settings, endpoint variants, perm enforcement) using a hand-crafted minimal STIX bundle (no network in tests).
  - `e2e/tests/m4-mitre.spec.ts` — **6 Playwright** against the live stack (calls `/mitre/sync` once in `beforeAll`).
  - `tasks/testing-m4.md`.

### Fixed (post-M4 spec-review pass)
- **Sync integrity guarantee**: `seed_mitre()` now refuses a custom URL without either `expected_sha256` or an explicit `allow_unverified=true`. Closes a "typo in `mitre_source_url` setting routes the seed to attacker JSON" footgun. CLI surfaces this via `--checksum-sha256` / `--skip-checksum`; API via `{"source", "expected_sha256", "allow_unverified"}` body.
- **`/diag/reset` consistency**: now truncates the `mitre_*` tables alongside `settings` so `GET /mitre/status` and `GET /mitre/tactics` agree after a reset (previously: catalogue rows persisted, but `mitre_last_sync` got wiped → status lied).
- **Spec drift §10 #4**: amended "14 tactics" → "≥ 14 tactics (v19 ships 15)" to reflect MITRE v8+ reality.

### Decisions (intentional)
- **Bundle "embarqué" interpreted as seed-time download + named-volume cache**, not "binary baked into the Docker image". Keeps the image ~150 MB, makes version bumps a constant edit, plays nicely with `make seed-mitre` re-runs. Air-gapped operators copy the file into the volume + pass `--source /data/mitre/<file>`.
- **Read endpoints unauthenticated-perm-wise but auth-required**: MITRE data is public reference material — no `mitre.read` perm. Status endpoint is similarly open (under `@require_auth`) to keep `/mitre/status` simple for the UI badge.
- **No `requests` / `httpx` dep added**: stdlib `urllib.request` is enough and avoids inflating the image.

### Validated end-to-end (M4 DoD)
- `make clean && make up && make migrate && make seed-mitre` → 15 tactics / 222 techniques / 475 sub-techniques / 254 links / 0 orphans / ~1.1 s.
- `make test-api` → **51 pytest pass** (1 health + 8 schema + 15 auth + 15 RBAC + 12 MITRE) in ~5 s.
- `make e2e` → **34 Playwright pass** (8 M0 + 4 M1 + 8 M2 + 8 M3 + 6 M4) in ~18 s.
- Spec-reviewer PASS after fixes applied.

### Added — M3 (RBAC: groups, permissions, users)
- **Permission catalogue** (`app/services/permissions_seed.py`): 31 atomic codes across 10 families (`user`, `group`, `invitation`, `test_template`, `scenario_template`, `mission`, `detection_level`, `setting`, `mitre.sync`). Seeded at boot **and** after `/setup` to handle a freshly truncated DB. Idempotent + additive on system groups (never removes a perm).
- **Default group bindings**: `admin` = all 31 codes; `redteam` = 8 (catalogue read + mission.{read,create,update,archive,write_red_fields} + detection_level.read); `blueteam` = 5 (catalogue read + mission.{read,write_blue_fields} + detection_level.read).
- **Users admin service + API** (`app/services/users.py`, `app/api/users.py`): list (q + is_active filter + pagination), get, patch (display_name/locale/is_active), soft-delete, set groups. Last-admin protection on update/delete/group-strip.
- **Groups admin service + API** (`app/services/groups.py`, `app/api/groups.py`): full CRUD with system-group protection (no rename, no delete), `PUT /groups/{id}/permissions` for the bindings. Admin system group's perm set is locked to "every perm" (preserves the bypass invariant).
- **Permissions read-only API** (`app/api/permissions.py`): `GET /permissions` returns the catalogue (admin or `group.read` holders).
- **Frontend admin pages** (`frontend/src/pages/Admin{Users,Groups,Invitations}Page.tsx`): list + edit modals using TanStack Query mutations, multi-select for perms grouped by family, copy-once invitation URL display.
- **Frontend chrome** (`Layout.tsx` + `RequireAdmin.tsx`): admin nav links shown only when `is_admin === true`; direct navigation to `/admin/*` by non-admins redirects to `/`. Server remains the arbiter.
- **`/diag/reset` now clears the rate-limit counters** so the Playwright suite can iterate without hitting `10/min/IP` budgets across spec files. Gated to non-prod environments only.
- **Testing**:
  - `tests/test_rbac.py` — **15 pytest integration tests** (39 backend total).
  - `e2e/tests/m3-rbac.spec.ts` — **8 Playwright tests** covering DoD §10 #2/#3 (28 e2e total).
  - `tasks/testing-m3.md` — manual + automated procedure.
- **Frontend api helpers**: `apiPatch`, `apiPut`, `apiDelete` added to `frontend/src/lib/api.ts`.

### Fixed (post-M3 spec-review pass)
- **Rate-limit scope clarified**: `app/core/rate_limit.py` now enables the limiter for `APP_ENV in ("prod", "staging")` instead of `prod` only — a public staging deployment without auth limits would be surprising. Dev/test stay unthrottled for Playwright ergonomics. Spec §6 NF-security applies to operator-facing deployments.
- **Admin perm invariant**: `set_group_permissions` refuses to alter the admin system group's perm set to anything other than the full catalogue (`SystemGroupProtected` → 409). The decorator bypass relies on `is_admin = "admin" in group_names`, but a future refactor could move to a perm-based check, so we keep the invariant.
- **LogRecord field collision**: `log.info("...", extra={"name": g.name})` raised `KeyError: "Attempt to overwrite 'name' in LogRecord"` because Python's logger reserves `name`. Renamed to `group_name`. Audited all other `extra=` payloads in `app/api/`+`app/services/` for the same trap.

### Validated end-to-end (M3 DoD)
- `make clean && make up && make migrate` → boot logs show `metamorph.permissions.seeded {perms_created: 31, perms_total: 31, bindings: {admin: 31, redteam: 8, blueteam: 5}}`.
- `make test-api` → **39 pytest pass** (1 health + 8 schema + 15 auth + 15 RBAC) in ~4 s.
- `make e2e` → **28 Playwright pass** (8 M0 + 4 M1 + 8 M2 + 8 M3) in ~16 s.
- Spec-reviewer pass: PASS verdict, 2 minor fixes applied (above), 2 anticipations noted for M12/M14 (no current action).

### Added — M2 (Auth, bootstrap, invitations)
- **Crypto plumbing**: `app.core.security` (Argon2id `time_cost=2 memory_cost=64MiB parallelism=2`, opaque-token SHA-256 helpers), `app.core.jwt_tokens` (HS256, claims `iss/sub/type/jti/iat/exp`, access 1h / refresh 30d).
- **Auth services** (`app.services.auth`): login, refresh **with token rotation + reuse-detection chain revoke**, logout (idempotent), change_password (forces logout-all).
- **Invitation services** (`app.services.invitations`): create, preview, accept, revoke. Token persisted only as SHA-256, default 7-day TTL.
- **Bootstrap** (`app.services.bootstrap` + `app.core.install_token`): seeds 3 system groups (`admin`/`redteam`/`blueteam`), mints a one-shot install token at first boot when `users` is empty, logs a banner with the raw token. CLI `flask --app app.cli metamorph print-install-token [--force]`.
- **Auth middleware** (`app.core.auth_decorators`): `@require_auth` populates `g.current_user`; `@require_perm("...")` checks atomic permissions; admin group bypasses the check (atomic perms land in M3).
- **API endpoints**:
  - `POST /api/v1/setup` (consume install token, create 1st admin) + `GET /api/v1/setup` (status).
  - `POST /api/v1/auth/login` + `POST /auth/refresh` + `POST /auth/logout` + `GET /auth/me` + `POST /auth/change-password`.
  - `POST /api/v1/invitations` (admin) + `GET /invitations` + `GET /invitations/preview/<token>` + `POST /invitations/accept/<token>` + `POST /invitations/<id>/revoke`.
  - `POST /api/v1/diag/reset` (test-only kill switch — wipes auth tables + mints fresh install token; only available in `dev`/`test`).
- **Rate limiting** (`flask-limiter`): 10/min/IP on `/auth/login`, `/auth/refresh`; 5/min on `/auth/change-password` and `/setup`; 10–20/min on invitation endpoints. Globally disabled when `APP_ENV=test`.
- **Refresh cookie** `metamorph_refresh`: HttpOnly + Secure + SameSite=Strict + Path=`/api/v1/auth/`.
- **Frontend auth state** (`frontend/src/lib/{api,auth}.ts`): access token in module memory, refresh in cookie, automatic 401-retry via `/auth/refresh` with reentrancy guard. `useAuth()` hook + `<RequireAuth>` route guard.
- **Frontend pages**: `/login`, `/setup`, `/register?token=…`, `/profile` (with change-password form), all in RTOps design. Protected layout: nav shows email + Logout when authenticated, Login + Setup links when not.
- **Frontend deps**: `@tanstack/react-query`, `react-router-dom`. Tanstack provider in `App.tsx` (will carry actual queries from M3+).
- **Email validation** (`app.api._validation.Email`): permissive RFC-shape regex that accepts internal TLDs (`.local`, `.corp`) — `pydantic.EmailStr` was too strict for red-team labs.
- **Testing**:
  - `tests/test_auth_flow.py` — **15 pytest integration tests** (24 backend total with M0/M1).
  - `e2e/tests/m2-auth.spec.ts` — **8 Playwright tests** covering setup → login → me → invitation → register → 2nd login → RBAC 403 → refresh rotation → logout (20 e2e total).
  - `tasks/testing-m2.md` — manual + automated procedure.

### Fixed (post-M2 spec-review pass)
- **Refresh cookie `Secure=True`** unconditionally (`backend/app/api/auth.py`). Modern browsers treat `localhost` as a secure context, so dev/test still works. Closes the silent-degradation found by the reviewer.
- **`/auth/refresh` rate-limit lowered to 10/min/IP** (`backend/app/api/auth.py`) to match spec §M2 ("10 req/min/IP on `/auth/*`").
- `/diag/reset` kept allowed in `dev` *and* `test` (a `make e2e` against a `make up` dev stack must be able to reset). Added a WARNING log when triggered in `dev` and a clear docstring; production envs (`prod`/`staging`) remain locked out.

### Known scope-creep (intentional, not retracted)
- Rate-limits on `/setup` (5/min), `/invitations/preview` (20/min), `/invitations/accept` (10/min) and `/auth/change-password` (5/min) were added in M2 even though §M2 only mandated `/auth/*`. Defensible (these are abuse-attractor endpoints), and noted here so M14 doesn't double-spec them.

### Added — M1 (DB schema & migrations)
- **23 tables** + `alembic_version` covering auth/RBAC (8), MITRE (4), templates (4), missions (6), evidence (1), settings/detection-levels (2), notifications (1).
- SQLAlchemy 2.x declarative models with Mapped[]/mapped_column(), grouped under `backend/app/models/{auth,mitre,template,mission,evidence,setting,notification}.py`.
- Alembic init: `alembic.ini`, `alembic/env.py` reading `app.core.config.settings.database_url`, `alembic/script.py.mako`, naming convention `pk_/fk_/ck_/uq_/ix_` enforced via `MetaData(naming_convention=...)` on `app.db.base.Base`.
- Reusable mixins in `app.db.mixins`: `UuidPkMixin` (uuid4 server-side), `TimestampMixin` (created_at/updated_at, server-default + onupdate), `SoftDeleteMixin` (deleted_at, no auto-injected index — declared explicitly per table to avoid mixin-vs-class `__table_args__` clobbering).
- Postgres-specific features used: `JSONB` for `settings.value` and `notifications.payload`; native `Uuid` columns; partial indexes (`WHERE deleted_at IS NULL` on 9 tables; `WHERE read_at IS NULL` on `notifications`); CHECK constraints for status/state/opsec_level/mitre_kind enums; `exactly_one_mitre_fk` CHECK on `test_template_mitre_tags`.
- **`mission_test_mitre_tags` deliberately denormalised** (no FK to `mitre_*` tables): copies `mitre_external_id`, `mitre_name`, `mitre_url` at tag time so a later MITRE re-sync that drops an entry cannot purge a mission's tags. Companion `test_template_mitre_tags` keeps FKs since templates are editable. (Spec §11 risk addressed.)
- Backend `pyproject.toml` deps: SQLAlchemy ≥2, Alembic ≥1.13, psycopg[binary] ≥3.1.
- New Makefile targets: `migrate`, `migrate-down`, `migrate-revision MSG=…`, `migrate-status`. The Dockerfile now ships `alembic.ini` + `alembic/` so the api container can run migrations directly.
- **Test stage in `backend/Dockerfile`** (`--target test`): runtime image + dev extras + `tests/` dir. New `make test-api` target spins an ephemeral container against the live DB on the compose network. Backend tests no longer require any local Python toolchain.
- `tests/test_schema.py` (8 integration tests + the existing M0 health test = 9 total): expected tables, expected timestamp/soft-delete columns, partial-index presence, expected FK pairs, expected CHECK constraints, alembic-at-head, and a negative INSERT proving the `exactly_one_mitre_fk` CHECK fires.
- `tasks/testing-m1.md` — manual + automated verification procedure.

### Fixed (post-M1 spec-review pass)
- Soft delete now consistent across snapshot-bearing tables: `mission_scenarios`, `mission_tests`, `mission_categories` gained `SoftDeleteMixin` + their `ix_<table>_active` partial index (M12 trash bin depends on this).
- `evidence_files` gained `TimestampMixin` (`created_at`/`updated_at`) on top of the domain `uploaded_at` (audit minimal everywhere, per M1 brief).
- `mission_members` gained `TimestampMixin`, replacing the bespoke `added_at` column.
- `scenario_template_tests` PK refactored to a UUID + `UNIQUE(scenario_template_id, position)` so the same test can appear at multiple positions in a scenario (chained operations).
- `SoftDeleteMixin.__table_args__` removed (silently clobbered by class `__table_args__`); each soft-delete table now declares `ix_<table>_active` explicitly. Documented in the mixin's docstring.
- `mission_test_mitre_tags` schema redesigned to denormalise MITRE labels (see "Added" entry above).
- Migration 0001 regenerated end-to-end after these fixes — `24765a5014b6` is the new HEAD.

### Validated end-to-end (M1 DoD)
- `make clean && make up && make migrate` from a vide DB → 27 tables, 32 FK, 9 CHECK, 14 UQ, 12 partial indexes.
- `make test-api` → **9 pytest pass** (1 health + 8 schema integration) in <1 s.
- `make e2e` → **12 Playwright pass** (8 M0 smoke + 4 M1 db visibility) in 3 s.

### Added (M1 visibility)
- New API endpoint `GET /api/v1/diag/db` exposes `alembic_revision` (short-hashable) and the public-schema `table_count`. Returns 503 with `{"reachable": false}` when Postgres is down.
- New `Database` card on the SPA home page consumes that endpoint, renders the revision short-hash and the count next to the existing `API` and `Roadmap` cards.
- Footer updated to `M0 bootstrap · M1 db schema`. Roadmap card now points to `M2 — Auth + JWT`.
- New e2e suite `e2e/tests/m1-db.spec.ts` (4 tests) covers the diag endpoint contract, the Database card rendering, and the footer/roadmap labels.

### Added — M0 (bootstrap)
- Repo scaffolding: `.gitignore`, `.env.example`, `Makefile`, `docker-compose.yml`, `README.md`, `CHANGELOG.md`.
- `docker-compose.yml` with three services: `db` (postgres:16-alpine, no host port), `api` (Flask 3, port 8000), `front` (nginx serving the Vite bundle, port 80).
- Named volumes `metamorph_db` and `metamorph_evidence` for data persistence.
- Backend skeleton: Flask app factory, JSON structured logging on stdout, `GET /api/v1/health` endpoint, multi-stage Dockerfile, `pyproject.toml` driven by `uv`.
- Frontend skeleton: Vite + React 18 + TypeScript strict + TailwindCSS, RTOps design tokens (`tasks/design.md`) translated into `tailwind.config.ts`, base UI primitives (`Card`, `Tag`, `SectionHeader`, `FlowNode`, `Button`), home page wired to `/api/v1/health`.
- Multi-stage frontend Dockerfile that builds the bundle and serves it via nginx, proxying `/api/*` to the api container.
- Pre-commit hook config: `ruff` for backend, `eslint` + `tsc --noEmit` for frontend.

### Validated
- `docker compose config` parses (validated via `pyyaml` since Docker is not installed in the dev shell).
- Every env var referenced by the compose file is documented in `.env.example`.
- All Python source files parse cleanly (`ast.parse`).
- All TS/JSON config files parse cleanly.

### Notes
- TLS termination is delegated to an external reverse proxy (per spec §6 NF-network). The compose stack exposes plain HTTP on `HOST_FRONT_PORT` (8080) and `HOST_API_PORT` (8000).
- The first-admin bootstrap token (M2) will be printed to the api container's stdout on first boot when the `users` table is empty.
- `tasks/spec.md` and `tasks/todo.md` remain authoritative; update them before changing scope.

### Fixed (M0 DoD validation pass on real podman)
- **FQDN image references** in `docker-compose.yml`, `backend/Dockerfile`, `frontend/Dockerfile`. Podman on Fedora enforces `short-name-mode=enforcing` for pulls (no TTY ⇒ no prompt ⇒ failure). Replaced `postgres:16-alpine` / `python:3.12-slim` / `node:20-alpine` / `nginx:1.27-alpine` with their `docker.io/library/…` qualified equivalents. Docker accepts the same prefix transparently.
- **`*.md` removed from `backend/.dockerignore` and `frontend/.dockerignore`**: `pyproject.toml` declared `readme = "README.md"`, but the file was being filtered out of the build context, so `hatchling.build.build_wheel` raised `OSError: Readme file does not exist: README.md`. Also removed the `readme` field itself from `pyproject.toml` to decouple the build from the doc.
- **`Card.tsx` type clash**: `CardProps extends HTMLAttributes<HTMLDivElement>` redefined `title` as `ReactNode`, but the native `title` is `string`. `tsc -b` failed with TS2430 during `vite build`. Switched to `Omit<HTMLAttributes<HTMLDivElement>, 'title'>`.
- **Explicit healthchecks added to compose `api` and `front`**: podman-compose 1.x doesn't surface healthchecks declared only in the `Dockerfile` via `inspect`. Mirroring them in `docker-compose.yml` makes `make inspect-health` actually see `healthy/unhealthy/starting` on every engine.
- **Suppressed `podman compose` external-provider banner** via `PODMAN_COMPOSE_WARNING_LOGS=false` exported from the Makefile.

### Validated end-to-end on podman 5.x (Fedora 43)
- `make up` → 3 containers, all 3 healthy after start_period.
- `make health` → `{"status":"ok","version":"0.1.0"}` via the front nginx proxy (port 8080) and direct API (port 8000).
- `make logs-api` → JSON-structured lines on stdout (`ts`, `level`, `logger`, `message`, custom fields).
- `make e2e` → **8/8 Playwright tests pass** in 2.5 s. Reports: `e2e/playwright-report/index.html` (529 KB, autoportant) + `junit.xml` (`tests=8 failures=0 skipped=0 errors=0`).

### Added (engine portability)
- Makefile auto-detects **docker** or **podman** at runtime and selects the matching compose driver (`docker compose`, `podman compose`, or legacy `podman-compose`). Override via `ENGINE=…` and/or `COMPOSE="…"`.
- New targets: `engine` (print detected runtime), `volumes` (list project-named volumes), `inspect-health` (health status of all 3 containers), `logs-api` (tail just the api), `health` (single curl probe). All engine-agnostic.
- `make help` now prints the active engine + compose driver in its footer.
- `tasks/testing-m0.md` and `README.md` rewritten to be engine-agnostic — raw `docker logs` / `docker volume ls` / `docker inspect` calls replaced with the new make targets.

### Added (M0 testing)
- `e2e/` Playwright project with chromium, HTML + JUnit XML reporters, traces / screenshots / videos kept on retry. Reports land in `e2e/playwright-report/`.
- `e2e/tests/m0-smoke.spec.ts` — 8 smoke tests covering the front rendering, the API proxy, the design tokens, the absence of any runtime CDN traffic (spec §7), and the CORS contract.
- Makefile targets `e2e-install`, `e2e`, `e2e-report`, `e2e-up`, `wait-healthy`.
- `tasks/testing-m0.md` — step-by-step manual + automated verification procedure for M0.
- Convention added to `tasks/todo.md`: every milestone N delivers `tasks/testing-m<N>.md` + at least one `e2e/tests/m<N>-*.spec.ts`, and the spec-reviewer subagent runs before marking the milestone done.

### Fixed (post-M0 spec-review pass)
- `.pre-commit-config.yaml` added at repo root: ruff + ruff-format on backend, eslint + tsc --noEmit + prettier --check on frontend, plus baseline whitespace/JSON/private-key checks. Documented `pre-commit install` in `README.md`.
- Self-hosted webfonts via `@fontsource/jetbrains-mono` and `@fontsource/ibm-plex-sans` (imported in `frontend/src/index.css`); dropped the Google Fonts `<link>` from `frontend/index.html` to honor spec §7 ("no runtime CDN").
- Refuse-to-boot guard in `backend/app/core/config.py`: when `APP_ENV != "dev"`, defaults / placeholders for `JWT_SECRET` and `POSTGRES_PASSWORD` raise at startup. New `APP_ENV` env var documented in `.env.example`, `README.md`, and `docker-compose.yml`.
- `make dev` now runs `dev-api` and `dev-front` in parallel via `make -j2` instead of just printing a hint.
- Removed dead `database_url` property from `Settings` (will be reintroduced in M1 with the SQLAlchemy/Alembic stack).
- Pinned Node engines to `>=20` in `frontend/package.json`.
- Reconciled M0 DoD wording in `tasks/todo.md` (HTTP via `HOST_FRONT_PORT`, with explicit note that prod TLS is external).
- Documented the `2xs/3xs/4xs` font-size aliases in `frontend/tailwind.config.ts` against the design.md §3 scale.