mimic-big/CHANGELOG.md

Changelog

All notable changes to Mimic. Format inspired by Keep a Changelog (https://keepachangelog.com). Versioning starts at 0.1.0 when sprint 0 lands.

[Unreleased]

Sprint 2 — user mgmt + engagement members + audit viewer (feature/backend-user-mgmt)

• USER_MANAGE permission (D-015) added to the F11 matrix; rt_lead only. Migration 202605230001_add_user_manage_permission adds user.manage to the permission table and ties it to the rt_lead group. The test_migration_seed_matches_current_matrix invariant is generalised to the union "initial frozen ∪ delta migrations" so future sprints can keep adding permissions via new migrations without editing the historical one.
• User CRUD (/api/v1/users):
  • GET paginated list (filter ?type=).
  • POST creates a user, hashes the password, wires the F11 group membership automatically, returns 409 email_taken on duplicate.
  • PATCH partial update; changing type realigns the global group membership and leaves per-engagement memberships untouched.
  • DELETE soft-disables via disabled_at; idempotent (returns 204 even when already disabled).
  • Every mutation writes an audit row (user.create / update / disable).
• Engagement members (/api/v1/engagements/<eid>/members):
  • GET, POST, DELETE. _engagement_or_404 runs before any membership query so an RT operator targeting a foreign engagement receives the same 404 as for a non-existent id (anti-enumeration).
  • role is a free-form ≤40-char label (D-017). Default "member".
  • 409 already_member on duplicate.
• Audit log viewer (/api/v1/audit/log): paginated, rt_lead only via AUDIT_READ. Filters: action, actor_id, resource_type, since, until (ISO 8601). Exposes prev_hash / row_hash so future clients can verify the chain.
• Pagination envelope (D-016): Page[T] schema {items, total, page, page_size} and PageQuery for parsing ?page=&page_size= (max 200). Used by /users and /audit/log this sprint; existing flat-array endpoints stay unchanged.
• Spec decisions D-015, D-016, D-017 logged.
• Tests: 11 new unit tests (Pydantic shapes + pagination bounds) + 5 new integration tests covering the critical MA6 scenario (rt_lead creates rt_operator → assigns engagement A → operator only sees A), the RBAC gate on USER_MANAGE, the 409 on duplicate emails, the audit pagination, and the soft-disable login-block path.
• docs/api.md extended with the sprint-2 surface; the typo /engagements<eid> → /engagements/<eid> fixed in passing.

Sprint 1 — backend follow-up fixes (feature/backend-auth-wiring)

• Global JSON error envelope — register @app.errorhandler(HTTPException) so every aborted request now flows through the same { "error": "<code>", "message": "<human>", "details"? } JSON shape that docs/api.md documented but only api_error() honoured before. 422 responses surface the Pydantic per-field list under details so the frontend can map errors back to form fields. New stable codes: bad_request, not_found, method_not_allowed, validation_error, forbidden, internal_error, etc. (see updated docs/api.md).
• strict_slashes=False on the URL map — /api/v1/engagements and /api/v1/engagements/ match the same handler. Removes the 308 redirect that some browsers drop the session cookie through.
• 5 new integration tests covering both slash variants, 422 validation_error envelope shape (incl. details), unknown-route 404, and 400 on a non-JSON body.
• docs/api.md rewritten: full error code table, 422 details example, trailing-slash policy, dropped trailing slashes from all endpoint headings.

Sprint 1 — backend auth wiring (feature/backend-auth-wiring)

• POST /api/v1/auth/login — local-credentials login. Body {username, password}; success returns the CurrentUser shape (user_id, username, display_name, role, permissions, groups) and sets a Flask session cookie. Failures return a uniform 401 invalid_credentials envelope; a bcrypt round runs against a dummy hash on unknown users to flatten the timing signal.
• POST /api/v1/auth/logout — clears the session, returns 204. Writes an auth.logout audit row.
• GET /api/v1/auth/me — rehydrates the frontend at boot; returns the current principal or 401 not_authenticated.
• Error envelope — every API failure now returns {error: "<code>", message: "<human>"}. LoginManager.unauthorized_handler is wired to the same shape so @login_required 401s match.
• Dev-only CORS — flask-cors wraps /api/* for the origins in MIMIC_CORS_ORIGINS only when MIMIC_ENV=development. Prod keeps same-origin via the reverse proxy.
• AuthUser extended — carries display_name + user_type so the serialiser can return them.
• Audit — auth.login and auth.logout rows go through the existing hash-chained writer.
• Docs — docs/api.md describes the contract the frontend consumes (login flow, CurrentUser shape, error envelope, MA6 tenant-scope behaviour).
• Tests — 5 unit tests on the schemas + serializer; integration scaffold test tests/integration/test_auth_engagement_e2e.py exercises the full login → /me → POST engagement → list → logout loop on a testcontainers Postgres.

Team decisions (2026-05-21)

• Q1 — SOC client collaboration in the live cockpit is assumed valid (no PoC sheet).
• Q2 — Mimic is deployed on RT infrastructure (not at client). SOC client connects over the internet through the existing RT reverse proxy (out of Mimic scope).
• Q3 — Project framed as "improve the existing shared sheet workflow", not "rebuild Caldera".
• T2 — C2 credentials stored in a dedicated c2_credential table with version + retirement (Fernet-encrypted config_json). Active row per engagement = retired_at IS NULL, max version.
• T3 — Jinja templating exposes two accessors: {{outputs.text}} (stdout) and {{outputs.blob("key")}} (binary, 10 MB cap, UTF-8 with latin-1 fallback).
• T4 — soc_session.token_opaque stores a bcrypt hash; the clear token is delivered out-of-band and never re-displayable.
• Auth — v1: local user/password (bcrypt + Flask session). v2: Keycloak OIDC mapping onto the same group model. RBAC is group-based from day one.

Sprint 0 in progress

Repo skeleton, data model, C2Connector ABC, Jinja2 sandbox, local auth + RBAC, flat CRUD, UX wireframes (mock data). No real connector, no reporting until PR1/PR2/PR3 land.

Backend skeleton (feature/backend-skeleton)

• backend/ Python 3.12+ project: pyproject.toml (ruff, mypy strict, pytest, coverage 70 %), Makefile (Docker/Podman auto-detect), multi-stage Dockerfile, compose.yml for Postgres dev DB, .env.example.
• Full §8 data model in SQLAlchemy 2 typed mapped classes: engagement, c2_credential, host, user, group, permission, group_permission, user_group, engagement_member, ttp, scenario, scenario_step, run, run_step, run_step_cleanup, detection, evidence, report, soc_session, audit_log. No ttp_version table (D-009 / H32 reaffirmed).
• Alembic baseline migration 202605210001_initial_schema: every table + enum + index + idempotent audit_log grants for the write-only Postgres role. Seeds the three F11 groups (rt_operator, rt_lead, soc_analyst) and their permission set (D-008).
• C2Connector ABC + Payload / TaskHandle / TaskResult / TaskStatus dataclasses + PayloadType enum + ConnectorFactory keyed on c2_type. Mythic payload map populated; Home stays empty until PR2.
• Jinja2 SandboxedEnvironment + regex_extract filter (google-re2 hard dependency per D-011 / B1 — RuntimeError at boot if absent, no re fallback) + {{ outputs.text }} / {{ outputs.blob() }} accessors reading gzip-compressed blobs (10 MB cap after decompression, UTF-8 → latin-1).
• Group-based RBAC: Permission + GroupName + GROUP_PERMISSIONS mirror the F11 matrix; @require_perm decorator + AuthUser Flask-Login wrapper that resolves the permission set from the user's groups.
• bcrypt password helpers + SOC opaque token (256-bit url-safe, bcrypt-hashed at rest, plain returned once).
• Hash-chained append-only audit writer (sprint 0 fills prev_hash / row_hash at insert; verifier shipped in v2).
• Flat CRUD blueprints: engagements / hosts / TTPs / scenarios + scenario steps. F3 invariant enforced (host.c2_type must match scenario.c2_type at compose time). Every mutation calls the hash-chained audit writer (MA5); created rows carry created_by_id (MA4); listings and per-engagement routes scope to engagement_member for RT operators (MA6 / F11).
• Content-addressed gzip blob store (mimic.storage.blob): streaming write with a max_bytes cap (raises BlobTooLarge mid-stream — MA2), atomic rename, 0o750 directory mode.
• mimic-cli (click): user create, db dump, db restore.
• pytest baseline: 56 unit tests passing (templating, regex_extract, password, soc_token, RBAC matrix, connector factory, audit hash, blob CAS, migration seed parity). Integration scaffold ready for testcontainers Postgres (/healthz smoke included).

Spec deltas applied in this sprint

Authoritative decisions implemented per tasks/spec-decisions.md:

• D-008 — Seeded groups = exactly the three F11 roles, permission matrix from F11.
• D-009 — No ttp_version table (H32 reaffirmed).
• D-011 — regex_extract fails loudly on no-match (raises TemplateError).
• D-012 — output_blob_ref stored in MIMIC_BLOB_ROOT (CAS gzip layout); evidence files live under MIMIC_EVIDENCE_ROOT (flat per-engagement).

Implementation arbitrations logged in this sprint:

• D-013 — audit_log hash chain (prev_hash / row_hash) shipped v1.
• D-014 — UUID columns use SQLAlchemy 2 native Uuid mapping; no type_annotation_map on the declarative base (Flask-SQLAlchemy incompatibility).

Code-review remediation (12d131c → feature/backend-skeleton)

• B1 — Dropped the re stdlib fallback in regex_extract. google-re2 is now a hard dependency (B1 / D-011); the module raises RuntimeError at import if absent.
• MA1 — Removed scripts/postgres-init/00-roles.sql (no more hardcoded CHANGE_ME password). Audit-writer role provisioning is the playbook's responsibility (D-010); backend/README.md documents the manual dev-only CREATE ROLE command.
• MA2 — store_blob now accepts a binary stream + max_bytes, streams sha256+gzip in 64 KB chunks, and raises BlobTooLarge mid-stream (cleans up the temp file). No more whole-buffer RAM load.
• MA3 — Inlined the F11 permission matrix in the initial Alembic migration; the runtime matrix is no longer imported there. A new unit test (test_migration_seed_matches_current_matrix) fails if the two drift apart.
• MA4 — created_by_id = current_user.id set in engagement, ttp, and scenario create endpoints.
• MA5 — Every mutation endpoint now writes an audit row through the hash-chained AuditWriter (F13).
• MA6 — RT operators only see engagements they are members of (engagement_member join on list, membership probe on get/put/delete/host/scenario/...). RT leads bypass.
• N4 — gunicorn declared in pyproject.toml dependencies (the Dockerfile CMD now resolves correctly).
• N6 — tests/integration/conftest.py keeps db.create_all() for now; commented TODO to switch over to Alembic once the playbook owns the audit role.
• M8 — Initial migration docstring no longer mentions ttp_version.

Verification on the latest commit: ruff check, ruff format --check, mypy --strict, and pytest tests/unit all pass; 56 unit tests green.