mimic-big/CHANGELOG.md

Changelog

All notable changes to Mimic. Format inspired by Keep a Changelog (https://keepachangelog.com). Versioning starts at 0.1.0 when sprint 0 lands.

[Unreleased]

Team decisions (2026-05-21)

• Q1 — SOC client collaboration in the live cockpit is assumed valid (no PoC sheet).
• Q2 — Mimic is deployed on RT infrastructure (not at client). SOC client connects over the internet through the existing RT reverse proxy (out of Mimic scope).
• Q3 — Project framed as "improve the existing shared sheet workflow", not "rebuild Caldera".
• T2 — C2 credentials stored in a dedicated c2_credential table with version + retirement (Fernet-encrypted config_json). Active row per engagement = retired_at IS NULL, max version.
• T3 — Jinja templating exposes two accessors: {{outputs.text}} (stdout) and {{outputs.blob("key")}} (binary, 10 MB cap, UTF-8 with latin-1 fallback).
• T4 — soc_session.token_opaque stores a bcrypt hash; the clear token is delivered out-of-band and never re-displayable.
• Auth — v1: local user/password (bcrypt + Flask session). v2: Keycloak OIDC mapping onto the same group model. RBAC is group-based from day one.

Sprint 0 in progress

Repo skeleton, data model, C2Connector ABC, Jinja2 sandbox, local auth + RBAC, flat CRUD, UX wireframes (mock data). No real connector, no reporting until PR1/PR2/PR3 land.

Backend skeleton (feature/backend-skeleton)

• backend/ Python 3.12+ project: pyproject.toml (ruff, mypy strict, pytest, coverage 70 %), Makefile (Docker/Podman auto-detect), multi-stage Dockerfile, docker-compose.yml for Postgres dev DB, .env.example.
• Full §8 data model in SQLAlchemy 2 typed mapped classes: engagement, c2_credential, host, user, group, permission, group_permission, user_group, engagement_member, ttp, scenario, scenario_step, run, run_step, run_step_cleanup, detection, evidence, report, soc_session, audit_log. No ttp_version table (D-009 / H32 reaffirmed).
• Alembic baseline migration 202605210001_initial_schema: every table + enum + index + idempotent audit_log grants for the write-only Postgres role. Seeds the three F11 groups (rt_operator, rt_lead, soc_analyst) and their permission set (D-008).
• C2Connector ABC + Payload / TaskHandle / TaskResult / TaskStatus dataclasses + PayloadType enum + ConnectorFactory keyed on c2_type. Mythic payload map populated; Home stays empty until PR2.
• Jinja2 SandboxedEnvironment + regex_extract filter (google-re2 hard dependency per D-011 / B1 — RuntimeError at boot if absent, no re fallback) + {{ outputs.text }} / {{ outputs.blob() }} accessors reading gzip-compressed blobs (10 MB cap after decompression, UTF-8 → latin-1).
• Group-based RBAC: Permission + GroupName + GROUP_PERMISSIONS mirror the F11 matrix; @require_perm decorator + AuthUser Flask-Login wrapper that resolves the permission set from the user's groups.
• bcrypt password helpers + SOC opaque token (256-bit url-safe, bcrypt-hashed at rest, plain returned once).
• Hash-chained append-only audit writer (sprint 0 fills prev_hash / row_hash at insert; verifier shipped in v2).
• Flat CRUD blueprints: engagements / hosts / TTPs / scenarios + scenario steps. F3 invariant enforced (host.c2_type must match scenario.c2_type at compose time). Every mutation calls the hash-chained audit writer (MA5); created rows carry created_by_id (MA4); listings and per-engagement routes scope to engagement_member for RT operators (MA6 / F11).
• Content-addressed gzip blob store (mimic.storage.blob): streaming write with a max_bytes cap (raises BlobTooLarge mid-stream — MA2), atomic rename, 0o750 directory mode.
• mimic-cli (click): user create, db dump, db restore.
• pytest baseline: 56 unit tests passing (templating, regex_extract, password, soc_token, RBAC matrix, connector factory, audit hash, blob CAS, migration seed parity). Integration scaffold ready for testcontainers Postgres (/healthz smoke included).

Spec deltas applied in this sprint

Authoritative decisions implemented per tasks/spec-decisions.md:

• D-008 — Seeded groups = exactly the three F11 roles, permission matrix from F11.
• D-009 — No ttp_version table (H32 reaffirmed).
• D-011 — regex_extract fails loudly on no-match (raises TemplateError).
• D-012 — output_blob_ref stored in MIMIC_BLOB_ROOT (CAS gzip layout); evidence files live under MIMIC_EVIDENCE_ROOT (flat per-engagement).

Implementation arbitrations logged in this sprint:

• D-013 — audit_log hash chain (prev_hash / row_hash) shipped v1.
• D-014 — UUID columns use SQLAlchemy 2 native Uuid mapping; no type_annotation_map on the declarative base (Flask-SQLAlchemy incompatibility).

Code-review remediation (12d131c → feature/backend-skeleton)

• B1 — Dropped the re stdlib fallback in regex_extract. google-re2 is now a hard dependency (B1 / D-011); the module raises RuntimeError at import if absent.
• MA1 — Removed scripts/postgres-init/00-roles.sql (no more hardcoded CHANGE_ME password). Audit-writer role provisioning is the playbook's responsibility (D-010); backend/README.md documents the manual dev-only CREATE ROLE command.
• MA2 — store_blob now accepts a binary stream + max_bytes, streams sha256+gzip in 64 KB chunks, and raises BlobTooLarge mid-stream (cleans up the temp file). No more whole-buffer RAM load.
• MA3 — Inlined the F11 permission matrix in the initial Alembic migration; the runtime matrix is no longer imported there. A new unit test (test_migration_seed_matches_current_matrix) fails if the two drift apart.
• MA4 — created_by_id = current_user.id set in engagement, ttp, and scenario create endpoints.
• MA5 — Every mutation endpoint now writes an audit row through the hash-chained AuditWriter (F13).
• MA6 — RT operators only see engagements they are members of (engagement_member join on list, membership probe on get/put/delete/host/scenario/...). RT leads bypass.
• N4 — gunicorn declared in pyproject.toml dependencies (the Dockerfile CMD now resolves correctly).
• N6 — tests/integration/conftest.py keeps db.create_all() for now; commented TODO to switch over to Alembic once the playbook owns the audit role.
• M8 — Initial migration docstring no longer mentions ttp_version.

Verification on the latest commit: ruff check, ruff format --check, mypy --strict, and pytest tests/unit all pass; 56 unit tests green.