2026-05-21 20:10:47 +02:00
|
|
|
|
# Changelog
|
|
|
|
|
|
|
chore: tighten gitignore, align README stack, formalize D-010 (Ansible)
- .gitignore: add Keycloak/Mythic/Fernet secret patterns (pfx, p12, token, kdbx,
credentials.json, secrets.json, service-account*.json), MSVC artifacts
(lib, exp, idb, ilk, tlog), dedup dist/build/ between Python and Node blocks.
- README.md: align Storage line on H38 (testcontainers Postgres for Postgres-
specific behavior, incl. unit tests of audit log / RBAC / write-only role).
- README.md: align Deploy line on D-007/D-010 — Docker + Ansible playbook,
reverse proxy explicitly out-of-Mimic.
- README.md: add proprietary internal use notice.
- CHANGELOG.md: convert markdown link to inline URL (no dangling reference).
- tasks/spec-decisions.md: add D-010 (Ansible for deployment playbook).
Addresses code-reviewer M1/M2/M3 + N2/N3/N4/N6 on commit 047583e.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 20:16:40 +02:00
|
|
|
|
All notable changes to Mimic. Format inspired by Keep a Changelog (https://keepachangelog.com).
|
2026-05-21 20:10:47 +02:00
|
|
|
|
Versioning starts at `0.1.0` when sprint 0 lands.
|
|
|
|
|
|
|
|
|
|
|
|
## [Unreleased]
|
|
|
|
|
|
|
2026-05-23 15:53:45 +02:00
|
|
|
|
### Sprint 2 — user mgmt + engagement members + audit viewer (`feature/backend-user-mgmt`)
|
|
|
|
|
|
|
|
|
|
|
|
- **`USER_MANAGE` permission** (D-015) added to the F11 matrix; `rt_lead` only.
|
|
|
|
|
|
Migration `202605230001_add_user_manage_permission` adds `user.manage` to
|
|
|
|
|
|
the `permission` table and ties it to the `rt_lead` group. The
|
|
|
|
|
|
`test_migration_seed_matches_current_matrix` invariant is generalised to
|
|
|
|
|
|
the union "initial frozen ∪ delta migrations" so future sprints can keep
|
|
|
|
|
|
adding permissions via new migrations without editing the historical one.
|
|
|
|
|
|
- **User CRUD** (`/api/v1/users`):
|
|
|
|
|
|
- `GET` paginated list (filter `?type=`).
|
|
|
|
|
|
- `POST` creates a user, hashes the password, wires the F11 group membership
|
|
|
|
|
|
automatically, returns `409 email_taken` on duplicate.
|
|
|
|
|
|
- `PATCH` partial update; changing `type` realigns the global group
|
|
|
|
|
|
membership and leaves per-engagement memberships untouched.
|
|
|
|
|
|
- `DELETE` soft-disables via `disabled_at`; idempotent (returns 204 even
|
|
|
|
|
|
when already disabled).
|
|
|
|
|
|
- Every mutation writes an audit row (`user.create` / `update` / `disable`).
|
|
|
|
|
|
- **Engagement members** (`/api/v1/engagements/<eid>/members`):
|
|
|
|
|
|
- `GET`, `POST`, `DELETE`. `_engagement_or_404` runs *before* any membership
|
|
|
|
|
|
query so an RT operator targeting a foreign engagement receives the same
|
|
|
|
|
|
404 as for a non-existent id (anti-enumeration).
|
|
|
|
|
|
- `role` is a free-form ≤40-char label (D-017). Default `"member"`.
|
|
|
|
|
|
- `409 already_member` on duplicate.
|
|
|
|
|
|
- **Audit log viewer** (`/api/v1/audit/log`): paginated, `rt_lead` only via
|
|
|
|
|
|
`AUDIT_READ`. Filters: `action`, `actor_id`, `resource_type`, `since`,
|
|
|
|
|
|
`until` (ISO 8601). Exposes `prev_hash` / `row_hash` so future clients can
|
|
|
|
|
|
verify the chain.
|
|
|
|
|
|
- **Pagination envelope** (D-016): `Page[T]` schema
|
|
|
|
|
|
`{items, total, page, page_size}` and `PageQuery` for parsing
|
|
|
|
|
|
`?page=&page_size=` (max 200). Used by `/users` and `/audit/log` this
|
|
|
|
|
|
sprint; existing flat-array endpoints stay unchanged.
|
|
|
|
|
|
- **Spec decisions** D-015, D-016, D-017 logged.
|
|
|
|
|
|
- **Tests**: 11 new unit tests (Pydantic shapes + pagination bounds) + 5 new
|
|
|
|
|
|
integration tests covering the critical MA6 scenario (`rt_lead creates
|
|
|
|
|
|
rt_operator → assigns engagement A → operator only sees A`), the RBAC
|
|
|
|
|
|
gate on `USER_MANAGE`, the 409 on duplicate emails, the audit pagination,
|
|
|
|
|
|
and the soft-disable login-block path.
|
|
|
|
|
|
- **`docs/api.md`** extended with the sprint-2 surface; the typo
|
|
|
|
|
|
`/engagements<eid>` → `/engagements/<eid>` fixed in passing.
|
|
|
|
|
|
|
fix(backend): JSON error envelope for every HTTPException + strict_slashes=False
Two issues spotted by ux-frontend consuming docs/api.md against the actual
code path:
1. `flask.abort(...)` returned the Werkzeug HTML error page for 400/403/404/
422/etc. — only the 401 paths going through `api_error()` and the
Flask-Login `unauthorized_handler` honoured the `{error, message}`
envelope the contract promised. The frontend's `ApiClientError.body`
parser was forced to fall back to a raw string, and the 422 case
could not surface Pydantic per-field errors.
Fix: register `@app.errorhandler(HTTPException)` that serialises every
`HTTPException` to the same JSON envelope. 422s gain a `details: [...]`
field holding the Pydantic `errors()` list (`loc` / `msg` / `type`),
matching the shape now documented in `docs/api.md`.
A `_HTTP_ERROR_CODES` map maps statuses to stable snake_case codes
(`bad_request`, `not_found`, `method_not_allowed`,
`validation_error`, `forbidden`, `internal_error`, ...). Unknown
statuses fall back to `http_error`.
`description` is `cast(object, ...)` because the Werkzeug stub pins it
to `str | None` while `flask.abort(..., description=<list>)` is the
officially supported way to smuggle a Pydantic errors list to the
handler.
2. `@bp.get("")` on the engagements blueprint produced `/api/v1/engagements`
(no slash). Hitting it with a trailing slash issued a 308 redirect,
and some browsers drop the session cookie across that hop.
Fix: `app.url_map.strict_slashes = False`. Both forms now match the
same handler without redirect.
5 new integration tests cover the new envelope shape (422 with details,
unknown 404, malformed-JSON 400) and the dual-slash matching. `docs/api.md`
rewritten to reflect the table of stable codes, the `details` shape, and
the no-trailing-slash convention. `CHANGELOG.md` gains a follow-up entry.
Verification: ruff check / mypy --strict / pytest tests/unit all green
(61 unit + 5 new integration).
2026-05-23 04:33:23 +02:00
|
|
|
|
### Sprint 1 — backend follow-up fixes (`feature/backend-auth-wiring`)
|
|
|
|
|
|
|
|
|
|
|
|
- **Global JSON error envelope** — register `@app.errorhandler(HTTPException)`
|
|
|
|
|
|
so every aborted request now flows through the same
|
|
|
|
|
|
`{ "error": "<code>", "message": "<human>", "details"? }` JSON shape that
|
|
|
|
|
|
`docs/api.md` documented but only `api_error()` honoured before. 422
|
|
|
|
|
|
responses surface the Pydantic per-field list under `details` so the
|
|
|
|
|
|
frontend can map errors back to form fields. New stable codes:
|
|
|
|
|
|
`bad_request`, `not_found`, `method_not_allowed`, `validation_error`,
|
|
|
|
|
|
`forbidden`, `internal_error`, etc. (see updated `docs/api.md`).
|
|
|
|
|
|
- **`strict_slashes=False`** on the URL map — `/api/v1/engagements` and
|
|
|
|
|
|
`/api/v1/engagements/` match the same handler. Removes the 308 redirect
|
|
|
|
|
|
that some browsers drop the session cookie through.
|
|
|
|
|
|
- **5 new integration tests** covering both slash variants, 422
|
|
|
|
|
|
`validation_error` envelope shape (incl. `details`), unknown-route 404,
|
|
|
|
|
|
and 400 on a non-JSON body.
|
|
|
|
|
|
- **`docs/api.md`** rewritten: full error code table, 422 `details`
|
|
|
|
|
|
example, trailing-slash policy, dropped trailing slashes from all
|
|
|
|
|
|
endpoint headings.
|
|
|
|
|
|
|
docs: add api.md contract for sprint 1 + update changelog
- docs/api.md: contract the frontend consumes — base URL, auth transport
(Flask session cookie, credentials: include), uniform error envelope,
MA6 tenant-scope behaviour (404 not 403), per-endpoint shape for
/auth/{login,logout,me} and /engagements GET/POST/GET-by-id, plus a
worked example walking through CLI bootstrap → login → POST engagement →
list → logout.
- CHANGELOG.md: sprint-1 entry summarising the three endpoints, the dev-
only CORS, the AuthUser extension, the audit rows, and the test
coverage.
2026-05-23 04:22:03 +02:00
|
|
|
|
### Sprint 1 — backend auth wiring (`feature/backend-auth-wiring`)
|
|
|
|
|
|
|
|
|
|
|
|
- **`POST /api/v1/auth/login`** — local-credentials login. Body `{username,
|
|
|
|
|
|
password}`; success returns the `CurrentUser` shape (`user_id`, `username`,
|
|
|
|
|
|
`display_name`, `role`, `permissions`, `groups`) and sets a Flask session
|
|
|
|
|
|
cookie. Failures return a uniform `401 invalid_credentials` envelope; a
|
|
|
|
|
|
bcrypt round runs against a dummy hash on unknown users to flatten the
|
|
|
|
|
|
timing signal.
|
|
|
|
|
|
- **`POST /api/v1/auth/logout`** — clears the session, returns `204`. Writes
|
|
|
|
|
|
an `auth.logout` audit row.
|
|
|
|
|
|
- **`GET /api/v1/auth/me`** — rehydrates the frontend at boot; returns the
|
|
|
|
|
|
current principal or `401 not_authenticated`.
|
|
|
|
|
|
- **Error envelope** — every API failure now returns
|
|
|
|
|
|
`{error: "<code>", message: "<human>"}`. `LoginManager.unauthorized_handler`
|
|
|
|
|
|
is wired to the same shape so `@login_required` 401s match.
|
|
|
|
|
|
- **Dev-only CORS** — `flask-cors` wraps `/api/*` for the origins in
|
|
|
|
|
|
`MIMIC_CORS_ORIGINS` only when `MIMIC_ENV=development`. Prod keeps
|
|
|
|
|
|
same-origin via the reverse proxy.
|
|
|
|
|
|
- **`AuthUser` extended** — carries `display_name` + `user_type` so the
|
|
|
|
|
|
serialiser can return them.
|
|
|
|
|
|
- **Audit** — `auth.login` and `auth.logout` rows go through the existing
|
|
|
|
|
|
hash-chained writer.
|
|
|
|
|
|
- **Docs** — `docs/api.md` describes the contract the frontend consumes
|
|
|
|
|
|
(login flow, CurrentUser shape, error envelope, MA6 tenant-scope behaviour).
|
|
|
|
|
|
- **Tests** — 5 unit tests on the schemas + serializer; integration scaffold
|
|
|
|
|
|
test `tests/integration/test_auth_engagement_e2e.py` exercises the full
|
|
|
|
|
|
login → /me → POST engagement → list → logout loop on a testcontainers
|
|
|
|
|
|
Postgres.
|
|
|
|
|
|
|
2026-05-21 20:10:47 +02:00
|
|
|
|
### Team decisions (2026-05-21)
|
|
|
|
|
|
|
|
|
|
|
|
- **Q1** — SOC client collaboration in the live cockpit is assumed valid (no PoC sheet).
|
|
|
|
|
|
- **Q2** — Mimic is deployed on RT infrastructure (not at client). SOC client connects over
|
|
|
|
|
|
the internet through the existing RT reverse proxy (out of Mimic scope).
|
|
|
|
|
|
- **Q3** — Project framed as "improve the existing shared sheet workflow", not "rebuild Caldera".
|
|
|
|
|
|
- **T2** — C2 credentials stored in a dedicated `c2_credential` table with version + retirement
|
|
|
|
|
|
(Fernet-encrypted `config_json`). Active row per engagement = `retired_at IS NULL`, max version.
|
|
|
|
|
|
- **T3** — Jinja templating exposes two accessors: `{{outputs.text}}` (stdout) and
|
|
|
|
|
|
`{{outputs.blob("key")}}` (binary, 10 MB cap, UTF-8 with latin-1 fallback).
|
|
|
|
|
|
- **T4** — `soc_session.token_opaque` stores a bcrypt hash; the clear token is delivered
|
|
|
|
|
|
out-of-band and never re-displayable.
|
|
|
|
|
|
- **Auth** — v1: local user/password (bcrypt + Flask session). v2: Keycloak OIDC mapping
|
|
|
|
|
|
onto the same group model. RBAC is group-based from day one.
|
|
|
|
|
|
|
|
|
|
|
|
### Sprint 0 in progress
|
|
|
|
|
|
|
|
|
|
|
|
Repo skeleton, data model, `C2Connector` ABC, Jinja2 sandbox, local auth + RBAC, flat CRUD,
|
|
|
|
|
|
UX wireframes (mock data). No real connector, no reporting until PR1/PR2/PR3 land.
|
2026-05-21 20:34:19 +02:00
|
|
|
|
|
|
|
|
|
|
#### Backend skeleton (`feature/backend-skeleton`)
|
|
|
|
|
|
|
|
|
|
|
|
- `backend/` Python 3.12+ project: `pyproject.toml` (ruff, mypy strict, pytest, coverage 70 %),
|
2026-05-22 19:49:16 +02:00
|
|
|
|
`Makefile` (Docker/Podman auto-detect), multi-stage `Dockerfile`, `compose.yml` for
|
2026-05-21 20:34:19 +02:00
|
|
|
|
Postgres dev DB, `.env.example`.
|
|
|
|
|
|
- Full §8 data model in SQLAlchemy 2 typed mapped classes: `engagement`, `c2_credential`,
|
|
|
|
|
|
`host`, `user`, `group`, `permission`, `group_permission`, `user_group`,
|
|
|
|
|
|
`engagement_member`, `ttp`, `scenario`, `scenario_step`, `run`, `run_step`,
|
|
|
|
|
|
`run_step_cleanup`, `detection`, `evidence`, `report`, `soc_session`, `audit_log`.
|
|
|
|
|
|
No `ttp_version` table (D-009 / H32 reaffirmed).
|
|
|
|
|
|
- Alembic baseline migration `202605210001_initial_schema`: every table + enum + index +
|
|
|
|
|
|
idempotent `audit_log` grants for the write-only Postgres role. Seeds the three F11
|
|
|
|
|
|
groups (`rt_operator`, `rt_lead`, `soc_analyst`) and their permission set (D-008).
|
|
|
|
|
|
- `C2Connector` ABC + `Payload` / `TaskHandle` / `TaskResult` / `TaskStatus` dataclasses +
|
|
|
|
|
|
`PayloadType` enum + `ConnectorFactory` keyed on `c2_type`. Mythic payload map populated;
|
|
|
|
|
|
Home stays empty until PR2.
|
2026-05-22 05:25:04 +02:00
|
|
|
|
- Jinja2 `SandboxedEnvironment` + `regex_extract` filter (`google-re2` hard dependency per
|
|
|
|
|
|
D-011 / B1 — `RuntimeError` at boot if absent, no `re` fallback) + `{{ outputs.text }}` /
|
|
|
|
|
|
`{{ outputs.blob() }}` accessors reading gzip-compressed blobs (10 MB cap after
|
|
|
|
|
|
decompression, UTF-8 → latin-1).
|
2026-05-21 20:34:19 +02:00
|
|
|
|
- Group-based RBAC: `Permission` + `GroupName` + `GROUP_PERMISSIONS` mirror the F11 matrix;
|
|
|
|
|
|
`@require_perm` decorator + `AuthUser` Flask-Login wrapper that resolves the permission set
|
|
|
|
|
|
from the user's groups.
|
|
|
|
|
|
- bcrypt password helpers + SOC opaque token (256-bit url-safe, bcrypt-hashed at rest, plain
|
|
|
|
|
|
returned once).
|
|
|
|
|
|
- Hash-chained append-only audit writer (sprint 0 fills `prev_hash` / `row_hash` at insert;
|
|
|
|
|
|
verifier shipped in v2).
|
|
|
|
|
|
- Flat CRUD blueprints: engagements / hosts / TTPs / scenarios + scenario steps. F3 invariant
|
2026-05-22 05:25:04 +02:00
|
|
|
|
enforced (host.c2_type must match scenario.c2_type at compose time). Every mutation calls
|
|
|
|
|
|
the hash-chained audit writer (MA5); created rows carry `created_by_id` (MA4); listings and
|
|
|
|
|
|
per-engagement routes scope to `engagement_member` for RT operators (MA6 / F11).
|
|
|
|
|
|
- Content-addressed gzip blob store (`mimic.storage.blob`): streaming write with a `max_bytes`
|
|
|
|
|
|
cap (raises `BlobTooLarge` mid-stream — MA2), atomic rename, `0o750` directory mode.
|
2026-05-21 20:34:19 +02:00
|
|
|
|
- `mimic-cli` (click): `user create`, `db dump`, `db restore`.
|
2026-05-22 05:25:04 +02:00
|
|
|
|
- pytest baseline: **56 unit tests passing** (templating, regex_extract, password, soc_token,
|
|
|
|
|
|
RBAC matrix, connector factory, audit hash, blob CAS, migration seed parity). Integration
|
|
|
|
|
|
scaffold ready for testcontainers Postgres (`/healthz` smoke included).
|
2026-05-21 20:34:19 +02:00
|
|
|
|
|
|
|
|
|
|
#### Spec deltas applied in this sprint
|
|
|
|
|
|
|
|
|
|
|
|
Authoritative decisions implemented per `tasks/spec-decisions.md`:
|
|
|
|
|
|
- **D-008** — Seeded groups = exactly the three F11 roles, permission matrix from F11.
|
|
|
|
|
|
- **D-009** — No `ttp_version` table (H32 reaffirmed).
|
|
|
|
|
|
- **D-011** — `regex_extract` fails loudly on no-match (raises `TemplateError`).
|
|
|
|
|
|
- **D-012** — `output_blob_ref` stored in `MIMIC_BLOB_ROOT` (CAS gzip layout); evidence
|
|
|
|
|
|
files live under `MIMIC_EVIDENCE_ROOT` (flat per-engagement).
|
|
|
|
|
|
|
|
|
|
|
|
Implementation arbitrations logged in this sprint:
|
|
|
|
|
|
- **D-013** — `audit_log` hash chain (`prev_hash` / `row_hash`) shipped v1.
|
|
|
|
|
|
- **D-014** — UUID columns use SQLAlchemy 2 native `Uuid` mapping; no `type_annotation_map`
|
|
|
|
|
|
on the declarative base (Flask-SQLAlchemy incompatibility).
|
2026-05-22 05:25:04 +02:00
|
|
|
|
|
|
|
|
|
|
#### Code-review remediation (`12d131c` → `feature/backend-skeleton`)
|
|
|
|
|
|
|
|
|
|
|
|
- **B1** — Dropped the `re` stdlib fallback in `regex_extract`. `google-re2` is now a hard
|
|
|
|
|
|
dependency (B1 / D-011); the module raises `RuntimeError` at import if absent.
|
|
|
|
|
|
- **MA1** — Removed `scripts/postgres-init/00-roles.sql` (no more hardcoded `CHANGE_ME`
|
|
|
|
|
|
password). Audit-writer role provisioning is the playbook's responsibility (D-010);
|
|
|
|
|
|
`backend/README.md` documents the manual dev-only `CREATE ROLE` command.
|
|
|
|
|
|
- **MA2** — `store_blob` now accepts a binary stream + `max_bytes`, streams sha256+gzip in
|
|
|
|
|
|
64 KB chunks, and raises `BlobTooLarge` mid-stream (cleans up the temp file). No more
|
|
|
|
|
|
whole-buffer RAM load.
|
|
|
|
|
|
- **MA3** — Inlined the F11 permission matrix in the initial Alembic migration; the runtime
|
|
|
|
|
|
matrix is no longer imported there. A new unit test
|
|
|
|
|
|
(`test_migration_seed_matches_current_matrix`) fails if the two drift apart.
|
|
|
|
|
|
- **MA4** — `created_by_id = current_user.id` set in `engagement`, `ttp`, and `scenario`
|
|
|
|
|
|
create endpoints.
|
|
|
|
|
|
- **MA5** — Every mutation endpoint now writes an audit row through the hash-chained
|
|
|
|
|
|
`AuditWriter` (F13).
|
|
|
|
|
|
- **MA6** — RT operators only see engagements they are members of (`engagement_member` join
|
|
|
|
|
|
on list, membership probe on `get`/`put`/`delete`/`host`/`scenario`/...). RT leads bypass.
|
|
|
|
|
|
- **N4** — `gunicorn` declared in `pyproject.toml` dependencies (the Dockerfile `CMD` now
|
|
|
|
|
|
resolves correctly).
|
|
|
|
|
|
- **N6** — `tests/integration/conftest.py` keeps `db.create_all()` for now; commented TODO to
|
|
|
|
|
|
switch over to Alembic once the playbook owns the audit role.
|
|
|
|
|
|
- **M8** — Initial migration docstring no longer mentions `ttp_version`.
|
|
|
|
|
|
|
|
|
|
|
|
Verification on the latest commit: `ruff check`, `ruff format --check`, `mypy --strict`, and
|
|
|
|
|
|
`pytest tests/unit` all pass; 56 unit tests green.
|