From e77ca906d49b95454802788f44b2e48eb2f7a6fe Mon Sep 17 00:00:00 2001 From: knacky Date: Fri, 22 May 2026 05:25:04 +0200 Subject: [PATCH] docs(backend): track sprint-0 follow-ups + flag integration migration gap - `tasks/todo.md`: B0.5 description updated (re2 hard dep, no fallback); add a "Backend follow-ups (sprint 1+)" section with M1-M7 + N1-N6 from the code-review verdict. - `CHANGELOG.md`: backend skeleton bullets refreshed (no re fallback, streaming blob store, audit + scope on CRUD, 56 unit tests); new "Code-review remediation" subsection lists B1 / MA1-MA6 / N4 / N6 / M8 with one-line rationale each. - `tests/integration/conftest.py`: leave `db.create_all()` in place but add an inline TODO (N6) pointing at the Alembic switchover that will exercise the F11 seed + audit-log role grants in CI. --- CHANGELOG.md | 45 ++++++++++++++++++++++++--- backend/tests/integration/conftest.py | 3 ++ tasks/todo.md | 44 ++++++++++++++++++++++++-- 3 files changed, 84 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f004b55..8aeb479 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -41,8 +41,10 @@ UX wireframes (mock data). No real connector, no reporting until PR1/PR2/PR3 lan - `C2Connector` ABC + `Payload` / `TaskHandle` / `TaskResult` / `TaskStatus` dataclasses + `PayloadType` enum + `ConnectorFactory` keyed on `c2_type`. Mythic payload map populated; Home stays empty until PR2. -- Jinja2 `SandboxedEnvironment` + `regex_extract` filter (google-re2 with `re` fallback) + - `{{ outputs.text }}` / `{{ outputs.blob() }}` accessors (10 MB cap, UTF-8 → latin-1). +- Jinja2 `SandboxedEnvironment` + `regex_extract` filter (`google-re2` hard dependency per + D-011 / B1 — `RuntimeError` at boot if absent, no `re` fallback) + `{{ outputs.text }}` / + `{{ outputs.blob() }}` accessors reading gzip-compressed blobs (10 MB cap after + decompression, UTF-8 → latin-1). - Group-based RBAC: `Permission` + `GroupName` + `GROUP_PERMISSIONS` mirror the F11 matrix; `@require_perm` decorator + `AuthUser` Flask-Login wrapper that resolves the permission set from the user's groups. @@ -51,10 +53,15 @@ UX wireframes (mock data). No real connector, no reporting until PR1/PR2/PR3 lan - Hash-chained append-only audit writer (sprint 0 fills `prev_hash` / `row_hash` at insert; verifier shipped in v2). - Flat CRUD blueprints: engagements / hosts / TTPs / scenarios + scenario steps. F3 invariant - enforced (host.c2_type must match scenario.c2_type at compose time). + enforced (host.c2_type must match scenario.c2_type at compose time). Every mutation calls + the hash-chained audit writer (MA5); created rows carry `created_by_id` (MA4); listings and + per-engagement routes scope to `engagement_member` for RT operators (MA6 / F11). +- Content-addressed gzip blob store (`mimic.storage.blob`): streaming write with a `max_bytes` + cap (raises `BlobTooLarge` mid-stream — MA2), atomic rename, `0o750` directory mode. - `mimic-cli` (click): `user create`, `db dump`, `db restore`. -- pytest baseline: **38 unit tests passing**, integration scaffold ready for testcontainers - Postgres (`/healthz` smoke included). +- pytest baseline: **56 unit tests passing** (templating, regex_extract, password, soc_token, + RBAC matrix, connector factory, audit hash, blob CAS, migration seed parity). Integration + scaffold ready for testcontainers Postgres (`/healthz` smoke included). #### Spec deltas applied in this sprint @@ -69,3 +76,31 @@ Implementation arbitrations logged in this sprint: - **D-013** — `audit_log` hash chain (`prev_hash` / `row_hash`) shipped v1. - **D-014** — UUID columns use SQLAlchemy 2 native `Uuid` mapping; no `type_annotation_map` on the declarative base (Flask-SQLAlchemy incompatibility). + +#### Code-review remediation (`12d131c` → `feature/backend-skeleton`) + +- **B1** — Dropped the `re` stdlib fallback in `regex_extract`. `google-re2` is now a hard + dependency (B1 / D-011); the module raises `RuntimeError` at import if absent. +- **MA1** — Removed `scripts/postgres-init/00-roles.sql` (no more hardcoded `CHANGE_ME` + password). Audit-writer role provisioning is the playbook's responsibility (D-010); + `backend/README.md` documents the manual dev-only `CREATE ROLE` command. +- **MA2** — `store_blob` now accepts a binary stream + `max_bytes`, streams sha256+gzip in + 64 KB chunks, and raises `BlobTooLarge` mid-stream (cleans up the temp file). No more + whole-buffer RAM load. +- **MA3** — Inlined the F11 permission matrix in the initial Alembic migration; the runtime + matrix is no longer imported there. A new unit test + (`test_migration_seed_matches_current_matrix`) fails if the two drift apart. +- **MA4** — `created_by_id = current_user.id` set in `engagement`, `ttp`, and `scenario` + create endpoints. +- **MA5** — Every mutation endpoint now writes an audit row through the hash-chained + `AuditWriter` (F13). +- **MA6** — RT operators only see engagements they are members of (`engagement_member` join + on list, membership probe on `get`/`put`/`delete`/`host`/`scenario`/...). RT leads bypass. +- **N4** — `gunicorn` declared in `pyproject.toml` dependencies (the Dockerfile `CMD` now + resolves correctly). +- **N6** — `tests/integration/conftest.py` keeps `db.create_all()` for now; commented TODO to + switch over to Alembic once the playbook owns the audit role. +- **M8** — Initial migration docstring no longer mentions `ttp_version`. + +Verification on the latest commit: `ruff check`, `ruff format --check`, `mypy --strict`, and +`pytest tests/unit` all pass; 56 unit tests green. diff --git a/backend/tests/integration/conftest.py b/backend/tests/integration/conftest.py index a9eea82..30bfac7 100644 --- a/backend/tests/integration/conftest.py +++ b/backend/tests/integration/conftest.py @@ -37,6 +37,9 @@ def app(postgres_dsn: str, monkeypatch: pytest.MonkeyPatch): application = create_app() with application.app_context(): + # TODO (N6 follow-up, sprint 1): run Alembic migrations instead of + # db.create_all() so the integration tests exercise the real schema + # including the audit_log role grants and the F11 seed. db.create_all() yield application db.session.remove() diff --git a/tasks/todo.md b/tasks/todo.md index d23c5ca..1adfe00 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -16,9 +16,10 @@ Repo skeleton + foundational modules. Nothing that depends on PR1/PR2/PR3. evidence, report, soc_session, c2_credential, audit_log). - [x] B0.4 — `C2Connector` ABC + dataclasses + `payload_type` enum + factory keyed on `c2_type`. Mythic payload map populated; Home stays empty until PR2. -- [x] B0.5 — Jinja2 SandboxedEnvironment, `regex_extract` filter (google-re2 with `re` - fallback) per D-011 semantics (fail-loud no-match), `{{ outputs.text }}` / - `{{ outputs.blob() }}` accessors with 10 MB cap. +- [x] B0.5 — Jinja2 SandboxedEnvironment, `regex_extract` filter (`google-re2` hard + dependency per D-011, raises `RuntimeError` at boot if absent — no `re` fallback), + fail-loud no-match, `{{ outputs.text }}` / `{{ outputs.blob() }}` accessors + reading gzip-compressed blobs with 10 MB cap. - [x] B0.6 — bcrypt password helpers + SOC opaque token (256-bit url-safe, bcrypt-hashed) + group-based RBAC matrix matching F11 + `@require_perm` decorator. - [x] B0.7 — Flat CRUD blueprints for engagements / hosts / TTPs / scenarios (incl. step @@ -26,6 +27,43 @@ Repo skeleton + foundational modules. Nothing that depends on PR1/PR2/PR3. - [x] B0.8 — pytest baseline: unit tests passing, integration scaffold ready (testcontainers Postgres + `/healthz` smoke). +## Backend follow-ups (sprint 1+) + +Tracked from code-review verdict on `feature/backend-skeleton` @ 12d131c: + +### MINOR (8) — to schedule + +- **M1** — Replace `parse_uuid` integer-ish lookup with `werkzeug` UUID converter on + the routes (``) to avoid the 404 on malformed strings being hidden by + the 400 path. +- **M2** — Add OpenAPI generation (Pydantic 2 + `flask-pydantic-openapi` or hand-rolled). +- **M3** — Wire `flask-limiter` for `/auth/local/login` (NF-network). +- **M4** — Replace string-based `Engagement.status` setter with a typed transition method. +- **M5** — Introduce per-engagement read view that pre-joins `engagement_member` for + RT operator dashboards (current per-request join is fine for v1 traffic, but + re-evaluate at scale). +- **M6** — `mimic-cli user create` does not handle the SOC user-type (intended, but + document and gate explicitly with a clean error message). +- **M7** — Add a `mimic-cli` `engagement add-member --role rt_operator` shortcut so + the F11 scoping in MA6 is reachable from the CLI without manual SQL. +- **M8** — _(fixed in MA1 follow-up commit)_ Initial migration docstring no longer + references `ttp_version`. + +### NIT (6) — opportunistic + +- **N1** — Sort imports inside `mimic.db.models.__init__` alphabetically for diff + stability. +- **N2** — Extract the `_engagement_or_404` duplicated body into a shared helper. +- **N3** — Replace the inline `Permission.TTP_PROMOTE not in perms` check in `ttps.py` + with a second `@require_perm`-style decorator. +- **N4** — _(fixed)_ `gunicorn` added to `pyproject.toml` dependencies. +- **N5** — Replace bare `getattr(current_user, "groups", frozenset())` accesses by a + thin `current_groups()` helper. +- **N6** — `tests/integration/conftest.py` uses `db.create_all()` instead of running + Alembic. Marked with a TODO; switch over once the F11 seed must be exercised in + integration. Plan: convert to `alembic upgrade head` once the audit role + bootstrap lives in the playbook (D-010). + ## Frontend (`ux-frontend`) - [ ] F0.1 — `frontend/` Vite + React + TypeScript strict + Tailwind 4 + TanStack Query 5,