Two changes scoped together since both stem from the post-PR2 wrap-up.
docs/podman-runner-setup.md (new, ~190 LOC):
Operational runbook for the gitea-runner host that drives CI. The first
attempt at install hit four traps that this archived version documents
so we don't lose the lesson:
1. `act_runner register` performs a sanity ping against the container
daemon before writing the credential. Without the Podman socket
mounted on the *register one-shot*, register fails silently and no
.runner file is produced. The runbook mounts the socket on both
register and daemon containers.
2. SELinux blocks rootless socket access by default. Quadlet
SecurityLabelDisable=true (or --security-opt label=disable for the
legacy CLI form) is the documented bypass. No-op on Debian, required
on RHEL/Fedora hosts.
3. The runner user UID is not 1000 on every host (gitea = 1005 here).
Quadlet `%U` substitution makes the unit portable; hardcoded UIDs
are explicitly called out as a sprint 0 mistake.
4. `podman generate systemd` is officially deprecated. Quadlet is the
only supported pattern going forward and is what this runbook ships;
legacy alternative is omitted on purpose.
Also captures: token placeholder convention (<TOKEN_FROM_GITEA_UI>,
never the real value in archived docs), single-use semantics, the
"secrets via file, not chat" convention, the `:X.Y.Z` pin policy versus
`:latest` in prod (ties into follow-up F-D1), and a decommissioning
section that cleans up state without nuking the user-level Podman socket.
tasks/todo.md:
New section "Frontend follow-ups (sprint 1+)" with F-D1..F-D5 from
code-reviewer on `chore/frontend-dockerfile` (649194b). All deferred,
none blocking. F-D1 (digest pinning) is project-wide and explicitly
references the backend image and the runner image alongside the
frontend ones for a single chore commit.
Production image for the frontend dist.
Stage 1 (build): node:22-alpine, `npm ci --ignore-scripts` from the
committed lockfile, `npm run build`. Output lands in /app/dist.
Stage 2 (runtime): docker.io/nginxinc/nginx-unprivileged:alpine.
- Upstream-maintained variant that runs as the nginx user (uid 101)
out of the box. /var/cache/nginx and /var/run/nginx are pre-owned,
no chown gymnastics needed in our layer. Vanilla nginx:alpine fails
at startup as non-root because client_temp mkdir is denied.
- Listens on 8080 (non-privileged port, matches the unprivileged
variant convention).
- nginx.conf serves /usr/share/nginx/html with SPA `try_files`
fallback for client-side routing, long-cache headers on
/assets/ (Vite hashed bundles), a plaintext /healthz endpoint
for Caddy / Prometheus blackbox, and server_tokens off.
.dockerignore excludes node_modules, dist, .vite, coverage,
playwright-report, .env*, .git, editor dirs. Keeps .env.example.
Smoke local validated with `podman build -t mimic-frontend:smoke .`
and `podman run -p 127.0.0.1:18080:8080`:
/healthz -> 200 "ok"
/ -> 200 index.html (508 B)
/spa/x -> 200 (SPA fallback)
/assets -> Cache-Control: max-age=31536000, public, immutable
The smoke workflow was scoped from inception to validate that the
freshly registered gitea-runner picks up jobs with the "linux" label.
It ran green on push of chore/podman-and-ci. Removing per the
"transient, removed after validation" plan recorded in the original
commit (1380672).
Three docs still referenced the old docker-compose.yml path. Replace
with compose.yml so a future reader cloning at this hash finds the
file at the documented path.
- CHANGELOG.md:31 — backend skeleton recap line.
- docs/architecture.md:28 — deployment artifacts note (D-010 scope).
- tasks/todo.md:9 — B0.1 task description.
Also adds a "CI follow-ups (sprint 1+)" section to tasks/todo.md
capturing the 3 MINOR + 6 NIT deferred from code-reviewer's review
of chore/podman-and-ci, plus a FERNET-KEY tracker for the secret
provisioning before c2_credential.config_fernet (D-004) is wired.
Two workflows under .gitea/workflows/:
- ci.yml — runs on push:main and every PR. Two parallel jobs:
* backend (python:3.12-slim-bookworm): apt deps for psycopg + WeasyPrint,
pip install -e backend[dev], ruff check + ruff format --check + mypy
--strict src + pytest tests/unit. Postgres 16 service for any
integration-style test, env wired via service hostname.
FERNET_KEY_TEST sourced from Gitea repo secret (no plain value in CI).
* frontend (node:22-alpine): npm ci, ESLint, TypeScript typecheck,
Vitest, Vite build.
Runner label: linux (matches gitea-runner registration).
Out of scope sprint 0: testcontainers Postgres integration tests
(Docker-in-Docker rootless setup deferred to nightly job) and
Playwright E2E (deferred to sprint 1+).
- smoke.yml — transient. Triggers only on push to this branch
(chore/podman-and-ci) and on workflow_dispatch. Validates that the
newly registered gitea-runner picks up jobs with the "linux" label.
Removed in a follow-up commit on this branch once green.
Compose v2 canonical filename (compose.yml) is recognized by both
docker compose and podman compose without preference. The previous
docker-compose.yml worked but signalled a Docker-first stance, while
target deployment is Podman 5.8+ rootless.
- Rename backend/docker-compose.yml -> backend/compose.yml.
- backend/README.md `make db-up` comment uses $(CONTAINER) to mirror
the Makefile auto-detect (lines 14-16: docker || podman).
- backend/README.md audit-writer bootstrap snippet hints at podman
fallback explicitly with `command -v` runtime sniff.
- backend/compose.yml comment for audit-writer mentions both runtimes.
No functional change. Makefile $(COMPOSE) target unchanged: Compose v2
discovers compose.yml first in its search order.
- `tasks/todo.md`: B0.5 description updated (re2 hard dep, no fallback);
add a "Backend follow-ups (sprint 1+)" section with M1-M7 + N1-N6 from
the code-review verdict.
- `CHANGELOG.md`: backend skeleton bullets refreshed (no re fallback,
streaming blob store, audit + scope on CRUD, 56 unit tests); new
"Code-review remediation" subsection lists B1 / MA1-MA6 / N4 / N6 / M8
with one-line rationale each.
- `tests/integration/conftest.py`: leave `db.create_all()` in place but
add an inline TODO (N6) pointing at the Alembic switchover that will
exercise the F11 seed + audit-log role grants in CI.
Three follow-ups on the flat CRUD blueprints triggered by code-review +
spec-analyst (MA4, MA5, MA6).
**MA4 — `created_by_id`** — engagements, TTPs and scenarios now record the
creator from `current_user.id` instead of leaving the FK NULL. The new
`api._helpers.current_user_id()` exposes the UUID safely (returns None when
the request is unauthenticated, e.g. during /healthz).
**MA5 — Audit log integration** — `api._helpers.audit_write(...)` wraps the
hash-chained `AuditWriter` and is called after every successful commit in
the 4 blueprints (engagement / host / ttp / scenario incl. step), recording
the actor, action, resource type/id, IP, user agent, and small metadata
(field list, names, engagement scope). F13 "Toute mutation tracée" now
holds end-to-end.
**MA6 — RT operator scope on engagements** — F11 limits RT operators to
"engagements assignés". The previous implementation let them list / read
every engagement and every nested resource. Fix: `is_rt_lead()` short-
circuits the check for RT leads; otherwise a membership probe against
`engagement_member` runs on every list/read and on `_engagement_or_404` in
`hosts.py` and `scenarios.py`. Listings now `JOIN engagement_member` and
filter by `current_user.id`.
`audit_write` casts `db.session` (a `scoped_session` proxy) to the unwrapped
`sqlalchemy.orm.Session` that `AuditWriter` expects; the two are
interchangeable at runtime.
The promotion-perm check on TTPs no longer needs a lazy `flask_login` import
since the decorator scope already brings `current_user` in.
Code-review MAJOR MA3. The initial Alembic migration imported the live
`mimic.rbac.matrix.GROUP_PERMISSIONS` to seed the `permission` / `group` /
`group_permission` rows. That breaks the Alembic invariant "a migration
produces the same schema regardless of when you replay it": a future tweak
to the runtime matrix would silently change the seeded baseline on a fresh
DB.
Two changes:
1. The migration now carries an *inline frozen snapshot* of the F11 matrix
(`_PERMISSIONS_FROZEN`, `_GROUP_PERMISSIONS_FROZEN`, `_GROUP_DESCRIPTIONS`).
The seed reads from these tuples/dicts only. If the canonical matrix
evolves, the next migration is responsible for the delta.
2. A new unit test `test_migration_seed_matches_current_matrix` enforces
that the frozen seed equals the runtime `Permission` enum and
`GROUP_PERMISSIONS` mapping. Drift now fails CI loudly with a hint to
write a new migration instead of editing the existing one.
Also: docstring no longer mentions `ttp_version` (M8 follow-up).
Code-review MAJOR MA2. The previous `store_blob(root, data: bytes)` signature
forced the entire payload into RAM before the 10 MB cap was checked — a
hostile-large output blob could OOM the worker before the limit even fired.
New signature: `store_blob(root, stream, *, max_bytes=10_485_760)`. The
implementation:
- reads from `stream` in 64 KB chunks;
- updates the sha256 + writes to `<root>/.tmp-<pid>-<rand>.gz` incrementally;
- raises `BlobTooLarge(max_bytes)` as soon as the running total crosses the
cap, then unlinks the partial temp file via `contextlib.suppress`;
- atomic-renames the temp file to the CAS path `<aa>/<bb>/<sha256>.gz` once
the stream finishes;
- sets `0o750` on the directory and `0o640` on the file with explicit
`os.chmod` (does not rely on the process umask).
Updated unit tests cover: BlobTooLarge enforcement (with temp-file cleanup),
multi-chunk happy path (1.5 MB payload exercising the 64 KB loop), and
`max_bytes <= 0` validation.
Code-review MAJOR MA1. The previous `scripts/postgres-init/00-roles.sql`
hardcoded a `CHANGE_ME` password for `mimic_audit_writer` and was bind-mounted
into the dev Postgres container; on prod boxes this risks lingering as the
real credential.
- The init script was removed in the previous commit alongside the dropped
scripts dir.
- `docker-compose.yml` no longer mounts a `docker-entrypoint-initdb.d`
directory; the audit-writer role provisioning is the Ansible playbook's
responsibility (D-010).
- `backend/README.md` documents the manual one-shot `CREATE ROLE` command
for local dev with a placeholder password.
Net effect: no `CHANGE_ME` credential reaches a container image / git history.
The Alembic migration's `audit_log` grant block stays idempotent — it is a
no-op when the role is absent.
Code-review BLOCKER B1. Reaffirms D-011: a `re` stdlib fallback defeats the
OPSEC-safe-regex guarantee because hostile C2 output can trigger catastrophic
backtracking. The `[:1MB]` slice cap does not mitigate that — re-evaluating
a malicious pattern over 1 MB of attacker-controlled text is still a worker
freeze.
- `mimic.templating.filters` now imports `re2` unconditionally and raises
`RuntimeError` at module load if the binding is absent. No `re` import,
no `_HAS_RE2` branch, no `_FALLBACK_MAX_INPUT`.
- `pyproject.toml` already pinned `google-re2 >= 1.1, < 2.0`; this commit
hardens the import path to actually enforce it.
- New test `test_re2_is_required` asserts the binding is wired in.
Pre-merge sanity per devops checklist (ruff format --check, mypy --strict).
Type fixes:
- ORM models: `Mapped[dict]` → `Mapped[dict[str, Any]]` (audit, scenario, run,
report, ttp, detection.artifact_files_json). Equivalent on Pydantic DTOs
(TtpBase.params_schema_json, ScenarioStepBase.params_override_json).
- Rename `TtpRead.current_version` → `TtpRead.version` to mirror the ORM
column (which itself was renamed in D-009 cleanup).
- Flask blueprints: add `-> ResponseReturnValue` to every view, plus typed
UUID params on `_validate_step_consistency`.
- `templating/filters.py`: rewrite the conditional re2 import so mypy can
narrow the union (`ModuleType | None`); the runtime branch on `_re2 is not
None` removes the unused-ignore that was triggered by warn_unused_ignores.
- `pyproject.toml`: add `flask_login.*` and `pythonjsonlogger.*` to the
`[[tool.mypy.overrides]]` `ignore_missing_imports` list (both ship without
typed marker).
- Misc: drop stale `# type: ignore` comments (`app.py:36`,
`rbac/decorators.py:35`) flagged by `warn_unused_ignores`. Keep
`logging.JsonFormatter` ignore because the symbol exists at runtime but is
not re-exported through the typed surface.
Formatting:
- `ruff format` applied (15 files normalized; line-length unchanged at 100).
Verification on this commit:
- `ruff check` → All checks passed.
- `ruff format --check` → 68 files already formatted.
- `mypy --strict src` → Success: no issues found in 54 source files.
- `pytest tests/unit` → 49 passed.
D-011 — `regex_extract(text, pattern, *, group=1, name=None)`:
- engine google-re2 (linear-time, ReDoS-safe), `re` fallback with 1 MB cap.
- first match only.
- no match → raises Jinja2 `TemplateError` (no silent default — cleanup
templates must fail loud when source string drifts).
- default capture is group 1 with fallback to group(0) when the pattern has
no groups; named groups via `name="<name>"`.
D-012 — `outputs.blob()`:
- reads the gzip-compressed CAS file from `MIMIC_BLOB_ROOT`.
- 10 MB cap is applied **after** decompression.
- decode UTF-8 with latin-1 fallback; never raises (missing / corrupt /
non-gzip blobs return empty string, logged at WARNING).
Unit tests rewritten to cover both the new fail-loud regex contract and
the gzip read path. 49 unit tests pass; ruff clean.
D-009 reaffirms spec H32: no `ttp_version` table. Replayability lives solely
on `run.snapshot_json`. The previous initial migration introduced a separate
`ttp_version` aggregate by mistake — removed here.
D-008 requires the bootstrap to seed exactly the three F11 groups
(`rt_operator`, `rt_lead`, `soc_analyst`) with exactly the F11 permission
matrix. The migration now:
- inserts every `Permission` enum value into the `permission` table,
- inserts the three groups with deterministic uuid5(NAMESPACE_DNS, ...) ids,
- inserts the matching `group_permission` rows from GROUP_PERMISSIONS.
Also renames `ttp.current_version` to `ttp.version` (matches §8 spec column
name; the value remains informational per H32 / D-009).
M1 — Single SessionProvider via nested router.
The previous router had two route entries with `path: '/'`
(Navigate, AppShell) plus a separate `/login` entry, each wrapped in
its own RootLayout. That instantiated SessionProvider three times,
forking state the moment session writes diverged across siblings.
Replaced by one Root route with SessionProvider + <Outlet />, and
index/login/AppShell-children nested underneath. RootLayout (the
per-tree wrapper) is now obsolete and deleted; the new Root component
lives in src/routing/Root.tsx (addresses NIT N4 as a side effect).
M2 — Typo: "pollign" → "polling" in LiveCockpitPage masthead.
M3 — Replace asymmetric `?? 'rt_operator'` / `?? 'soc_analyst'`
fallbacks in LiveCockpitPage with an explicit `if (!user) return null;`
guard placed after all hooks (rules-of-hooks). AppShell already
redirects unauthenticated visitors to /login, so the guard documents
the invariant rather than introducing one.
NITs N1-N3, N5-N7 recorded in tasks/todo.md as sprint 1+ follow-ups.
- Flask app factory wires SQLAlchemy / Migrate / Login / SocketIO and
registers every blueprint. /healthz smoke endpoint included.
- Pydantic 2 DTOs (request/response) for engagement / host / TTP /
scenario aggregates with from_attributes=True conversion.
- Flat CRUD blueprints under /api/v1/:
* engagements (list / create / get / put / delete-as-archive)
* hosts (engagement-scoped CRUD)
* library/ttps (CRUD; promote requires the lead-only TTP_PROMOTE)
* scenarios + steps (F3 invariant enforced: host.c2_type must match
scenario.c2_type at compose time, 400 otherwise).
- @require_perm guards every endpoint per the F11 matrix.
- audit/ writer is hash-chained from v1 (SHA-256 of canonical record
plus previous hash). The SQL-level write-only role enforcement ships
in the deploy playbook (idempotent grants run at migration time).
- mimic-cli (click): user create (seeds RT operator/lead with group
membership), db dump / db restore (manual pg_dump/pg_restore, R-O1).
No orchestrator, no WebSocket, no report generation — those land after
PR1/PR2/PR3.
- Permission enum + GroupName enum + GROUP_PERMISSIONS mapping mirror
the F11 matrix in code (verifiable against the spec table in tests).
- @require_perm decorator: 401 on anonymous, 403 on missing permission,
passes through otherwise. Pure-function user_has() for unit-testing.
- AuthUser (Flask-Login wrapper) resolves the permission set from a
User's groups; load_user is the Flask-Login user_loader.
- bcrypt password hashing helpers (12 rounds by default, configurable).
- SOC opaque token (D-006): secrets.token_urlsafe(32), bcrypt-hashed at
rest, plain value returned once at creation and never re-displayable.
- Group-based RBAC from day one (D-003) — Keycloak OIDC in v2 maps onto
the same group model.
- CleanupRenderer wraps jinja2.sandbox.SandboxedEnvironment with
StrictUndefined (no autoescape — shell context, not HTML).
- Custom filter regex_extract(text, pattern, group=1, default='') uses
google-re2 for linear-time matching (ReDoS-safe) and falls back to
re with a 1 MB input cap when re2 is absent.
- StepOutputs exposes {{ outputs.text }} and {{ outputs.blob('name') }}.
blob() decodes UTF-8 with latin-1 fallback, hard-capped at 10 MB
(consistent with F8 evidence limit, D-005).
- render_cleanup() is the module-level convenience wrapper.
- abstract C2Connector with authenticate / list_hosts / execute_task /
get_task_result / cancel_task / execute_cleanup; stream_task_output
optional v1 (NotImplementedError).
- Payload / TaskHandle / TaskResult / TaskStatus frozen dataclasses.
- UnsupportedPayloadType raised when no native command maps to the
chosen (c2_type, payload_type) pair.
- Mythic payload_type → native command map populated (spec §7 table).
- HOME map left empty until PR2 is closed.
- ConnectorFactory: register_connector decorator + build(c2_type) that
instantiates + authenticates via an injected config resolver.
No real Mythic / Home implementations land in this sprint.
- pyproject.toml with ruff + mypy strict + pytest + coverage >=70%
- Makefile with Docker/Podman auto-detect
- Multi-stage Dockerfile (python:3.12-slim-bookworm, non-root user)
- docker-compose.yml for Postgres dev DB
- alembic.ini wired to src/mimic/db/migrations
- scripts/postgres-init/00-roles.sql seeds the audit writer role
- .env.example documents every MIMIC_* var (no secrets committed)
- Role enum (rt_operator, rt_lead, soc_analyst) aligned with spec §3 / F11.
Frontend predicates (isRT, isLead, isSOC) drive layout only — backend
remains the source of truth for permissions (D-008).
- SessionContext split into Provider (TSX) and hook (useSession) to satisfy
react-refresh/only-export-components.
- AppShell composes StatusRail (link health, active run, UTC clock, build) +
Sidebar (role-conditional nav with keyboard shortcut hints) + Outlet.
Unauthenticated visitors redirect to /login.
- StatusRail uses pulsing status-dot pattern and label-system micro-typo
(uppercase 10px, 0.08em tracking) to evoke an instrument-panel header.
- Router (createBrowserRouter): /login outside the shell, all app routes
nested inside the shell. RootLayout extracted to its own file for
fast-refresh compliance.
Routes (sprint 0, flat):
/login LoginPage
/engagements EngagementsPage
/library TtpLibraryPage (RT only — gated client-side, will
be re-enforced by backend RBAC)
/scenarios ScenarioComposerPage (RT only)
/runs LiveCockpitPage
/reports ReportPage
/audit AuditPage (lead RT only)
Sub-routes under /engagements/:eid land in sprint 1+ when real scoping
arrives.
D-011 freezes the regex_extract Jinja filter signature
`regex_extract(text, pattern, *, group=1, name=None)`, google-re2 engine,
raise on no-match — unblocks backend B0.5 templating sandbox.
D-012 splits storage in two pools: `blobs/` (CAS sha256 + gzip) for C2
binary outputs and `evidence/` (flat per engagement) for user uploads,
10 MB per-blob cap, no global quota v1.
Q-001 and Q-002 removed from open-questions.md (resolved).
Q-003/Q-004/Q-005 marked `deferred` with explicit re-open conditions.
Captures the four grey areas team-lead flagged in the sprint 0 brief
(regex_extract semantics, output_blob_ref storage, /hosts/sync merge
behaviour, payload_type↔home-C2 mapping) plus stale-host policy.
No decisions taken: each entry lists options, a recommended default
if no decision is reached, and a "becomes blocking when…" trigger.
Resolved questions will move to spec-decisions.md as D-NNN entries.
D-008 frames the group-based RBAC layout as an OIDC-prep mechanism that must
seed exactly the three F11 spec roles and their canonical permission matrix.
Custom groups remain out of v1 scope.
D-009 reaffirms H32: replayability lives only on run.snapshot_json. The
ttp_version table listed in B0.2 must be dropped from the initial migration.
