mimic-big/docs/architecture.md

# Mimic — sprint 0 architecture

This is the **as-of-sprint-0** mirror of what's committed in
`feature/backend-skeleton` and `feature/frontend-skeleton`. It does not invent
beyond the code. When you read this and the code disagrees, the code wins —
file a doc fix.

Authoritative sources outside this file:

- **Frozen spec** — `RT-SecondBrain/Projects/Mimic — Spec.md` (vault).
- **Implementation arbitrations** — `tasks/spec-decisions.md` (D-001..D-014).
- **Open questions** — `tasks/open-questions.md` (Q-003..Q-005, deferred).

## Repository layout

```
mimic/
├── backend/        # Flask + SQLAlchemy + Alembic + Jinja sandbox + RBAC + CLI
├── frontend/       # Vite + React 19 + TS strict + Tailwind 4 + TanStack Query 5
├── docs/           # This file (architecture). ADRs land in tasks/spec-decisions.md.
├── tasks/          # Sprint backlog (todo.md), decisions, open questions, lessons.
├── CHANGELOG.md    # Keep-a-Changelog flavoured.
└── README.md       # Entry point + status + stack.
```

Deployment artifacts (Ansible playbook, prod compose) live outside the repo
in the RT infra repo (D-010). Mimic ships only Dockerfiles and a dev
`docker-compose.yml`.

## Backend module tree

```
backend/src/mimic/
├── app.py            # Flask app factory (register blueprints, extensions, error handlers)
├── config.py         # Env-driven settings (no hardcoded secrets — NF-network)
├── extensions.py     # db (Flask-SQLAlchemy 3), login_manager, …
├── logging.py        # JSON structured logger
│
├── db/
│   ├── base.py             # Declarative Base + UuidPkMixin + TimestampsMixin
│   ├── types.py            # Python enums mirrored to Postgres ENUMs
│   ├── models/             # SQLAlchemy 2 typed mapped classes (§8 aggregates)
│   └── migrations/         # Alembic env + initial schema (202605210001)
│
├── rbac/
│   ├── matrix.py           # Permission enum + GROUP_PERMISSIONS (F11 source of truth)
│   └── decorators.py       # @require_perm Flask decorator
│
├── auth/
│   ├── identity.py         # current_user wiring (Flask-Login)
│   ├── password.py         # bcrypt helpers
│   └── soc_token.py        # 256-bit url-safe opaque tokens, bcrypt-hashed (D-006)
│
├── audit/
│   └── log.py              # Append-only writer + hash-chain (D-013)
│
├── templating/
│   ├── sandbox.py          # Jinja2 SandboxedEnvironment
│   └── filters.py          # regex_extract (google-re2, raise on no-match — D-011)
│
├── storage/
│   └── blob.py             # CAS sha256 + gzip pool (MIMIC_BLOB_ROOT — D-012)
│
├── connectors/
│   ├── base.py             # C2Connector ABC + dataclasses (Payload, TaskHandle, TaskResult)
│   ├── factory.py          # Factory keyed on engagement/scenario c2_type
│   └── payload_map.py      # payload_type → native command (Mythic populated, Home stub)
│
├── api/                    # Flat CRUD blueprints (sprint 0 — no orchestration yet)
│   ├── engagements.py
│   ├── hosts.py
│   ├── ttps.py
│   └── scenarios.py        # Enforces F3 invariant: host.c2_type == scenario.c2_type
│
├── schemas/                # Pydantic v2 schemas (request/response validation)
│
└── cli/                    # mimic-cli (Click)
    ├── db.py               # migrate / seed / dump / restore (NF-state R-O1)
    └── user.py             # user create
```

## Frontend module tree

```
frontend/
├── src/
│   ├── main.tsx            # Vite entry, mounts <App />
│   ├── App.tsx             # Router root
│   ├── routes/             # Role-aware route definitions
│   ├── layout/             # Shell (sidebar, header, role-conditional menus)
│   ├── components/         # Wireframe components on mock data (F0.3)
│   ├── theme/              # Tailwind tokens (dark-first), Logo placeholder
│   └── lib/                # TanStack Query client + helpers
├── playwright.config.ts    # E2E skeleton (no real auth wired sprint 0)
└── vite.config.ts
```

## Persistence — §8 aggregates

| Aggregate | Notes vs spec |
|---|---|
| `user` | §8 + `display_name`, `last_login_at` (bonus, OPSEC R-O3). |
| `permission`, `group`, `group_permission`, `user_group` | RBAC layout (D-003, D-008). 3 groups seeded by migration (`rt_operator`, `rt_lead`, `soc_analyst`). |
| `engagement` | §8 + free-text `description`. `c2_type` default = `mythic`. |
| `engagement_member` | Role is a free `String(40)` — see "Known WARN" below. |
| `c2_credential` | Non-spec aggregate, arbitrated D-004 (Fernet-encrypted, versioned, rotation = insert + retire). |
| `host` | §8 verbatim. `c2_type` must match its scenario at run start (F3). |
| `ttp` | §8 + `is_stealth_variant` (R-O2 marker stripping) + `is_published` (TTP_PROMOTE F11). **No `ttp_version` table** (D-009 / H32). |
| `scenario`, `scenario_step` | §8 verbatim. `(scenario_id, order_idx)` unique. `c2_type` carried on scenario (H33). |
| `run` | `snapshot_json` (JSONB) is the **single** replay source (H32). |
| `run_step` | §8 + `order_idx`, `resolved_payload_text` (final payload with OPSEC marker — H34, audit-friendly). |
| `run_step_cleanup` | 1-1 with `run_step` via `UNIQUE(run_step_id)`. Status enum `pending/success/failed/partial` (F15, R-T5). |
| `detection`, `evidence` | §8 verbatim. |
| `report` | `content_sha256` referenced in PDF footer + JSON + MD (H19, H24). |
| `soc_session` | `token_opaque` renamed `token_hash` (bcrypt — D-006). Bonus `last_used_at`. |
| `audit_log` | §8 + `prev_hash`/`row_hash` (D-013 — chain stored from v1, verifier in v2) + `source_ip`/`user_agent`/`comment` (forensic). |

### Postgres-level OPSEC

- Audit append-only enforced at SQL level: `mimic_audit_writer` role gets
  `INSERT` only on `audit_log`; `UPDATE/DELETE/TRUNCATE` revoked from `PUBLIC`.
  Idempotent grants in the migration; the deployment playbook (Ansible)
  creates the roles (D-002, D-007, D-010).
- Hash chain (D-013): every row stores `row_hash = sha256(prev_hash || ts ||
  actor_id || action || resource_type || resource_id || metadata_json)`. The
  verifier is **not** wired sprint 0; columns and writer logic ship so v2
  enables enforcement without a destructive migration.

## RBAC — F11 mirrored as code

`backend/src/mimic/rbac/matrix.py` is the canonical permission map. Spec F11
table is read 1:1 into `GROUP_PERMISSIONS`. The migration seeds exactly three
groups (D-008):

| Group | Permission count | Notes |
|---|---|---|
| `rt_operator` | 10 | Includes `ENGAGEMENT_READ` (scope `(assignés)` to be applied at endpoint level). |
| `rt_lead` | All (~21) | `ALL_PERMISSIONS`. |
| `soc_analyst` | 3 | `ENGAGEMENT_READ_OWN`, `DETECTION_ADD`, `REPORT_READ`. |

Two F11 cells got a finer split (no semantic drift):

- `RUN_START` ∥ `RUN_CONTROL` — both lead-only, sum equivalent to F11 "Démarrer / contrôler".
- `ENGAGEMENT_READ` (RT, full list) ∥ `ENGAGEMENT_READ_OWN` (SOC, own session scope).

Decorator: `@require_perm(Permission.X)` on every Flask view. `current_user`
resolved by Flask-Login (local password v1) or future Keycloak claim mapping
(v2). SOC analysts authenticate through a separate token-based middleware
(see "Auth" below).

## Authentication

Two flows live side-by-side (D-003):

- **RT operators / leads** — username + bcrypt password (v1) + Flask
  server-side session. v2: OIDC Keycloak claim-to-group mapping, **no app
  code change** (the RBAC tables already accept any group name).
- **SOC analysts** — opaque 256-bit URL-safe tokens (`secrets.token_urlsafe(32)`),
  bcrypt-hashed in `soc_session.token_hash`, plain token returned **once** in
  the API response, delivered out-of-band (D-006). Scope: one engagement.
  Revocation = `revoked_at` set; immediate effect via DB check.

Mimic itself listens on localhost; HTTPS, TLS, and IP allowlisting are owned
by the existing RT Caddy reverse proxy (D-007). Mimic-side: no HSTS, no cert
mgmt.

## Cleanup templating (F15)

Jinja2 SandboxedEnvironment in `templating/sandbox.py` with two custom
accessors (D-005):

- `{{ outputs.text }}` — pulls `run_step.output_text` (stdout, UTF-8 with
  latin-1 fallback, silent refusal on non-decodable).
- `{{ outputs.blob("<key>") }}` — pulls a blob from `MIMIC_BLOB_ROOT`, hard
  cap 10 MB.

Custom filter `regex_extract(text, pattern, *, group=1, name=None)` —
google-re2 (no backrefs, linear time), first match only, **raises** on
no-match (D-011). Templating drift fails loudly at step run.

Resolved command lands in `run_step_cleanup.resolved_command_text` (the
literal sent to the C2) and `run_step.resolved_payload_text` for the
payload itself (audit + NF-OPSEC marker visibility).

## C2 abstraction

```
              ┌────────────────────────────────────────────┐
              │ orchestrator (sprint > 0)                   │
              │   start_step(run_step) → polling 500 ms     │
              └────────────────────────────────────────────┘
                        │ uses
                        ▼
              ┌────────────────────────────────────────────┐
              │ connectors.factory  (keyed on c2_type)      │
              └────────────────────────────────────────────┘
                        │ instantiates
                        ▼
       ┌──────────────────────┐         ┌──────────────────────┐
       │ MythicConnector       │         │ HomeConnector         │
       │ (PR1 — pending docs) │         │ (PR2 — stub          │
       │ Mythic GraphQL+REST   │         │   NotImplementedError)│
       └──────────────────────┘         └──────────────────────┘
                        │
              authenticate / list_hosts / execute_task /
              get_task_result / cancel_task / execute_cleanup
              (stream_task_output optional v1, exploited v2)
```

`payload_type` is a neutral internal enum (§7 of spec). Mapping to native
commands lives in `connectors/payload_map.py` — Mythic populated, Home empty
(blocked by PR2). `UnsupportedPayloadType` raised on miss → UI surfaces
"incompatible C2".

## Storage — file pools

Two filesystem pools (D-012):

```
$MIMIC_BLOB_ROOT     ── content-addressed (CAS) + gzip
  └── <aa>/<bb>/<sha256>.gz       run_step.output_blob_ref → <sha256>

$MIMIC_EVIDENCE_ROOT ── flat per engagement
  └── <engagement_id>/<evidence_id>.<ext>
```

Per-blob cap 10 MB. No global quota v1 — OS-level monitoring (node_exporter).
F12 archival CLI will own retention (post-sprint-0).

## Sprint 0 happy-path flow (current scope)

```
RT operator logs in              ── auth/identity (bcrypt + Flask session)
        │
        ▼
GET /api/v1/engagements           ── api/engagements:list_engagements
                                     @require_perm(ENGAGEMENT_READ)
                                     [WARN: scope (assignés) not applied — see below]
        │
        ▼
POST /api/v1/engagements          ── creates draft engagement
POST /api/v1/engagements/:id/hosts── seeds host inventory (manual v1)
POST /library/ttps                ── creates TTP draft
POST /engagements/:id/scenarios   ── composes scenario (c2_type fixed at create)
POST /engagements/:id/scenarios/:sid/steps ── adds ordered steps
        │
        ▼
[orchestration, F15 cleanup, F7 cotation, F9 report : sprint > 0]
```

WebSocket cockpit, run orchestrator, cleanup wiring, report renderer, OIDC,
and the two real C2 connectors are all **post-sprint-0**.

## Known WARN — to revisit later

- **`audit_log` chain has no runtime verifier.** Columns and write logic
  ship per D-013, but tampering detection is v2. Until then, the chain is a
  **forensic** trail (replay offline), not an **enforcement** trail. Owner:
  whoever picks up the H30 v2 ticket.
- **`engagement_member.role` is `String(40)`** — free text, no enum. Risk:
  future drift. Watch when implementing F11 enforcement on the
  `member.manage` endpoints.
- **`GET /engagements` ignores the `(assignés)` scope** — `@require_perm`
  alone admits any rt_operator. Scope-applicative check
  (`engagement_member` join) is a code-reviewer item, flagged MAJOR by
  team-lead. Sprint 0 leaves the endpoint flat by design; F11 closure ships
  with that fix.
- **Q-003 / Q-004 / Q-005 deferred** — see `tasks/open-questions.md`. None
  block sprint 0; each carries a `re-open when …` trigger.

## Decisions anticipated vs v2 (for future-me)

| Sprint 0 ships | Spec said | Why |
|---|---|---|
| `audit_log.prev_hash` / `row_hash` columns + chained writer | H30 puts hash chain in v2 | D-013 — adding columns later is a destructive migration; verifier stays v2. |
| `c2_credential` table (versioned, retiring) | Spec §8 omits it | D-004 — separating Fernet-encrypted blobs from the application engagement metadata is safer than embedding `config_json`. |
| Two storage pools (`blobs/` CAS + `evidence/` flat) | H20 says "local disk v1" | D-012 — split keeps deduplication for C2 outputs and clean archival for evidence; OS-level quota only. |
| Group-based RBAC tables from day 1 | F11 lists fixed roles | D-003 + D-008 — preserves F11 semantics exactly while making OIDC v2 a config change, not a code change. |

## Pointers

- Frozen spec: `RT-SecondBrain/Projects/Mimic — Spec.md` (vault).
- Decisions log: `tasks/spec-decisions.md` (D-001..D-014).
- Open questions: `tasks/open-questions.md` (Q-003..Q-005 deferred).
- Sprint 0 backlog: `tasks/todo.md`.
- Changes journal: `CHANGELOG.md`.