mimic-big/docs/deploy.md

# Mimic — production deployment

Operational guide for rolling Mimic out on the RT infrastructure. Scope is
the **application repo only** — the Ansible playbook that automates the
host preparation lives in the separate RT infra repository (D-010), and
the Caddy reverse proxy is owned by the RT platform (D-007). This document
references both without duplicating them.

For CI/runner setup, see [`docs/podman-runner-setup.md`](./podman-runner-setup.md).
For architectural context, see [`docs/architecture.md`](./architecture.md).

## Audience

Whoever pushes a new Mimic version to production. Assumes familiarity with
Podman rootless, systemd user units, and PostgreSQL DSN syntax.

## Host prerequisites

| Component | Version | Notes |
| --- | --- | --- |
| OS | Linux x86_64 | Tested on Debian 12 and Fedora 41. SELinux-aware. |
| Podman | ≥ 5.0 | Rootless mode mandatory. Verify with `podman info --format '{{.Host.Security.Rootless}}'` returns `true`. |
| systemd | user mode | `loginctl enable-linger <mimic-user>` so user services survive logout. |
| PostgreSQL | 16 | Reachable from the Mimic container. Local socket fine; networked instance fine. |
| Reverse proxy | Caddy (out-of-Mimic) | Provides TLS, IP allowlist, and SOC session token plumbing. Configured in the RT infra repo. |

The deployment user (referred to as `<mimic-user>` below) is typically a
dedicated `mimic` system account. Reusing the `gitea` user is acceptable
for single-tenant hosts but not recommended in multi-app scenarios.

## Filesystem layout

| Path | Owner | Mode | Purpose |
| --- | --- | --- | --- |
| `/var/lib/mimic/blobs` | `<mimic-user>:<mimic-user>` | `0750` | Content-addressed C2 output blobs (D-012). Default for `MIMIC_BLOB_ROOT`. |
| `/var/lib/mimic/evidence` | `<mimic-user>:<mimic-user>` | `0750` | User-uploaded evidence (F8). Default for `MIMIC_EVIDENCE_ROOT`. |
| `/var/log/mimic` | `<mimic-user>:<mimic-user>` | `0750` | Application logs if file-logging is enabled. JSON to stdout by default. |
| `~<mimic-user>/.config/containers/systemd/` | `<mimic-user>` | `0700` | Quadlet units for the backend + frontend containers. |

The Ansible playbook in the RT infra repo creates these paths with the
correct permissions. Manual provisioning equivalent:

```bash
sudo install -d -o <mimic-user> -g <mimic-user> -m 0750 \
  /var/lib/mimic/blobs /var/lib/mimic/evidence /var/log/mimic
```

## Environment variables

Loaded from the systemd unit `Environment=` directives or a separate
`.env` file mounted into the container. All variables are prefixed
`MIMIC_` (Pydantic Settings convention, see `backend/src/mimic/config.py`).

### Required in production

| Variable | Example | Effect |
| --- | --- | --- |
| `MIMIC_ENV` | `production` | Switches default cookie / log behaviour. |
| `MIMIC_SECRET_KEY` | `$(python -c 'import secrets; print(secrets.token_urlsafe(32))')` | Flask session cookie HMAC. Rotating it invalidates every live session — schedule a maintenance window. |
| `MIMIC_FERNET_KEY` | `$(python -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())')` | Symmetric key encrypting `c2_credential.config_json_fernet`. **Required** in prod. `Fernet(b"")` would crash on first credential decrypt; the empty default in `config.py` exists only so tests can boot. |
| `MIMIC_DATABASE_URL` | `postgresql+psycopg://mimic_app:<pw>@postgres:5432/mimic` | Main app DSN. The role behind it must NOT have `INSERT` on `audit_log` (NF-AUDIT append-only contract). |
| `MIMIC_DATABASE_AUDIT_URL` | `postgresql+psycopg://mimic_audit_writer:<pw>@postgres:5432/mimic` | Write-only DSN used by the audit writer. The role has `INSERT` on `audit_log` and nothing else. See [Bootstrap the audit role](#bootstrap-the-audit-role). |

### Required with safe defaults

| Variable | Default | Comment |
| --- | --- | --- |
| `MIMIC_BLOB_ROOT` | `/var/lib/mimic/blobs` | Override only if the data partition lives elsewhere. |
| `MIMIC_EVIDENCE_ROOT` | `/var/lib/mimic/evidence` | Same. |
| `MIMIC_SESSION_COOKIE_SECURE` | `true` | Must stay `true` behind Caddy/TLS. Set `false` only for the dev compose. |
| `MIMIC_SESSION_COOKIE_SAMESITE` | `Lax` | `Strict` if SOC console is on the same eTLD+1 as Mimic. |
| `MIMIC_LOG_LEVEL` | `INFO` | `DEBUG` is verbose, do not enable in prod without a reason. |
| `MIMIC_LOG_JSON` | `true` | Required for log shipping. Disable only for human debugging. |
| `MIMIC_CORS_ORIGINS` | `[]` (none) | Set to the public Mimic URL if frontend and backend are served from different origins. |

### Never set in production

`MIMIC_DATABASE_URL` and `MIMIC_DATABASE_AUDIT_URL` must point to two
different roles. Pointing them at the same role defeats the audit
append-only guarantee — caught by code review N5 (see
`tasks/todo.md` § CI follow-ups).

## Secrets management

Three secrets must never appear in container images, git history, or
agent transcripts: `MIMIC_SECRET_KEY`, `MIMIC_FERNET_KEY`, and the
PostgreSQL password embedded in the two DSNs.

Recommended flow (matches the team-wide "secrets via file, not chat"
convention):

1. Generate secrets once per environment on the deploy host:

   ```bash
   umask 077
   install -d -m 0700 ~/secrets
   python -c 'import secrets; print(secrets.token_urlsafe(32))' > ~/secrets/SECRET_KEY
   python -c 'from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())' > ~/secrets/FERNET_KEY
   ```

2. Reference the files from the systemd unit via `EnvironmentFile=` (one
   `KEY=VALUE` per line) **or** mount them as in-container files and
   read them with `MIMIC_FERNET_KEY_FILE` equivalent indirection. Today
   the app reads `MIMIC_FERNET_KEY` directly; the file-based path is
   tracked as a follow-up.

3. Back up the secret material to the RT password vault, not anywhere
   else. Losing `FERNET_KEY` after C2 credentials are persisted means
   the data is permanently unreadable (no recovery key by design).

4. Rotating `MIMIC_FERNET_KEY` requires a re-encryption pass over
   `c2_credential.config_json_fernet`. The Ansible playbook ships a
   maintenance task for it; it is not exposed in the application CLI.

## Container images

| Component | Image | Tag policy |
| --- | --- | --- |
| Backend | `backend/Dockerfile`, built and pushed by CI | Pin `:X.Y.Z` per release. Never `:latest` in prod (follow-up F-D1). |
| Frontend | `frontend/Dockerfile`, built and pushed by CI | Same policy. Served by `nginxinc/nginx-unprivileged:alpine` listening on 8080. |
| PostgreSQL | `postgres:16-alpine` | Pin a minor tag (`16.4-alpine`) in production compose. |

The backend image listens on **5000** as user `mimic` (uid 1001). The
frontend image listens on **8080** as user `nginx` (uid 101).

## PostgreSQL setup

The application user (`mimic_app`) is created by the Ansible playbook
with `LOGIN` and ownership over the application database. It does **not**
get `INSERT` on `audit_log` — that grant goes to a separate role, see
below.

### Bootstrap the audit role

`mimic_audit_writer` exists to enforce the NF-AUDIT append-only contract.
The Alembic baseline migration grants `INSERT ON audit_log` to this role
if it exists, idempotently. Create the role before running migrations
(the Ansible playbook does this; manual equivalent):

```sql
-- run as a Postgres superuser, against the mimic database
CREATE ROLE mimic_audit_writer LOGIN PASSWORD '<paste-from-vault>';
```

Then expose its DSN as `MIMIC_DATABASE_AUDIT_URL`. The application boots
even if the role is missing (the grant block is a no-op), but every
audit write will fail at runtime — fail-loud preferred over silent data
loss.

### Apply migrations

The backend container runs Alembic at startup via its entrypoint, against
the `MIMIC_DATABASE_URL` DSN. To apply manually:

```bash
podman exec -it mimic-backend alembic upgrade head
```

A schema downgrade (rollback procedure below) uses the same surface in
reverse.

## Quadlet units

Both containers run under the `<mimic-user>` systemd user instance via
Quadlet. Example backend unit
(`~<mimic-user>/.config/containers/systemd/mimic-backend.container`):

```ini
[Unit]
Description=Mimic backend
After=network-online.target

[Container]
Image=registry.try2get.in/mimic-backend:X.Y.Z
ContainerName=mimic-backend
PublishPort=127.0.0.1:5000:5000
EnvironmentFile=%h/secrets/mimic-backend.env
Volume=/var/lib/mimic/blobs:/var/lib/mimic/blobs:Z
Volume=/var/lib/mimic/evidence:/var/lib/mimic/evidence:Z

[Service]
Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target
```

Frontend unit is structurally identical, listening on `127.0.0.1:8080`.
Caddy fronts both. Activation:

```bash
systemctl --user daemon-reload
systemctl --user enable --now mimic-backend.service mimic-frontend.service
```

The reverse proxy configuration on Caddy (out-of-Mimic) terminates TLS
and forwards `https://<mimic-domain>/api/*` → `127.0.0.1:5000`, every
other path → `127.0.0.1:8080`.

## Smoke validation

Once the stack is up:

```bash
# From the deploy host, behind Caddy
curl -fsS https://<mimic-domain>/healthz
# → "ok"

# Direct to the backend (should not be reachable externally — sanity)
curl -fsS http://127.0.0.1:5000/healthz
# → "ok"

# Verify audit role is wired
podman exec -it mimic-backend python -c 'from mimic.config import get_settings; \
    print(get_settings().database_audit_url is not None)'
# → True
```

If any of these fail, do **not** announce the release. Investigate via
`journalctl --user -u mimic-backend.service -e`.

## Upgrade procedure

Steady-state release flow:

1. CI builds `mimic-backend:X.Y.Z` and `mimic-frontend:X.Y.Z` and pushes
   them to `registry.try2get.in`. The tag policy is the same as the
   sprint 0 follow-up F-D1.
2. Update the Quadlet `.container` files on the deploy host to point at
   the new tags (single line each).
3. `systemctl --user daemon-reload`.
4. `systemctl --user restart mimic-backend.service mimic-frontend.service`.
   Quadlet pulls the new image automatically.
5. Run smoke validation. Tail logs for one minute.

If the release ships schema changes, Alembic runs `upgrade head` on
container start — the migration is the **first** thing the entrypoint
does. A failed migration prevents the new container from accepting
traffic and leaves the previous container's exit code visible in
`journalctl`.

## Rollback procedure

A rollback covers both image and schema. The schema rollback is
optional and only required when the new release includes a non-additive
migration.

```bash
# Image-level rollback only (additive schema, no data shape change)
sed -i 's|Image=.*mimic-backend:.*|Image=registry.try2get.in/mimic-backend:<previous>|' \
  ~/.config/containers/systemd/mimic-backend.container
systemctl --user daemon-reload
systemctl --user restart mimic-backend.service

# Schema-affecting rollback
podman exec -it mimic-backend alembic downgrade <previous-revision>
# then image rollback as above
```

Always confirm the target Alembic revision matches the previous image's
shipped revision before downgrading — there is no enforcement and a
mismatch is recoverable but unpleasant.

## Open items captured in `tasks/todo.md`

- `FERNET-KEY` (CI follow-ups) — provision `FERNET_KEY_TEST` Gitea secret
  for CI so integration tests can exercise the encrypted-credential path.
- `F-D1` (Frontend follow-ups) — pin every production image by minor +
  digest. This document already mandates the policy; F-D1 is the
  implementation step.
- `F-D2` (Frontend follow-ups) — decide whether Caddy or the in-image
  `HEALTHCHECK` owns liveness probing. Currently neither is wired.
- `F-D3` — security response headers ownership (Caddy vs nginx.conf).