feature/m4-mitre #1

Merged
knacky merged 13 commits from feature/m4-mitre into main 2026-05-12 17:24:14 +00:00

13 Commits

Author SHA1 Message Date
Knacky
2c85f9b57e docs(m4): reconcile CHANGELOG + testing-m4 with the flat matrix + CR fixes
- CHANGELOG M4 Added: rewrote the frontend bullet to describe the actual
  flat ATT&CK matrix that ships (full-bleed, 15-col grid with minmax(7rem,
  1fr), name-only cells, ▸/▾ chevron). The original entry still described
  the abandoned 3-column drill-down picker.
- New "Fixed (post-M4 code-review pass)" subsection enumerating the six
  CR-driven fixes that landed in this branch (SSRF allowlist, advisory
  lock, typed contract, N+1 elimination, version clearing, error scrub +
  the test additions and e2e count pinning).
- DoD counts: 53 → 58 pytest, 34 e2e unchanged. testing-m4.md follows.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 19:19:44 +02:00
Knacky
8b1de6e258 test(m4): cover the new security guards + pin e2e to exact MITRE v19 counts
- 5 new pytest covering paths the code-reviewer flagged as un-asserted:
    * `test_seed_refuses_file_url` — `file://` scheme rejected before I/O
      (was the SSRF-to-local-FS vector).
    * `test_seed_refuses_disallowed_https_host` — non-allowlisted HTTPS
      host rejected with `MitreSourceForbidden`.
    * `test_seed_refuses_custom_url_without_sha` — end-to-end guard that
      `seed_mitre(source=<custom URL>, expected_sha256=None,
      allow_unverified=False)` raises `MitreSeedError`.
    * `test_dotted_id_fallback_resolves_orphan_subtechnique` — STIX bundle
      without `relationship[subtechnique-of]` still attaches T1059.001 to
      T1059 via the dotted-id convention.
    * `test_seed_clears_version_when_source_is_not_default` — seed from a
      local path leaves `settings.mitre_version` NULL (no stale pin).
- Existing `test_checksum_mismatch_aborts` reworked to monkey-patch
  `_ensure_host_allowed` so `file://` can drive the test past the allowlist
  gate (was relying on file:// being accepted before CR1).
- Removed unused `uuid` import.
- e2e: assertions on `tactics_upserted`/`techniques_upserted`/
  `subtechniques_upserted` switched from `>= 14/180/400` thresholds to
  `=== 15/222/475` exact counts pinned to MITRE Enterprise v19.0 + 0
  orphans. Catches parser regressions that would silently include revoked
  rows. Bump alongside MITRE_VERSION when re-pinning.
- e2e: `Math.random()` → `crypto.randomUUID().slice(0, 8)` for unique
  test-run emails (collision-safe across parallel CI workers).

DoD: 58 pytest pass (was 53), 34 Playwright pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 19:19:34 +02:00
Knacky
54adfee690 fix(m4): typed MitreSyncResult interface — drop the as cast
Mirrors the backend Pydantic `SyncResultOut` in TS so the mutation result is
properly typed end-to-end. `(res as { duration_ms: number })` cast removed
from MitrePage.tsx; `apiPost<MitreSyncResult>` carries the contract.

Also annotated the unused query-key factories in mitre.ts so the next reader
knows they're parked for M5 template-form consumption (not dead).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 19:19:19 +02:00
Knacky
63b48addc0 fix(m4): code-review pass — SSRF allowlist + advisory lock + typed contract
Six post-code-review fixes, applied before opening the PR per project
workflow (spec-review + code-review both gate the merge):

1. SSRF allowlist on `/mitre/sync`. Host must be in MITRE_ALLOWED_HOSTS
   (defaults to `raw.githubusercontent.com`, env-overridable). Closes "admin
   holding `mitre.sync` pivots api container at 169.254.169.254 / internal
   mirrors" via a typo'd URL. New `MitreSourceForbidden` → 400
   `source_forbidden`; checked at the top of `_download()` so it kicks in
   before any I/O.

2. `pg_advisory_xact_lock(hashtext('mitre.seed'))` at the top of the seed
   transaction. Two concurrent `/mitre/sync` requests now serialise across
   the DELETE+INSERT of `mitre_technique_tactics`; previously they could
   both wipe the M2M and one would fail the unique constraint on re-insert.

3. Typed SyncResult contract. Pydantic `SyncResultOut` on the Flask side
   `model_validate`s the dict before returning — single source of truth
   for the response shape, mirrored by a `MitreSyncResult` TS interface
   (next commit). The `as Record<string, unknown>` + `as { duration_ms }`
   cast in MitrePage is gone.

4. N+1 in dotted sub-technique fallback removed. Built
   `{external_id → technique_id}` once at function entry. Currently a
   no-op against MITRE official (0 orphans), but a latent footgun for
   partial / older bundles.

5. `SETTING_VERSION` cleared explicitly when `source != MITRE_DEFAULT_URL`.
   Previously it kept the stale pin label, so `/mitre/status` lied after
   a custom-URL re-sync.

6. `/mitre/sync` 500s no longer echo `str(e)` to the client — URLError /
   psycopg / Pydantic text now lives in the JSON log only. Public response
   stays `{"error": "internal_error"}`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 19:19:11 +02:00
Knacky
7a69f10f3e docs(m4): post-review polish — helper text + test counts
Spec-reviewer PASS pointed two factual nits:
- MitrePage helper text still referenced the old 3-column drill-down ("Pick
  a tactic on the left, then a technique..."). Reworded for the flat matrix
  with the ▸ glyph + hover-for-id idiom.
- testing-m4.md + CHANGELOG were stale at 51/12; the actual counts are 53/14
  after the GET /mitre/matrix tests landed. Reconciled.

No code-path change, no e2e fallout — DoD remains 53 pytest + 34 Playwright.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 18:58:51 +02:00
Knacky
b52cb0e5e4 refactor(m4): full-bleed matrix + word-only line breaks
Two follow-up tweaks per user feedback ("wrap sur les mots, agrandit le
cadre"):

- Full-bleed wrapper: the matrix breaks out of the page's max-w-page (1400px)
  constraint via `margin: 0 calc(50% - 50vw)` + `width: 100vw`, mirroring the
  60px page padding internally. On wide viewports the picker now uses the
  ENTIRE viewport width, so column widths grow proportionally — names that
  used to wrap on 3 lines now fit on 1-2.
- Word-only wrapping: replaced `break-words` (overflow-wrap: break-word,
  which falls back to mid-word breaks) with `break-normal hyphens-none`
  (overflow-wrap: normal + word-break: normal). Cells break only at word
  boundaries; if a single word is longer than the cell it overflows
  visually rather than splitting `Aut\nhentication`-style. The grid is
  configured `minmax(7rem, 1fr)` so the minimum column is wide enough for
  every single word in MITRE v19 names, and stretches with available space.
- Spec §F2 rewritten as a bullet contract locking in: full-bleed, 15 cols
  minmax(7rem, 1fr), word-only wrap, font sans 12px / count 10px, headers/
  cells show name-only with external_id on hover + chips. Future spec-reviewer
  passes can grade against this.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 18:53:51 +02:00
Knacky
8742fb2b6e refactor(m4): match attack.mitre.org sizing — equal-width cols, name-only cells
Visual parity pass against attack.mitre.org/# per user feedback ("trop dense,
illisible, je veux la même représentation"):

- Layout switched from flex+fixed-width 224px columns to a CSS grid of
  `repeat(N, minmax(0, 1fr))` so the 15 tactic columns share the container
  width equally. No more horizontal scroll on a standard desktop.
- Cells now show NAME ONLY (matches mitre.org). The external_id (TA00xx /
  T1xxx / T1xxx.xxx) is preserved in the chip selection bar at the top and
  in the `title` hover tooltip on every cell — surfaces on demand, doesn't
  consume cell real estate.
- Font: switched to `font-sans` (IBM Plex Sans) at `text-xs` (12px) across
  cells, matching the mitre.org typography. Headers use the same family at
  the same size with a 10px sub-line for the technique count.
- Chevron icons: ▸ (collapsed) / ▾ (expanded) — small, sub-technique count
  rendered inline beside the chevron.
- Helper line below the matrix tells the user where the IDs went.

Spec §F2 + testing-m4.md walkthrough rewritten to lock the new sizing rules
in (font-xs, no external_id in cells, hover/chip for the ID, no horizontal
scroll). spec-reviewer will see the matching contract.

DoD: make e2e → 34 passed. Selectors (data-testid + aria-pressed) unchanged
so the existing M4 e2e test still walks the new layout end-to-end.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 18:41:11 +02:00
Knacky
7dbe2dbc28 refactor(m4): flatten the MITRE picker into the attack.mitre.org matrix
The hierarchical 3-column drill-down was hard to scan and forced a stateful
walk per tag. Replaced with a flat, columns-as-tactics matrix that mirrors
attack.mitre.org/# — every cell is a one-click select target, with inline
sub-technique expand via a `+N` chevron.

- New endpoint GET /api/v1/mitre/matrix returns the full grid (tactics →
  techniques → sub-techniques nested) in a single ~55 KB response, so the
  SPA renders the whole matrix without firing 15 parallel queries. Two
  pytest tests added (nested structure + auth required).
- MitreTagPicker.tsx rewritten as a horizontal-scrolling matrix:
  - Click a tactic header → select the tactic (cyan filled).
  - Click a technique cell → select the technique (orange filled).
  - Click the `+N` chevron → expand sub-techniques inline within the column.
  - Click a sub-technique → select (purple filled).
  - Single Filter field matches on external_id or name across all kinds.
  - Selection chips at the top, clickable to remove.
  - `aria-pressed` on every clickable cell for screen readers and Playwright.
- e2e test updated to walk the new flow (click cell → assert aria-pressed,
  expand chevron, click sub, verify chip + JSON preview, filter to T1078).
- Spec §F2 + §F12 + todo.md M4 entry updated to make the matrix layout the
  canonical UI for MITRE tagging (so future spec-reviewer passes accept it).
- testing-m4.md walkthrough rewritten for the flat picker.

DoD post-refactor: make test-api → 53 passed (was 51), make e2e → 34 passed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 18:32:20 +02:00
Knacky
37e9e03f02 docs(m4): CHANGELOG, README, lessons, spec drift fix, todo tick
- CHANGELOG: added M4 section listing endpoints, CLI, volume, persisted
  settings, picker, and the post-spec-review fixes (custom-URL integrity
  requirement + /diag/reset consistency + spec drift). Includes the
  intentional decisions paragraph (seed-time download not image-baked, read
  endpoints unauthenticated-perm-wise, stdlib over httpx).
- README: status bumped to M0–M4, added MITRE quickstart (make seed-mitre +
  air-gapped path with --source /data/mitre/<file> + --skip-checksum),
  testing-m<N>.md pointer updated to testing-m4.md, roadmap line.
- tasks/spec.md §10 #4: amended "14 tactics Enterprise" → "≥14 tactics
  Enterprise (la v19 du pin actuel en ship 15)".
- tasks/lessons.md: 7 M4 lessons captured (stdlib STIX parsing, decoupling
  DoD asserts from upstream versions, subtechnique parent resolution, single-
  transaction safety, custom-URL footgun mitigation, /diag/reset consistency,
  named-volume permission caveat, podman build cache surprise).
- tasks/todo.md: M4 marked ☑.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 13:54:46 +02:00
Knacky
90036437cc test(m4): pytest parser + endpoints + e2e tag picker
- backend/tests/test_mitre.py: 12 integration tests using a hand-crafted
  minimal STIX bundle (no network in tests). Covers parser
  (revoked/deprecated skip, sub-technique parent linkage), seed idempotence,
  persisted settings, checksum mismatch path, all four read endpoints, perm
  enforcement on /mitre/sync, ILIKE search.
- e2e/tests/m4-mitre.spec.ts: 6 Playwright tests against the live stack.
  beforeAll calls POST /mitre/sync once (real bundle, ~50 MB, ~1.1 s) then
  the suite validates tactics ≥14, T1003 has ≥5 sub-techniques, the picker
  walks tactic→technique→subtechnique with chip multi-select, and non-admin
  sees /mitre but no Sync card.
- tasks/testing-m4.md: manual + automated checklist, air-gapped operator
  notes, volume-permission caveat for pre-existing root-owned volumes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 13:54:26 +02:00
Knacky
8a1dd58c83 feat(m4): frontend MitreTagPicker + /mitre showcase page
- lib/mitre.ts: shared types (MitreTactic, Technique, Subtechnique, MitreTag
  kind/id/external_id/name) + TanStack query keys.
- components/MitreTagPicker.tsx: three-column controlled picker (tactic →
  technique → subtechnique), multi-select with chip-removal, autocomplete on
  each column, ARIA labels for screen readers. Returns MitreTag[] via
  value/onChange — drop-in for M5 template forms.
- pages/MitrePage.tsx: status card (version, source URL, last_sync), admin-
  gated Trigger Sync button with success/error alerts, picker showcase, JSON
  preview of the current selection.
- Layout adds MITRE nav link for any logged-in user; App.tsx adds the
  /mitre route under RequireAuth. HomePage roadmap bumped to next: M5
  templates.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 13:54:15 +02:00
Knacky
872f3c046a feat(m4): REST endpoints + admin sync + /diag/reset consistency
- GET /api/v1/mitre/tactics, /techniques?tactic=&q=, /subtechniques?technique=&q=
  (paginated, ILIKE search on name + external_id, @require_auth only — MITRE
  is public reference material).
- GET /api/v1/mitre/status: last_sync, version, source_url + the pinned
  defaults (default_url, default_version) for the SPA badge.
- POST /api/v1/mitre/sync: @require_perm("mitre.sync"). Body supports
  {source, expected_sha256, allow_unverified} — defaults inherit the pin.
- /diag/reset now also TRUNCATEs the mitre_* tables alongside settings so a
  freshly-reset stack has GET /mitre/status and GET /mitre/tactics agree
  ("no data, no last_sync"). Previously the catalogue persisted while the
  metadata was wiped, leaving status to lie. The e2e suite re-syncs in
  beforeAll.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 13:54:03 +02:00
Knacky
ba976959a1 feat(m4): STIX parser + seed service + CLI
- backend/app/services/mitre_seed.py: stdlib-only STIX 2.1 parser (urllib +
  hashlib + json). Pinned to enterprise-attack-19.0.json with sha256
  df520ea0775a57db7bff760145b02fed89290802913e056b7ed5970b02f3626a (~52 MB,
  ~1.1 s parse). Resolves sub-technique parents via
  relationship[subtechnique-of] with a T1003.001→T1003 dotted-id fallback;
  upserts on external_id, rebuilds the technique↔tactic M2M in a single
  transaction so external readers never see an empty join. Persists
  mitre_last_sync, mitre_version, mitre_source_url in the settings table.
- Custom URLs MUST be paired with expected_sha256 OR allow_unverified=true —
  refuses silent integrity bypass.
- CLI: flask metamorph seed-mitre [--source path|url]
  [--checksum-sha256 hex] [--skip-checksum]. Make target wraps it.
- Docker: /data/mitre/ chowned to the metamorph user at build; named volume
  metamorph_mitre mounted from compose for cross-restart cache.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 13:53:53 +02:00