test(m5): playwright spec + docs (CHANGELOG, README, lessons, testing-m5)
- 4 Playwright tests: API CRUD round-trip, scenario reorder via PUT, SPA list + opsec filter, SPA scenario list rendering with ordered tests. - afterAll restores the stable admin (admin@metamorph.local) per the test_admin memory rule. - CHANGELOG M5 section + Fixed subsections for the LogRecord 'name' collision and the React `currentTarget` vs `target` quirk. - README status bumps to M0-M5. - tasks/lessons.md captures the new patterns (sentinel pattern for partial-update, FK ordering in /diag/reset, dnd-kit stable IDs). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
27
CHANGELOG.md
27
CHANGELOG.md
@@ -4,6 +4,33 @@ All notable changes to this project will be documented here. Format: [Keep a Cha
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Added — M5 (Test & scenario templates)
|
||||||
|
- **CRUD `test_templates`** (`app/services/test_templates.py` + `app/api/test_templates.py`):
|
||||||
|
- Fields: name, description, objective, procedure (markdown), prerequisites (markdown), expected result red, expected detection blue, OPSEC level (`low/medium/high`), free tags (TEXT[]), expected IOCs (TEXT[]).
|
||||||
|
- Polymorphic MITRE tag set (`(kind, external_id)` ↔ exactly one of `tactic_id`/`technique_id`/`subtechnique_id`). The wire payload uses ATT&CK external IDs — server resolves to UUIDs.
|
||||||
|
- Filters: `q` (LIKE on name/description), `tactic`/`technique`/`subtechnique` (joined via subquery on the polymorphic tag table), `opsec`, `tag` (array contains).
|
||||||
|
- REST: `GET /test-templates`, `GET /test-templates/{id}`, `POST /test-templates`, `PUT /test-templates/{id}` (partial, with explicit `_UNSET` sentinel so omitted fields stay untouched), `DELETE /test-templates/{id}` (soft).
|
||||||
|
- **CRUD `scenario_templates`** (`app/services/scenario_templates.py` + `app/api/scenario_templates.py`):
|
||||||
|
- Ordered list of test_templates with `position` (UNIQUE `scenario_template_id, position`).
|
||||||
|
- Reorder via full replace: `PUT /scenario-templates/{id}/tests` deletes the join rows and re-inserts at positions `0..N-1` — clean atomic op that respects the UNIQUE constraint without a 2-phase position shuffle.
|
||||||
|
- The same test can appear multiple times (chained operations).
|
||||||
|
- REST: `GET`/`POST`/`PATCH` (metadata) / `DELETE` (soft) on `/scenario-templates`.
|
||||||
|
- **Frontend**:
|
||||||
|
- `lib/templates.ts` — typed client + queryKey factory.
|
||||||
|
- `pages/AdminTestsPage.tsx` — list + filters (q, tactic, opsec, tag) + modal with full field set + embedded `<MitreTagPicker>` for tags.
|
||||||
|
- `pages/AdminScenariosPage.tsx` — list + modal with **@dnd-kit/sortable** vertical drag-and-drop on the ordered test list. New deps: `@dnd-kit/core`, `@dnd-kit/sortable`, `@dnd-kit/utilities`.
|
||||||
|
- `components/MarkdownField.tsx` — lean textarea with markdown hint (no heavy editor dep; rendering happens at display time in M7).
|
||||||
|
- Nav adds **Tests** and **Scenarios** links (admin-gated).
|
||||||
|
- **/diag/reset** truncates the 4 new tables before the MITRE block — the `scenario_template_tests.test_template_id` FK is `ON DELETE RESTRICT`, so the order matters.
|
||||||
|
- **Testing**:
|
||||||
|
- `backend/tests/test_templates.py` — **19 pytest** (create/list/filter by tactic+opsec+tag, MITRE tag resolution + replacement on update, soft-delete, perm gating, scenario create+reorder+delete, soft-deleted test linking semantics).
|
||||||
|
- `e2e/tests/m5-templates.spec.ts` — **4 Playwright** (API CRUD round-trip, scenario reorder, SPA list + opsec filter, SPA scenario list rendering with ordered tests).
|
||||||
|
- `tasks/testing-m5.md`.
|
||||||
|
|
||||||
|
### Fixed (M5 implementation)
|
||||||
|
- **`LogRecord` key collision**: `log.info(..., extra={"name": ...})` raises `KeyError("Attempt to overwrite 'name' in LogRecord")` because `name` is reserved by Python's stdlib logging. Renamed to `template_name`.
|
||||||
|
- **React `currentTarget` null in deferred state updaters**: `onChange={(e) => setX((prev) => ({ ...prev, q: e.currentTarget.value }))}` blanked the page on the first user input because `currentTarget` is cleared after the listener bubble ends, before React invokes the updater. Switched all M5 handlers to `e.target.value`, which persists on the synthetic event.
|
||||||
|
|
||||||
### Added — M4 (MITRE ATT&CK Enterprise)
|
### Added — M4 (MITRE ATT&CK Enterprise)
|
||||||
- **STIX 2.1 parser + upsert** (`app/services/mitre_seed.py`): stdlib-only (`urllib.request` + `hashlib`), pinned to Enterprise v19.0 (`enterprise-attack-19.0.json`, sha256 `df520ea0…`). Parses 25k+ STIX objects → 15 tactics, 222 techniques, 475 sub-techniques in ~1.1 s. Skips revoked + deprecated, resolves sub-technique parents via `relationship[subtechnique-of]` with a `T1003.001 → T1003` dotted-id fallback, copies kill-chain phases into the `mitre_technique_tactics` M2M.
|
- **STIX 2.1 parser + upsert** (`app/services/mitre_seed.py`): stdlib-only (`urllib.request` + `hashlib`), pinned to Enterprise v19.0 (`enterprise-attack-19.0.json`, sha256 `df520ea0…`). Parses 25k+ STIX objects → 15 tactics, 222 techniques, 475 sub-techniques in ~1.1 s. Skips revoked + deprecated, resolves sub-technique parents via `relationship[subtechnique-of]` with a `T1003.001 → T1003` dotted-id fallback, copies kill-chain phases into the `mitre_technique_tactics` M2M.
|
||||||
- **CLI**: `flask metamorph seed-mitre [--source <path|url>] [--checksum-sha256 <hex>] [--skip-checksum]` (`app/cli.py`). `make seed-mitre` wraps it.
|
- **CLI**: `flask metamorph seed-mitre [--source <path|url>] [--checksum-sha256 <hex>] [--skip-checksum]` (`app/cli.py`). `make seed-mitre` wraps it.
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
Collaborative purple-team platform. Red team logs the tests they execute (procedure, command, timestamp); blue team annotates each test with detection evidence (alerts, logs, files). At the end of an engagement, Metamorph generates a standalone reveal.js slide deck classified by MITRE ATT&CK tactic.
|
Collaborative purple-team platform. Red team logs the tests they execute (procedure, command, timestamp); blue team annotates each test with detection evidence (alerts, logs, files). At the end of an engagement, Metamorph generates a standalone reveal.js slide deck classified by MITRE ATT&CK tactic.
|
||||||
|
|
||||||
> **Status**: M0–M4 delivered (bootstrap → DB schema → auth → RBAC → MITRE ATT&CK reference). See `tasks/spec.md` for the full specification and `tasks/todo.md` for the milestone-by-milestone plan.
|
> **Status**: M0–M5 delivered (bootstrap → DB schema → auth → RBAC → MITRE ATT&CK reference → test & scenario templates). See `tasks/spec.md` for the full specification and `tasks/todo.md` for the milestone-by-milestone plan.
|
||||||
|
|
||||||
## Stack
|
## Stack
|
||||||
|
|
||||||
@@ -11,6 +11,7 @@ Collaborative purple-team platform. Red team logs the tests they execute (proced
|
|||||||
- **Auth (M2+)**: JWT access (1h) + refresh (30d), Argon2id, invite-link enrollment.
|
- **Auth (M2+)**: JWT access (1h) + refresh (30d), Argon2id, invite-link enrollment.
|
||||||
- **RBAC (M3+)**: atomic permissions (31 codes) bundled into custom groups; 3 system groups seeded (`admin` / `redteam` / `blueteam`).
|
- **RBAC (M3+)**: atomic permissions (31 codes) bundled into custom groups; 3 system groups seeded (`admin` / `redteam` / `blueteam`).
|
||||||
- **MITRE ATT&CK (M4+)**: Enterprise reference catalogue pinned to v19.0, seedable via `make seed-mitre`.
|
- **MITRE ATT&CK (M4+)**: Enterprise reference catalogue pinned to v19.0, seedable via `make seed-mitre`.
|
||||||
|
- **Template catalogue (M5+)**: reusable `test_templates` (markdown procedure, OPSEC level, free tags, expected IOCs, MITRE tags) + ordered `scenario_templates` with drag-and-drop reordering. Admin pages at `/admin/tests` and `/admin/scenarios`.
|
||||||
- **Delivery**: docker-compose. TLS termination is expected to be handled by an external reverse proxy in production.
|
- **Delivery**: docker-compose. TLS termination is expected to be handled by an external reverse proxy in production.
|
||||||
|
|
||||||
## Quickstart
|
## Quickstart
|
||||||
|
|||||||
253
e2e/tests/m5-templates.spec.ts
Normal file
253
e2e/tests/m5-templates.spec.ts
Normal file
@@ -0,0 +1,253 @@
|
|||||||
|
import { expect, test, type APIRequestContext, type Page } from '@playwright/test';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* M5 — Test + Scenario template catalogue.
|
||||||
|
*
|
||||||
|
* Verifies CRUD on /test-templates and /scenario-templates plus the admin SPA
|
||||||
|
* pages. We do NOT seed the full MITRE bundle here — M4 already covers that
|
||||||
|
* suite. This spec only needs ONE technique resolvable from a STIX-like
|
||||||
|
* shape (we ride on the same `/diag/reset` then re-seed MITRE so tag refs
|
||||||
|
* resolve).
|
||||||
|
*/
|
||||||
|
|
||||||
|
const ADMIN_EMAIL = `admin-${crypto.randomUUID().slice(0, 8)}@metamorph.local`;
|
||||||
|
const ADMIN_PASSWORD = 'AdminPass1234!';
|
||||||
|
|
||||||
|
async function resetAndMintToken(request: APIRequestContext): Promise<string> {
|
||||||
|
const r = await request.post('/api/v1/diag/reset');
|
||||||
|
expect(r.status()).toBe(200);
|
||||||
|
return (await r.json()).install_token as string;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loginAndGetAccess(
|
||||||
|
request: APIRequestContext,
|
||||||
|
email: string,
|
||||||
|
password: string,
|
||||||
|
): Promise<string> {
|
||||||
|
const r = await request.post('/api/v1/auth/login', { data: { email, password } });
|
||||||
|
expect(r.status()).toBe(200);
|
||||||
|
return (await r.json()).access_token as string;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loginViaSpa(page: Page, email: string, password: string) {
|
||||||
|
await page.goto('/login');
|
||||||
|
await page.getByLabel(/email/i).fill(email);
|
||||||
|
await page.getByLabel(/password/i).fill(password);
|
||||||
|
await page.getByRole('button', { name: /sign in/i }).click();
|
||||||
|
await expect(page.getByTestId('me-email')).toHaveText(email);
|
||||||
|
}
|
||||||
|
|
||||||
|
test.describe.configure({ mode: 'serial' });
|
||||||
|
|
||||||
|
test.describe('M5 — Template catalogue', () => {
|
||||||
|
test.beforeAll(async ({ request }) => {
|
||||||
|
const installToken = await resetAndMintToken(request);
|
||||||
|
const setup = await request.post('/api/v1/setup', {
|
||||||
|
data: { install_token: installToken, email: ADMIN_EMAIL, password: ADMIN_PASSWORD },
|
||||||
|
});
|
||||||
|
expect(setup.status()).toBe(201);
|
||||||
|
// MITRE re-sync — picker + tag refs rely on the canonical bundle.
|
||||||
|
const access = await loginAndGetAccess(request, ADMIN_EMAIL, ADMIN_PASSWORD);
|
||||||
|
const sync = await request.post('/api/v1/mitre/sync', {
|
||||||
|
headers: { Authorization: `Bearer ${access}` },
|
||||||
|
});
|
||||||
|
expect(sync.status()).toBe(200);
|
||||||
|
});
|
||||||
|
|
||||||
|
test.afterAll(async ({ request }) => {
|
||||||
|
// Restore the stable admin (cf. memory feedback_metamorph_test_admin):
|
||||||
|
// any wipe should leave admin@metamorph.local / AdminPass1234! usable.
|
||||||
|
const installToken = await resetAndMintToken(request);
|
||||||
|
await request.post('/api/v1/setup', {
|
||||||
|
data: {
|
||||||
|
install_token: installToken,
|
||||||
|
email: 'admin@metamorph.local',
|
||||||
|
password: 'AdminPass1234!',
|
||||||
|
},
|
||||||
|
});
|
||||||
|
// Re-seed MITRE so subsequent manual sessions don't see an empty matrix.
|
||||||
|
const access = await loginAndGetAccess(request, 'admin@metamorph.local', 'AdminPass1234!');
|
||||||
|
await request.post('/api/v1/mitre/sync', {
|
||||||
|
headers: { Authorization: `Bearer ${access}` },
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// === API smoke ============================================================
|
||||||
|
|
||||||
|
test('CRUD test-templates via API', async ({ request }) => {
|
||||||
|
const access = await loginAndGetAccess(request, ADMIN_EMAIL, ADMIN_PASSWORD);
|
||||||
|
const auth = { Authorization: `Bearer ${access}` };
|
||||||
|
|
||||||
|
// Create
|
||||||
|
const r1 = await request.post('/api/v1/test-templates', {
|
||||||
|
headers: auth,
|
||||||
|
data: {
|
||||||
|
name: 'phish-link',
|
||||||
|
description: 'send a phishing email with tracked link',
|
||||||
|
objective: 'land a click',
|
||||||
|
procedure_md: '1. craft mail\n2. send\n3. await click',
|
||||||
|
opsec_level: 'low',
|
||||||
|
tags: ['phish', 'initial-access'],
|
||||||
|
expected_iocs: ['phish@example.com'],
|
||||||
|
mitre_tags: [
|
||||||
|
{ kind: 'tactic', external_id: 'TA0001' },
|
||||||
|
{ kind: 'technique', external_id: 'T1566' },
|
||||||
|
],
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(r1.status(), await r1.text()).toBe(201);
|
||||||
|
const created = await r1.json();
|
||||||
|
expect(created.name).toBe('phish-link');
|
||||||
|
expect(created.mitre_tags.length).toBe(2);
|
||||||
|
expect(created.tags).toContain('phish');
|
||||||
|
|
||||||
|
// Update — partial: change opsec only
|
||||||
|
const r2 = await request.put(`/api/v1/test-templates/${created.id}`, {
|
||||||
|
headers: auth,
|
||||||
|
data: { opsec_level: 'high' },
|
||||||
|
});
|
||||||
|
expect(r2.status()).toBe(200);
|
||||||
|
const updated = await r2.json();
|
||||||
|
expect(updated.opsec_level).toBe('high');
|
||||||
|
expect(updated.name).toBe('phish-link'); // untouched
|
||||||
|
|
||||||
|
// List + filter by tactic
|
||||||
|
const r3 = await request.get('/api/v1/test-templates?tactic=TA0001', {
|
||||||
|
headers: auth,
|
||||||
|
});
|
||||||
|
expect(r3.status()).toBe(200);
|
||||||
|
const list = await r3.json();
|
||||||
|
expect(list.items.map((it: { name: string }) => it.name)).toContain('phish-link');
|
||||||
|
|
||||||
|
// Reject unknown MITRE
|
||||||
|
const r4 = await request.post('/api/v1/test-templates', {
|
||||||
|
headers: auth,
|
||||||
|
data: {
|
||||||
|
name: 'bad',
|
||||||
|
mitre_tags: [{ kind: 'technique', external_id: 'T9999' }],
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(r4.status()).toBe(400);
|
||||||
|
expect((await r4.json()).error).toBe('unknown_mitre_tag');
|
||||||
|
|
||||||
|
// Soft-delete
|
||||||
|
const r5 = await request.delete(`/api/v1/test-templates/${created.id}`, {
|
||||||
|
headers: auth,
|
||||||
|
});
|
||||||
|
expect(r5.status()).toBe(200);
|
||||||
|
const r6 = await request.get('/api/v1/test-templates', { headers: auth });
|
||||||
|
expect(
|
||||||
|
(await r6.json()).items.map((it: { name: string }) => it.name),
|
||||||
|
).not.toContain('phish-link');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('Scenario template: create + reorder + soft-delete', async ({ request }) => {
|
||||||
|
const access = await loginAndGetAccess(request, ADMIN_EMAIL, ADMIN_PASSWORD);
|
||||||
|
const auth = { Authorization: `Bearer ${access}` };
|
||||||
|
|
||||||
|
async function mkTest(name: string): Promise<string> {
|
||||||
|
const r = await request.post('/api/v1/test-templates', {
|
||||||
|
headers: auth,
|
||||||
|
data: { name },
|
||||||
|
});
|
||||||
|
expect(r.status()).toBe(201);
|
||||||
|
return (await r.json()).id as string;
|
||||||
|
}
|
||||||
|
|
||||||
|
const a = await mkTest('scn-step-a');
|
||||||
|
const b = await mkTest('scn-step-b');
|
||||||
|
const c = await mkTest('scn-step-c');
|
||||||
|
|
||||||
|
// Create with [a, b, c]
|
||||||
|
const r1 = await request.post('/api/v1/scenario-templates', {
|
||||||
|
headers: auth,
|
||||||
|
data: { name: 'ordered-scenario', test_template_ids: [a, b, c] },
|
||||||
|
});
|
||||||
|
expect(r1.status()).toBe(201);
|
||||||
|
const sc = await r1.json();
|
||||||
|
expect(sc.tests.map((t: { test_template_name: string }) => t.test_template_name)).toEqual([
|
||||||
|
'scn-step-a',
|
||||||
|
'scn-step-b',
|
||||||
|
'scn-step-c',
|
||||||
|
]);
|
||||||
|
|
||||||
|
// Reorder → [c, a, b]
|
||||||
|
const r2 = await request.put(`/api/v1/scenario-templates/${sc.id}/tests`, {
|
||||||
|
headers: auth,
|
||||||
|
data: { test_template_ids: [c, a, b] },
|
||||||
|
});
|
||||||
|
expect(r2.status()).toBe(200);
|
||||||
|
const after = await r2.json();
|
||||||
|
expect(after.tests.map((t: { test_template_name: string }) => t.test_template_name)).toEqual([
|
||||||
|
'scn-step-c',
|
||||||
|
'scn-step-a',
|
||||||
|
'scn-step-b',
|
||||||
|
]);
|
||||||
|
|
||||||
|
// Soft-delete the scenario.
|
||||||
|
const r3 = await request.delete(`/api/v1/scenario-templates/${sc.id}`, { headers: auth });
|
||||||
|
expect(r3.status()).toBe(200);
|
||||||
|
const list = await (await request.get('/api/v1/scenario-templates', { headers: auth })).json();
|
||||||
|
expect(list.items.map((it: { name: string }) => it.name)).not.toContain('ordered-scenario');
|
||||||
|
});
|
||||||
|
|
||||||
|
// === SPA smoke ============================================================
|
||||||
|
|
||||||
|
test('SPA — admin sees the test catalogue and can filter', async ({ page, request }) => {
|
||||||
|
// Seed two tests up front via the API — exercise the SPA list + filter
|
||||||
|
// pipeline without fighting the heavy create-modal (covered by API tests).
|
||||||
|
const access = await loginAndGetAccess(request, ADMIN_EMAIL, ADMIN_PASSWORD);
|
||||||
|
const auth = { Authorization: `Bearer ${access}` };
|
||||||
|
await request.post('/api/v1/test-templates', {
|
||||||
|
headers: auth,
|
||||||
|
data: { name: 'spa-list-fast', opsec_level: 'low', tags: ['fast'] },
|
||||||
|
});
|
||||||
|
await request.post('/api/v1/test-templates', {
|
||||||
|
headers: auth,
|
||||||
|
data: { name: 'spa-list-slow', opsec_level: 'high' },
|
||||||
|
});
|
||||||
|
|
||||||
|
await loginViaSpa(page, ADMIN_EMAIL, ADMIN_PASSWORD);
|
||||||
|
await page.goto('/admin/tests');
|
||||||
|
|
||||||
|
await expect(page.getByText('spa-list-fast')).toBeVisible();
|
||||||
|
await expect(page.getByText('spa-list-slow')).toBeVisible();
|
||||||
|
|
||||||
|
await page.getByTestId('filter-opsec').selectOption('high');
|
||||||
|
await expect(page.getByText('spa-list-slow')).toBeVisible();
|
||||||
|
await expect(page.getByText('spa-list-fast')).toBeHidden();
|
||||||
|
});
|
||||||
|
|
||||||
|
test('SPA — scenario list shows ordered tests with their position', async ({ page, request }) => {
|
||||||
|
// Seed a 3-test scenario via the API; the SPA must render the order as
|
||||||
|
// saved. Pointer-event drag is flaky in CI, and the API-level reorder
|
||||||
|
// test already covers the persistence pipeline.
|
||||||
|
const access = await loginAndGetAccess(request, ADMIN_EMAIL, ADMIN_PASSWORD);
|
||||||
|
const auth = { Authorization: `Bearer ${access}` };
|
||||||
|
const ids: string[] = [];
|
||||||
|
for (const name of ['drag-1', 'drag-2', 'drag-3']) {
|
||||||
|
const r = await request.post('/api/v1/test-templates', {
|
||||||
|
headers: auth,
|
||||||
|
data: { name },
|
||||||
|
});
|
||||||
|
ids.push((await r.json()).id);
|
||||||
|
}
|
||||||
|
const scResp = await request.post('/api/v1/scenario-templates', {
|
||||||
|
headers: auth,
|
||||||
|
data: {
|
||||||
|
name: 'spa-rendered-scenario',
|
||||||
|
test_template_ids: [ids[2], ids[0], ids[1]],
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const scId = (await scResp.json()).id;
|
||||||
|
|
||||||
|
await loginViaSpa(page, ADMIN_EMAIL, ADMIN_PASSWORD);
|
||||||
|
await page.goto('/admin/scenarios');
|
||||||
|
|
||||||
|
const card = page.locator(`[data-testid="scenario-row-${scId}"]`);
|
||||||
|
await expect(card).toBeVisible();
|
||||||
|
await expect(card.getByText('1. drag-3')).toBeVisible();
|
||||||
|
await expect(card.getByText('2. drag-1')).toBeVisible();
|
||||||
|
await expect(card.getByText('3. drag-2')).toBeVisible();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -78,6 +78,18 @@ project: Metamorph
|
|||||||
- **`podman compose stop api` puis `up -d api` casse les dépendances** entre containers (`db` healthy → `api` depends on it) : podman-compose ne résout pas la chaîne de deps quand on cible un seul service. Pour un override d'env, mieux vaut `make down && APP_ENV=test make up`.
|
- **`podman compose stop api` puis `up -d api` casse les dépendances** entre containers (`db` healthy → `api` depends on it) : podman-compose ne résout pas la chaîne de deps quand on cible un seul service. Pour un override d'env, mieux vaut `make down && APP_ENV=test make up`.
|
||||||
- **`/diag/reset` test-only** : exposer un endpoint qui truncate la DB est tentant pour les e2e mais ouvre une grosse surface en cas de fuite. Compromise actuel : autorisé en `dev` ET `test` (pas en prod), avec un log `WARNING` à chaque appel. Si jamais on déploie une stack dev publique, **désactiver** l'endpoint via env var.
|
- **`/diag/reset` test-only** : exposer un endpoint qui truncate la DB est tentant pour les e2e mais ouvre une grosse surface en cas de fuite. Compromise actuel : autorisé en `dev` ET `test` (pas en prod), avec un log `WARNING` à chaque appel. Si jamais on déploie une stack dev publique, **désactiver** l'endpoint via env var.
|
||||||
|
|
||||||
|
## 2026-05-12 — M5 templates + scenarios
|
||||||
|
|
||||||
|
- **`extra={"name": ...}` dans `log.info()` crash silencieusement** — Python's `logging.LogRecord` réserve `name` (le logger name). Coût : 500 sur le POST, message peu parlant (`KeyError: "Attempt to overwrite 'name' in LogRecord"`). Fix : renommer la clé (`template_name`). Liste réservée à éviter : `name`, `msg`, `args`, `levelname`, `levelno`, `pathname`, `filename`, `module`, `funcName`, `created`, `msecs`, `lineno`, `thread`, `threadName`, `process`. Pattern : préfixer les clés extra par l'entité (`template_name`, `group_id`, `user_id` est OK mais `id` aussi est piégeux dans certains setups).
|
||||||
|
- **React 18 + `setX((prev) => ({...prev, val: e.currentTarget.value }))` → page blanche au 1er input.** `e.currentTarget` est cleared après la fin du bubble, AVANT que l'updater fonctionnel exécute. Le synthetic event survit (pas de pooling depuis React 17), mais `currentTarget` est setté/cleared par le dispatcher. Fix : `e.target.value` (qui persiste sur le synthetic event), ou capturer `const v = e.currentTarget.value;` avant le `setX`. À garder en tête : tout `onChange` qui passe par un updater fonctionnel doit lire `e.target`, pas `e.currentTarget`.
|
||||||
|
- **Sentinel `Any = object()` plutôt que `... (Ellipsis)`** pour les "field unset" optional en service Python. Pyright voit `... = object()` correctement comme `Any`, alors que `description: str | None | object = ...` rend `description.strip()` invalide. Pattern : `_UNSET: Any = object()` au top du module + `description: Any = _UNSET` dans la signature + `if description is not _UNSET: ...`. Net + typecheck-friendly.
|
||||||
|
- **Postgres UNIQUE(scenario_id, position) + position-swap = ON CONFLICT pendant l'UPDATE.** Pour réordonner, le pattern naïf (UPDATE position) viole la contrainte sur le 1er swap. Trois options : (a) full delete + re-insert dans la même tx [retenu, atomique + lisible], (b) shift d'offset (UPDATE position = position + 1000 puis renumérotation), (c) deferred constraint. (a) gagne en simplicité — la liste rarement >50 éléments, le coût est négligeable.
|
||||||
|
- **`@dnd-kit/sortable` requires `useSortable({ id })` IDs to be unique and stable across renders.** Si on utilise un index numérique comme id, drag-and-drop ne réagit pas. Utiliser `test_template_id` (UUID stable) marche directement.
|
||||||
|
- **Frontend deps ajoutés à `package.json` sans `package-lock.json`** : le Dockerfile fait `npm install --no-audit --no-fund` sur fallback. OK pour M5 (3 deps `@dnd-kit/*`). À l'avenir, freeze un lockfile avant M14 pour build reproductibles.
|
||||||
|
- **Playwright `getByTestId` est défini par `testIdAttributeName: 'data-testid'`** dans `playwright.config.ts`. Pour qu'un test-id descende sur l'input via TextField, il faut que `...rest` soit spread sur l'input (déjà OK dans `TextField.tsx`). Mais avec un wrapper `<div><label/><input/></div>`, `getByTestId` matche le DIV si le test-id est dessus. Bien le mettre sur l'élément interactif (input/button), pas sur le container.
|
||||||
|
- **`/diag/reset` truncate order matters** : `scenario_template_tests.test_template_id` est FK `ON DELETE RESTRICT`, donc il faut truncate `scenario_template_tests` AVANT `test_templates`. Hierarchy : `scenario_template_tests → scenario_templates → test_template_mitre_tags → test_templates → mitre_*`. Maintenant inscrite dans `diag.py`.
|
||||||
|
- **Modal embarquant le `MitreTagPicker` complet (15 cols × 50 techniques)** : le picker se charge via `/mitre/matrix` (~94 KB). Affichage instantané, OK. Pour de futurs modals lourds, considérer le lazy-render derrière un toggle ou tab.
|
||||||
|
|
||||||
<!--
|
<!--
|
||||||
Template for future entries:
|
Template for future entries:
|
||||||
|
|
||||||
|
|||||||
131
tasks/testing-m5.md
Normal file
131
tasks/testing-m5.md
Normal file
@@ -0,0 +1,131 @@
|
|||||||
|
---
|
||||||
|
type: testing
|
||||||
|
milestone: M5
|
||||||
|
date: "2026-05-12"
|
||||||
|
project: Metamorph
|
||||||
|
---
|
||||||
|
|
||||||
|
# Testing M5 — Templates : tests unitaires & scénarios
|
||||||
|
|
||||||
|
## 1. Lancement de la stack
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make clean
|
||||||
|
make up
|
||||||
|
make migrate
|
||||||
|
make seed-mitre # tag picker needs the catalogue
|
||||||
|
```
|
||||||
|
|
||||||
|
> L'admin stable `admin@metamorph.local / AdminPass1234!` est restauré
|
||||||
|
> automatiquement par le hook `afterAll` du spec e2e M5 — mais la 1ʳᵉ fois,
|
||||||
|
> bootstrappe-le via `/setup` ou laisse les tests faire le travail.
|
||||||
|
|
||||||
|
## 2. Tests automatisés
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make test-api # 77 tests pytest dont 19 M5 (CRUD, perm, mitre tags, reorder)
|
||||||
|
make e2e # 38 tests Playwright dont 4 M5 (API CRUD + scenario reorder + SPA list/filter)
|
||||||
|
```
|
||||||
|
|
||||||
|
Rapport HTML : `e2e/playwright-report/`. JUnit : `e2e/playwright-report/junit.xml`.
|
||||||
|
|
||||||
|
## 3. Smoke navigateur
|
||||||
|
|
||||||
|
### Pré-requis
|
||||||
|
- Stack `make up` + admin loggé.
|
||||||
|
- MITRE seedé (vérifier via `/mitre`).
|
||||||
|
|
||||||
|
### 3.1 Catalogue de tests (`/admin/tests`)
|
||||||
|
1. Cliquer **Tests** dans la nav admin → page chargée.
|
||||||
|
2. Cliquer **+ New test** → modal s'ouvre avec :
|
||||||
|
- Champs : Name, Description, Objective, Procedure (markdown), Prerequisites, Red expected, Blue expected, OPSEC, Free tags, Expected IOCs.
|
||||||
|
- Sous-section **MITRE ATT&CK tags** : matrice complète, mêmes interactions que `/mitre`.
|
||||||
|
3. Remplir au minimum `Name=phish-link`, OPSEC=`low`, ajouter 2 tags MITRE (ex. `TA0001 + T1566`) → **Create** → carte apparaît dans la liste avec chips OPSEC + MITRE.
|
||||||
|
4. Cliquer **Edit** sur la carte → modal pré-remplie, modifier OPSEC à `high` → **Save** → la card est repeinte avec l'accent rouge OPSEC.
|
||||||
|
5. Filtres en haut :
|
||||||
|
- `Search` (full-text q sur nom/description)
|
||||||
|
- `Tactic external_id` (ex. `TA0001`)
|
||||||
|
- `OPSEC` (select : —all— / low / medium / high)
|
||||||
|
- `Free tag` (mot-clé libre)
|
||||||
|
6. Cliquer **Delete** sur une carte → confirm popup → la card disparaît (soft-delete : visible via `?include_deleted=true` côté API).
|
||||||
|
|
||||||
|
### 3.2 Catalogue de scénarios (`/admin/scenarios`)
|
||||||
|
1. Cliquer **Scenarios** dans la nav admin.
|
||||||
|
2. **+ New scenario** → modal.
|
||||||
|
- Champs Name + Description.
|
||||||
|
- Catalogue picker en bas : champ de recherche + liste des tests dispos (max 50).
|
||||||
|
3. Cliquer 3 tests dans le catalogue → ils s'empilent dans la liste ordonnée avec leurs indices `01/02/03`.
|
||||||
|
4. **Drag-and-drop** : empoigner la poignée `☰` à gauche d'une ligne et glisser vers le haut/bas → la liste se réordonne. La grille met à jour les indices au relâchement.
|
||||||
|
5. **Save** → carte apparaît avec un Tag « N tests » + l'aperçu des 4 premiers tests dans l'ordre choisi.
|
||||||
|
6. Re-ouvrir Edit → l'ordre est persisté côté serveur (vérifie le numéro 01, 02, 03 dans la modal).
|
||||||
|
7. Supprimer un `test_template` dont un scénario dépend (via `/admin/tests`) → la card scénario marque le test en rose dans le résumé (`test_template_deleted: true`).
|
||||||
|
|
||||||
|
### 3.3 Permissions
|
||||||
|
1. Inviter Bob via Admin > Invitations sans groupe → Bob peut se logger mais reçoit `403` sur `/api/v1/test-templates`.
|
||||||
|
2. Lui attacher un groupe avec seulement `test_template.read` → Bob voit `/admin/tests`... non, l'UI gate sur `is_admin`. La perm seule donne l'accès API ; l'UI ne l'expose pas pour les non-admins (par design M5).
|
||||||
|
3. Bob tente `POST /api/v1/test-templates` → `403` (manque `test_template.create`).
|
||||||
|
|
||||||
|
## 4. Smoke API
|
||||||
|
|
||||||
|
### 4.1 Login admin
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ACCESS=$(curl -sX POST http://localhost:8080/api/v1/auth/login \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d '{"email":"admin@metamorph.local","password":"AdminPass1234!"}' | jq -r .access_token)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.2 Créer un test taggué MITRE
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -sX POST http://localhost:8080/api/v1/test-templates \
|
||||||
|
-H "Authorization: Bearer $ACCESS" \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d '{
|
||||||
|
"name": "lsass-dump",
|
||||||
|
"opsec_level": "high",
|
||||||
|
"tags": ["creds"],
|
||||||
|
"mitre_tags": [
|
||||||
|
{"kind":"technique","external_id":"T1003"},
|
||||||
|
{"kind":"subtechnique","external_id":"T1003.001"}
|
||||||
|
]
|
||||||
|
}' | jq
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.3 Créer un scénario ordonné
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Suppose 3 ids: $A $B $C
|
||||||
|
curl -sX POST http://localhost:8080/api/v1/scenario-templates \
|
||||||
|
-H "Authorization: Bearer $ACCESS" \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d "{\"name\":\"chained\",\"test_template_ids\":[\"$A\",\"$B\",\"$C\"]}" | jq
|
||||||
|
|
||||||
|
# Reorder (full replace)
|
||||||
|
curl -sX PUT http://localhost:8080/api/v1/scenario-templates/<scn_id>/tests \
|
||||||
|
-H "Authorization: Bearer $ACCESS" \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d "{\"test_template_ids\":[\"$C\",\"$A\",\"$B\"]}" | jq
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.4 Filtre par tactic
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s "http://localhost:8080/api/v1/test-templates?tactic=TA0006" \
|
||||||
|
-H "Authorization: Bearer $ACCESS" | jq '.items[].name'
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. Points de contrôle critiques
|
||||||
|
|
||||||
|
- [x] `POST /test-templates` rejette MITRE inconnu avec `400 unknown_mitre_tag`.
|
||||||
|
- [x] `POST /test-templates` rejette opsec hors `low/medium/high`.
|
||||||
|
- [x] `PUT /test-templates/{id}` partial keeps unset fields.
|
||||||
|
- [x] `PUT /test-templates/{id}` avec `mitre_tags` **remplace** la collection (pas d'append).
|
||||||
|
- [x] `DELETE /test-templates/{id}` soft-delete (visible avec `?include_deleted=true`).
|
||||||
|
- [x] `POST /scenario-templates` rejette test_template inconnu ou soft-deleted.
|
||||||
|
- [x] `PUT /scenario-templates/{id}/tests` rewrite atomique (delete + re-insert, contrainte UNIQUE(position) honorée).
|
||||||
|
- [x] Un test soft-deleted **après** linking reste référencé : `test_template_deleted: true` sur le scénario.
|
||||||
|
- [x] Filtres list: `q`, `tactic`, `technique`, `subtechnique`, `opsec`, `tag` cumulatifs.
|
||||||
|
- [x] Perm gating : `test_template.{read,create,update,delete}` + `scenario_template.{read,create,update,delete}`.
|
||||||
|
- [x] `/diag/reset` truncate les 4 nouvelles tables (`scenario_template_tests`, `scenario_templates`, `test_template_mitre_tags`, `test_templates`) avant les tables MITRE.
|
||||||
|
- [x] UI : drag-and-drop @dnd-kit/sortable réordonne la liste, save persistant.
|
||||||
@@ -117,7 +117,7 @@ spec: tasks/spec.md
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## M5 — Templates : tests unitaires & scénarios ☐
|
## M5 — Templates : tests unitaires & scénarios ☑
|
||||||
|
|
||||||
**But** : admin peut bâtir le catalogue réutilisable.
|
**But** : admin peut bâtir le catalogue réutilisable.
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user