Skip to main content

The Engine & the Scanner Contract

A multi-tenant security-scanning platform. Pluggable scanners probe an organization's external attack surface and security products, and the engine normalizes everything into Assets → Checks → Findings → Tasks, enriches and finalizes them in one deterministic pass, and serves the result over a REST API.

  • Modules (all live): EASM (non-intrusive external discovery), DAST (active web-app testing), and Cloud/CSPM (agentless AWS posture via Prowler).
  • Stack: Python 3.12 · FastAPI · PostgreSQL (schema-per-tenant, async SQLAlchemy 2.0) · Celery. Broker: SQS in production, Redis for local dev. Runtime: ECS Fargate in production, Docker Compose locally.

This document is the engine / codebase view — what each part is, where it lives in the tree, and why it is built that way. Diagrams are Mermaid (rendered inline on GitHub).

:::info Production topology For the as-built production architecture — the SQS broker, scale-to-zero ECS workers, the two-hop cloud chain, and the Jarvis investigator — see Platform Architecture. Where this page shows Redis or Docker Compose, that is the local-dev stack; production swaps the broker to SQS and runs on ECS Fargate. :::

1. Table of contents​

  1. System context
  2. Multi-tenancy (schema-per-tenant)
  3. The scanner contract (plugin model)
  4. The EASM pipeline
  5. Job orchestration & idempotency
  6. Data model
  7. Normalizer
  8. The Finalization Pass
  9. Knowledge base (no-fallback + must-map)
  10. Secret validator
  11. Severity model & provenance
  12. Classification & ownership
  13. Finding lifecycle & reconciliation
  14. Suppression
  15. Reporting
  16. Enrichment sources
  17. Template mapping (EASM vs DAST)
  18. REST API surface
  19. Repository map (where everything lives)
  20. Configuration
  21. Running it

2. System context​

Why: the API stays thin and synchronous (create resources, enqueue work, read results); all heavy scanning runs in the Celery worker so a scan that takes 40 minutes never blocks a request. The broker (SQS in production, Redis locally) decouples them; Postgres is the single source of truth and the intermediate state store for the pipeline.

3. Multi-tenancy (schema-per-tenant)​

Each tenant gets its own Postgres schema (tenant_<slug>). A small shared schema holds the tenant registry. The same ORM models map to whichever schema the session is bound to, via SQLAlchemy's schema_translate_map.

  • Where: app/db/tenancy.py (tenant_session, provision_tenant_schema), app/db/base.py (TenantBase with schema=None, SharedBase with schema="shared").
  • Why: strong data isolation per client without per-tenant databases, and one codebase. A tenant session can still join the shared registry because shared tables are never translated.
  • Provisioning a tenant = CREATE SCHEMA + metadata.create_all under the translate map. (Production evolution: Alembic per schema.)

4. The scanner contract (plugin model)​

Every scanner — native recon, an OSS-tool wrapper, or a third-party API provider — implements the same interface and returns the same normalized shapes. The engine never needs to know which kind it ran.

  • Where: app/scanners/base.py (contract), app/scanners/registry.py (@register decorator + lookup), app/scanners/easm/* (the plugins).
  • Three types (ScannerType): native, oss_wrapper, api_provider.
  • consumes (InputKind) declares what a scanner takes as input (SEEDS / SUBDOMAINS / HOSTS / OWNED_HOSTS / HOST_PORTS / URLS / IP_ADDRESSES). The engine resolves the actual targets from the DB between stages.
  • Why: new scanners are pure plugins — drop a class with @register, declare consumes, and the engine wires it in. The engine, normalizer, schema, and API never change.

The candidate shapes a scanner emits:

CandidateBecomesNotes
AssetCandidateAssetmay carry parent (type, value) for hierarchy
CheckCandidateCheck (+ Finding if raises_finding_on matches status)the (asset, issue, status) triple
FindingCandidateFinding (+ Task)direct finding with severity/evidence
raw dictsRawResultunmodified tool output for audit

5. The EASM pipeline​

A scan job runs scanners in a fixed order. Each stage reads its targets from the asset store the previous stage populated — the database replaces recon.sh's intermediate files.

  • Where: app/engine/runner.py (EASM_PIPELINE, run_scan_job, _resolve_targets).
  • Non-intrusive: EASM nuclei profiles exclude injection/payload tags (INTRUSIVE_TAGS in app/scanners/manifests.py). Active testing is DAST's job.
  • Ownership-aware: naabu consumes OWNED_HOSTS — it never actively scans IPs that resolve only to third-party/cloud infrastructure (see §13).

6. Job orchestration & idempotency​

  • Where: app/tasks/scan_tasks.py (run_job_task, max_retries=0), app/tasks/celery_app.py, app/engine/runner.py.
  • Idempotency (why it matters): run_scan_job only executes a PENDING job. A Celery redelivery or retry of an already-running/finished job is a no-op. The task uses max_retries=0 because retrying the whole job would re-run every scanner and duplicate ScannerRun rows. Per-scanner failures are isolated (one failed run → job PARTIAL), never aborting later stages.

7. Data model​

  • Shared schema: Tenant (app/models/shared.py).
  • Per-tenant: Scope, Seed, Asset, Check, ScanJob, ScannerRun, RawResult, Finding, Task, FindingEvent, SuppressionRule (app/models/*.py).
  • Check = a (asset, issue, status) coverage record — keeps both passes and fails, so coverage (not just problems) is auditable. A failing check with raises_finding_on set auto-derives a Finding.
  • Finding = a real problem. dedup_key (unique) makes re-scans idempotent: the same issue on the same asset updates last_seen instead of duplicating. Carries KB content (description/risk/remediation_steps), classification, severity + provenance, and evidence.

8. Normalizer​

  • Where: app/scanners/normalizer.py.
  • Why: scanners stay dumb — they emit (type, value) references and let the normalizer resolve foreign keys, dedup, and upsert. This is what makes the pipeline file-free: each stage writes assets the next stage reads.

9. The Finalization Pass (the heart)​

One deterministic post-scan stage turns raw detections into high-quality, de-noised findings. It runs once per scan, in this order:

  • Where: app/finalize/pass_.py (orchestrator) + the modules it calls: classify.py, overlay.py, authstate.py, severity.py, ownership.py, knowledge.py + kb_data.py, secrets.py, reconcile.py, suppression.py; plus app/enrich/{cve,version_intel}.py. (Shared/third-party-IP down-rank now lives in finalize/ownership.py, applied inside the pass.)
  • Why one pass: "one full pipeline per run" — severity, classification, enrichment, KB, validation, dedup, and lifecycle are interdependent, so they run together with a recorded provenance chain (severity_reason) instead of scattered ad-hoc steps. It is wrapped so it can never fail a scan.

10. Knowledge base​

Every finding gets a finding-specific description, risk, and exactly 4 remediation steps — never a vague fallback. Resolution is layered:

  • Where: app/finalize/kb_data.py (the KB dict + TEMPLATE_KB map + PORT_TO_KEY), app/finalize/knowledge.py (kb_key, apply_knowledge, finding_type, scrubbers, severity FLOOR).
  • KB-authoritative (why): the entry overwrites any scanner-template self-description, so output is the product's explanation ("an exposed credential lets...") not the template's ("Detect if AWS is used"). The template's evidence (matched URL, extracted values) is preserved.
  • No fallback / must-map (why): if a real finding resolves only to the generic catch-all, it is flagged (evidence.kb_unmapped) and logged so a curated entry gets authored — the system tells you where coverage is missing instead of silently shipping vague text.
  • Style rules enforced: say "threat actor" (never "attacker"), avoid the word "significant", no dashes in prose. Each entry carries a reference (OWASP, CIS, MDN, vendor docs).
  • Coverage layers: (1) curated class entries [live] · (2) per-template authored entries — 2,460 finding-specific entries (description/risk/4 steps) for every low+ severity EASM template, in data/kb/templates.json, resolved after curated overrides + CVE/EOL resolvers but before the heuristic [live] · (3) per-CVE description + fixed version from NVD + KEV required action (app/enrich/cve_detail.py, cached) [live] · (4) MITRE CWE backbone by cwe_id for any remaining tail [planned].

11. Secret validator​

Secret-detection templates use broad regexes (any 40-char base64 string is a "candidate AWS secret"), which fire on public tokens. This deterministic stage (no LLM) filters them — the gitleaks/trufflehog approach.

  • Where: app/finalize/secrets.py (validate_secret), wired into app/finalize/pass_.py.
  • Verdicts: confirmed → CRITICAL · access_key_id → HIGH (policy) · unconfirmed → INFO · false_positive → hidden. Every verdict records its reason on evidence.secret_validation_reason.

12. Severity model & provenance​

Severity is computed, not taken at face value, and every step is recorded.

  • Where: app/finalize/severity.py (compute), knowledge.py (FLOOR), app/enrich/cve.py (KEV/EPSS), app/finalize/ownership.py (shared/third-party down-rank).
  • Why provenance: severity_reason records the modifier chain (e.g. ["base:medium", "↑ EPSS 0.94", "↓ third-party hosted"]) so a rating is auditable, never a black box.

13. Classification & ownership​

Classification (FindingClass) decides visibility and triage:

ClassMeaningEffect
inventorydetection/fingerprint, not an issuehidden from default findings, excluded from reports
expected_exposurepublic by design (VPN, webmail, login)down-ranked, "verify hardening"
should_not_be_publicadmin tooling / data storeraised to ≥ HIGH
issuea genuine problemkept
  • Where: app/finalize/classify.py, overlay.py (curated keyword map), authstate.py (gated vs open).

Ownership guard — never attribute third-party infrastructure to the client:

  • Where: app/finalize/ownership.py + InputKind.OWNED_HOSTS resolution in app/engine/runner.py.
  • Why: a host like ec2-...amazonaws.com reached via the client's DNS belongs to the provider; its CVEs/ports are not the client's to fix. The guard keeps those out of the client's owned findings and off the active-scan list.

14. Finding lifecycle & reconciliation​

  • Where: app/finalize/reconcile.py (full scans only), FindingStatus (app/models/enums.py), FindingEvent history (app/models/event.py), GET /findings/{id}/history.
  • Why: clients confirm mitigations and track progress over recurring scans; reconciliation distinguishes "fixed" (confirmed_resolved) from "came back" (reopened) by comparing last_seen against the scan start time.
  • Per-module isolation: every Finding carries the module that produced it (easm / dast / cloud), set by the normalizer from the scanner-key prefix. Reconciliation and dedup are scoped to the scanning module — an EASM scan only reconciles/dedups EASM findings, a DAST scan only DAST findings, etc. So modules never age, auto-resolve, or dedup each other's findings, and a new module inherits the full lifecycle for free. (Cooldown is likewise per-(org, module).) Finalize's global re-rating stays scope-wide by design.

15. Suppression​

  • Where: app/finalize/suppression.py, SuppressionRule (app/models/suppression.py), /suppressions CRUD, PATCH /findings/{id}.
  • What: persistent triage rules matched each finalize by template_id, cve, category, title, or asset_pattern → false_positive / accepted_risk. false_positive is hidden from the default findings view.
  • Why: a triage decision should stick across future scans without re-judging.

16. Reporting​

  • Where: app/reporting/easm_report.py — python -m app.reporting.easm_report <tenant> <scope_id> [out.html].
  • What it does: excludes inventory / false-positives / duplicates from the findings section; renders matched URL + a response snippet as evidence; groups findings by issue with affected hosts; merges end-of-life software + its CVEs into one card per host showing the group's worst exploit-probability and a priority-vs-historical split (historical CVEs flagged as version-inferred / possibly backport-patched); lists third-party-hosted findings separately. No internal tool/API names appear in output.

17. Enrichment sources​

SourceUsed forKey?Where
Shodan InternetDBports/CPEs/tags/CVEs per IPfreeenrich/
GreyNoise / AbuseIPDB / OTX / VirusTotalIP reputationyesenrich/
Censys Platformexternal scan dataPATenrich/
NVDserver-product CVE-by-versionyesenrich/version_intel.py
OSV.devOSS-component CVE-by-versionfreeenrich/version_intel.py
endoflife.dateEOL datesfreeenrich/version_intel.py
CISA KEVexploited-in-wild flaglocal feedenrich/cve.py
EPSS (first.org)exploit-probabilityfreeenrich/cve.py
  • Why: subfinder uses some of the same APIs only for subdomain names; enrichment extracts everything else (ports, vulns, reputation, ASN) and merges it onto assets, emitting checks/findings. Target-facing tools use a branded S.E.T user-agent (app/scanners/fingerprint.py).

18. Template mapping (EASM vs DAST)​

  • Where: app/scanners/manifests.py + docs/TEMPLATE_MAPPING.md.
  • EASM runs a non-intrusive set (technologies, misconfiguration, exposures, exposed-panels, takeovers, default-logins, CVEs, SSL/DNS/network) with INTRUSIVE_TAGS excluded — no payloads.
  • DAST (planned) runs the active set (dast/ + http/fuzzing/ + the injection-class CVEs) and adds a crawl stage (katana + waybackurls).
  • Why: EASM observes; DAST attacks. Each module runs only its mapped set via one NucleiScanner that selects templates by nuclei_profile(self.module).

19. REST API surface​

The public API is org-key authenticated (X-Master-Key + X-Admin-Key + X-Org-Key) under /api/v1/orgs and /api/v1/org/... — domains, scans, findings, tasks, feedback, and the DAST surface. See the API Reference for that surface. The routes below are the internal/operator layer (legacy X-Tenant: <slug> header, gated behind the admin key). The former X-Tenant /scans and /findings routers have been removed — those are served only by the org-key API.

MethodPathPurpose
POST/GET/api/v1/tenantsregister / list tenants (shared)
POST/GET/api/v1/scopes, GET /scopes/{id}scopes (domains/ranges)
GET/api/v1/assetsdiscovered assets
GET/api/v1/checks, /checks/coveragecoverage triples + rollup
GET/PATCH/api/v1/tasks, /tasks/{id}remediation tasks
POST/GET/DELETE/api/v1/suppressionssuppression rules
POST/api/v1/imports/triplesingest recon.sh NDJSON
GET/api/v1/scannersscanner catalog
GET/healthliveness
  • Public / org-key API: app/api/v1/orgs.py, org.py, admin_keys.py.
  • Operator routes (above): the remaining app/api/v1/*.py, app/main.py.

20. Repository map​

src/app/
├── main.py                  FastAPI app + lifespan
├── config.py                settings (env-driven)
├── api/v1/                  REST routers (tenants/scopes/scans/findings/...)
├── db/
│   ├── base.py              SharedBase / TenantBase, mixins
│   ├── session.py           async engine + session factory
│   └── tenancy.py           schema-per-tenant sessions + provisioning
├── models/                  ORM: shared(Tenant) + per-tenant entities + enums
├── scanners/
│   ├── base.py              the scanner contract + candidate shapes
│   ├── registry.py          @register + lookup
│   ├── normalizer.py        candidates → DB (resolve/dedup/upsert/derive)
│   ├── manifests.py         nuclei profiles, INTRUSIVE_TAGS, module mapping
│   ├── fingerprint.py       branded user-agent config
│   └── easm/                the EASM plugins (subfinder/dnsx/naabu/httpx/nuclei/enrich/_ports)
├── engine/runner.py         pipeline orchestration + target resolution + idempotency
├── tasks/                   Celery app + scan task (max_retries=0)
├── enrich/                  cve(KEV/EPSS) · version_intel(EOL/NVD/OSV)
├── finalize/                THE FINALIZATION PASS
│   ├── pass_.py             orchestrator (order of operations)
│   ├── classify.py/overlay.py/authstate.py   classification
│   ├── severity.py          computed severity + provenance
│   ├── ownership.py         owned vs third-party guard
│   ├── kb_data.py           KB entries + TEMPLATE_KB + PORT_TO_KEY
│   ├── knowledge.py         KB resolution, scrub, floors, must-map guard
│   ├── secrets.py           deterministic secret validator
│   ├── reconcile.py         lifecycle reconciliation
│   └── suppression.py       persistent triage rules
├── reporting/easm_report.py grouped HTML report
└── ingest/triples.py        recon.sh NDJSON import bridge

21. Configuration​

  • Where: app/config.py (Pydantic settings), .env (gitignored), .env.example (template), secrets/ (gitignored provider configs).
  • Key settings: DATABASE_URL, Redis URL, enrichment API keys, NVD_API_KEY, owned_ip_ranges / owned_asns (the ownership allow-list).
  • Why gitignored: API keys never enter git. secrets/ and *provider-config* are excluded; only .env.example is tracked.

22. Running it​

docker compose up --build        # postgres + redis + api + worker
# API: http://localhost:8000   (docs at /docs)
# create a tenant, a scope with seed domains, then POST /scans
# generate a report:
docker exec <worker> python -m app.reporting.easm_report <tenant> <scope_id> out.html
  • The worker image bundles the ProjectDiscovery Go toolchain (Go 1.24, GOTOOLCHAIN=auto); Go tools are copied after pip install so the Go httpx binary wins over the Python httpx CLI stub.
  • No local Python needed on the host — everything runs in containers.
  • STATUS.md — current build state, validated results, known issues.
  • TEMPLATE_MAPPING.md — per-module nuclei template selection + counts.
  • BUILD_PLAN_finding_quality.md — the finalization-pass design.
  • BUILD_PLAN_knowledge_base.md — KB layering plan (curated + CVE-DB + CWE).
  • BUILD_PLAN_lifecycle.md — status model, reconciliation, scheduler.
  • FINDINGS_QA_PUNCHLIST.md — finding-quality issues found in review + fixes.