scanners — Implementation Reference
The deep companion to the Engine & the Scanner Contract page. Every feature, with how, where (file), and why, down to the small details (user-agent, ports, thresholds, regexes). Values are quoted from source.
1. Identity on the wire: the S.E.T signature & user-agent
Where: app/scanners/fingerprint.py (logic) + app/config.py (the strings).
Why: scans must be transparently attributable to S.E.T by default
("authorized-scan"), but able to blend in as a browser where a WAF would block a
branded scanner. Our own outbound API calls always identify S.E.T to the provider.
Two surfaces:
| Surface | Default UA | Where set |
|---|---|---|
| Target-facing tools (httpx, nuclei, katana) | S.E.T-Scanner/1.0 (+https://set.security; authorized-scan) | settings.scan_user_agent_branded |
| Browser blend-in (opt-in) | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 | settings.scan_user_agent_browser |
| Our outbound API/enrichment calls | S.E.T-Scanner/1.0 | settings.api_user_agent |
How it is applied:
target_user_agent(settings, params)resolves the UA: an explicitparams["user_agent"]wins; elseparams["ua_mode"](branded|browser); elsesettings.scan_ua_mode(defaultbranded).target_header_args(settings, params)returns["-H", "User-Agent: <ua>"], which every ProjectDiscovery target-facing scanner appends to its command.api_headers(settings)returns{"User-Agent": settings.api_user_agent}for the enrichers' HTTP clients.
Per-scan override (how a caller changes it): pass it in the ScanJob.params
at POST /scans, e.g. {"ua_mode": "browser"} or
{"user_agent": "CustomAgent/1.0"}. The engine threads params into every
ScannerContext, so the override reaches the tools for that one job only.
2. Configuration reference (app/config.py)
Pydantic settings, loaded from environment / .env (case-insensitive). Notable:
| Setting | Default | Purpose |
|---|---|---|
database_url | postgresql+asyncpg://scanners:scanners@localhost:5432/scanners | async DB DSN |
db_shared_schema | shared | shared registry schema |
db_tenant_schema_prefix | tenant_ | per-tenant schema prefix |
celery_broker_url / celery_result_backend | redis://localhost:6379/0 · /1 | Celery |
shodan_api_key, virustotal_api_key, abuseipdb_api_key, otx_api_key, greynoise_api_key | None | enrichment keys |
censys_pat / censys_org_id | None | Censys Platform API |
nvd_api_key | None | raises NVD rate limit for version→CVE |
owned_ip_ranges / owned_asns | "" | ownership allow-list (CSV); see owned_ranges_list, owned_asns_list |
subfinder_bin … nuclei_bin | tool name | OSS binary paths |
scan_max_concurrency | 5 | scan concurrency rail |
scan_default_timeout_seconds | 600 | per-scanner default timeout |
scan_ua_mode | branded | branded | browser |
scan_user_agent_branded / _browser / api_user_agent | see §1 | the UA strings |
kev_file | data/feeds/cisa-kev.json | local CISA KEV feed |
Why env-driven + gitignored .env: keys never enter source; .env.example
documents the shape, secrets/ holds provider configs (both gitignored).
3. Scanner-by-scanner reference (app/scanners/easm/)
All share the contract in app/scanners/base.py. is_available() lets the
engine skip a scanner gracefully (missing binary or key) instead of failing the
job. Target-facing tools attach the UA via target_header_args.
| Scanner | key | type | consumes | runs | notable flags |
|---|---|---|---|---|---|
| Subfinder | easm.subfinder | oss_wrapper | SEEDS | subfinder | -silent -all -d <domains> |
| dnsx | easm.dnsx | oss_wrapper | SUBDOMAINS | dnsx | -silent -json -a -aaaa -cname -resp |
| Native DNS | easm.native_dns | native | SEEDS | dnspython | brute wordlist + A/AAAA/CNAME; dangling CNAME → takeover finding |
| crt.sh | easm.api_crtsh | api_provider | SEEDS | crt.sh CT logs | ?q=%.<domain>&output=json; expired-cert finding |
| naabu | easm.naabu | oss_wrapper | OWNED_HOSTS | naabu | -silent -p <sensitive ports> |
| httpx | easm.httpx | oss_wrapper | HOSTS | httpx | -silent -json -title -tech-detect -status-code + UA |
| nuclei (detect) | easm.nuclei_detect | oss_wrapper | URLS | nuclei | profile EASM_NUCLEI_DETECT, -irr -rate-limit 150 -c 25, 900s |
| nuclei (CVE) | easm.nuclei_cve | oss_wrapper | URLS | nuclei | profile EASM_NUCLEI_CVE, 2400s |
| Shodan | easm.enrich_shodan | api_provider | IP_ADDRESSES | InternetDB (free) | ports/CPEs/tags/CVEs → asset attrs + per-CVE findings |
| GreyNoise | easm.enrich_greynoise | api_provider | IP_ADDRESSES | community API | malicious classification → HIGH |
| AbuseIPDB | easm.enrich_abuseipdb | api_provider | IP_ADDRESSES | /v2/check | score≥75 crit / ≥50 high / else med |
| OTX | easm.enrich_otx | api_provider | IP_ADDRESSES | pulses | raises only if ≥2 pulses (noise guard) |
| VirusTotal | easm.enrich_virustotal | api_provider | IP_ADDRESSES | /v3/ip | malicious≥4 crit / ≥2 high; concurrency 4 (free tier) |
| Censys | easm.enrich_censys | api_provider | IP_ADDRESSES | Platform API | needs PAT + Org ID; ports/ASN inventory |
Tool execution harness (app/scanners/_proc.py — run_tool): streams
stdout line by line, 16 MB per-line buffer (asyncio default 64 KB is too
small for nuclei JSONL with captured responses), deadline-based timeout that
returns partial output rather than losing a long run, and oversized-line skip on
LimitOverrunError. have(binary) backs is_available().
Enricher base (_enrich.py): filters non-global IPs (ipaddress.is_global),
default concurrency 10, 15s timeout, branded UA via api_headers, per-IP errors
are swallowed (one bad IP never fails the run). Subclasses implement
endpoint/query/request_headers/auth/map.
4. Sensitive ports (app/scanners/easm/_ports.py)
SENSITIVE_PORTS: {port: (label, severity)} — naabu emits one Check per
(host, sensitive port); open ports above INFO become Findings. Open-port severity
is re-derived at finalize from this map, so tuning applies without re-scanning.
21 FTP=MEDIUM · 22 SSH=MEDIUM · 23 Telnet=CRITICAL · 25 SMTP=LOW · 53 DNS=LOW · 80 HTTP=INFO · 110 POP3=MEDIUM · 137/139 NetBIOS=HIGH · 443 HTTPS=INFO · 445 SMB=CRITICAL · 465 SMTPS=LOW · 502 Modbus=HIGH · 587 Submission=LOW · 2375 Docker API=CRITICAL · 3306 MySQL=HIGH · 3389 RDP=HIGH · 5000 Docker Registry=HIGH · 5432 PostgreSQL=HIGH · 5601 Kibana=HIGH · 5900 VNC=CRITICAL · 5985/5986 WinRM=HIGH · 6379 Redis=CRITICAL · 8080/8443 alt-HTTP(S)=LOW · 10250 Kubelet=HIGH · 27017 MongoDB=CRITICAL · 44818 EtherNet/IP=HIGH · 1962 PCWorx=HIGH
5. Nuclei template profiles (app/scanners/manifests.py)
NucleiProfile fields: module, templates, exclude_ids (-eid),
exclude_tags (-etags), tags (include-only -tags), dast (-dast),
workflows. nuclei_profile(module) selects the profile (defaults to EASM
detect).
INTRUSIVE_TAGS (excluded from EASM, reserved for DAST): sqli, xss, rce, lfi, rfi, ssrf, xxe, ssti, crlf, injection, oast, deserialization, traversal, prototype-pollution, csti, fileupload, command-injection, code-injection, auth-bypass, open-redirect, fuzz, dast, intrusive, bruteforce (and variants).
| Profile | templates | tags |
|---|---|---|
EASM_NUCLEI_DETECT | technologies, misconfiguration, exposed-panels, exposures, takeovers, default-logins, miscellaneous, ssl, dns, network | exclude_tags=INTRUSIVE_TAGS, -eid http-missing-security-headers |
EASM_NUCLEI_CVE | cves, vulnerabilities, iot, cnvd | exclude_tags=INTRUSIVE_TAGS (version/banner CVEs, no payloads) |
DAST_NUCLEI | dast/, http/fuzzing/ | dast=True |
DAST_NUCLEI_CVE | cves, vulnerabilities, cnvd, iot | tags=INTRUSIVE_TAGS (only the payload CVEs) |
Why: EASM observes (no payloads); DAST attacks. Same NucleiScanner code,
profile selects templates per module.
6. Normalizer (app/scanners/normalizer.py)
Turns candidates into rows. Asset upsert dedups by (type, value) and only
upgrades INACTIVE→ACTIVE (never downgrades a verified asset); attributes merge
by dict union. Check dedup_key = dedup_key or "<scanner>:<label>:<issue>".
Finding dedup_key = explicit, else sha256("<scanner>|<assetref>|<cat>|<title>")[:32].
Check→Finding auto-derivation fires when raises_finding_on == status.
Task auto-created when create_task=True, priority mapped from severity
(critical→URGENT, high→HIGH, medium→MEDIUM, low/info→LOW). Returns
NormalizeStats.
7. The Finalization Pass internals (app/finalize/)
Order in pass_.py: classify → KEV/EPSS + ownership → computed severity →
version intel → KB + floors → secret validation → open-port severity →
dedup (source precedence) → third-party guard → suppression → reconciliation
(full scans only) → task escalation. Returns a counts dict (findings, kev,
inventory, third_party, secret_fp, duplicates, suppressed, delta, …).
7.1 Severity (severity.py)
compute_severity() picks a base — CVSS band (`≥9 CRIT · ≥7 HIGH · ≥4 MED ·
0 LOW · else INFO
), else source label, else current — then applies modifiers: **KEV → force CRITICAL**, **EPSS ≥ 0.5 → raise to HIGH**, **shared/CDN + reputation → drop to INFO**.ORDER = {INFO:0…CRITICAL:4}. Every step is appended toseverity_reason(the audit chain). Type **floors** inknowledge.py:exposed_secret/admin_exposed/default_login/takeover ≥ HIGH,exposed_file/eol_software/weak_tls ≥ MEDIUM`.
7.2 Classification (classify.py + overlay.py + authstate.py)
Hard rules first: cve_id / category=exposure / category=reputation → ISSUE.
Otherwise the curated overlay decides, in priority order:
SHOULD_NOT_BE_PUBLIC (admin tooling, data stores, .git/.env, cpanel, fortigate
admin, …) → EXPECTED_EXPOSURE (sslvpn, owa/webmail, citrix, customer-portal —
public by design) → ISSUE_KEYWORDS (eol, weak, cve-, traversal, misconfig, …) →
INVENTORY (-detect, wappalyzer, favicon, options-method, missing-sri,
cookies-without, waf-detect, …) → else ISSUE (fail-safe). authstate.auth_state()
refines expected portals as gated (401/403/password form/SSO redirect) vs open.
7.3 Ownership guard (ownership.py)
classify_ownership(asset, owned_ranges, owned_asns) → owned | cloud | cdn | unknown via, in order: CIDR allow-list → ASN allow-list → CDN keywords
(cloudflare/akamai/fastly/…) → cloud keywords + PTR (.amazonaws.com,
.cloudapp.azure.com, …) → unknown. is_active_scannable = owned|unknown (naabu
OWNED_HOSTS only scans these); is_third_party = cloud|cdn (host-level findings
→ INFO + third_party_hosted).
7.4 Knowledge base (knowledge.py + kb_data.py)
kb_key() resolves in order: TEMPLATE_KB (explicit template-id/check map) →
_INJECTION tags (DAST classes) → _TEXT_KEY (directory-listing/cors/headers) →
finding_type() heuristic → PORT_TO_KEY for open ports → generic.
apply_knowledge() is KB-authoritative (overwrites template prose with the
entry's description/risk/4 steps; keeps evidence), prefixes the CVE id for
known_cve, and runs the must-map guard: a non-inventory finding that
resolves only to generic is flagged evidence.kb_unmapped + logged.
Scrubbing: strips source suffixes ((shodan)…), - Detect/Detection
template artifacts, blocked refs (projectdiscovery/nuclei-templates), and rewrites
tool/API names to neutral phrasing ("internet-exposure intelligence", "the
scanner"). KB = ~40 entries; TEMPLATE_KB ~16 maps; PORT_TO_KEY ~17 ports.
7.5 Secret validator (secrets.py)
validate_secret() → confirmed | access_key_id | unconfirmed | false_positive.
Patterns: access key id (AKIA|ASIA|AROA|AIDA|…)[0-9A-Z]{16}; secret candidate
(?<![A-Za-z0-9/+=])[A-Za-z0-9/+]{40}(?![A-Za-z0-9/+=]). Deny shapes: reCAPTCHA
6L[0-9A-Za-z_-]{38}, Google AIza[0-9A-Za-z_-]{35}, hex [0-9a-f]{32,64}, JWT
eyJ…. Public contexts (drop): x-amz-credential/signature (presigned),
__viewstate/scriptresource.axd (ASP.NET), grecaptcha/data-sitekey.
Confirm a secret only if a secret keyword (aws_secret_access_key, …) is within
~48 chars AND entropy ≥ 4.0 AND mixed character classes. Verdicts map to
false_positive→hidden, unconfirmed→INFO, access_key_id→HIGH,
confirmed→CRITICAL.
7.6 Dedup & source precedence (pass_.py)
Signature: CVE → (cve, id, asset); versioned eol/tls/cve → (type, asset, version); bare port-open → (port, asset, port#); else (class, asset, title).
Keeper chosen by source precedence nuclei(3) > httpx(2) > naabu(1), then more
references, then longer description. Losers flagged evidence.duplicate.
7.7 Lifecycle (reconcile.py) & suppression (suppression.py)
Reconcile runs on full scans only. "Seen this scan" = last_seen ≥ scan_started_at. Transitions: resolved+gone → confirmed_resolved;
resolved+still-seen → reopened; new/in-progress gone ≥ 2 scans
(CLEAN_THRESHOLD) → auto confirmed_resolved. Every change writes a
FindingEvent; per-scan delta {new, reopened, confirmed_resolved, still_open, auto_resolved} lands in job.stats.finalize.delta. Suppression matches active
rules by template_id | cve | category | title | asset_pattern →
false_positive | accepted_risk.
8. Enrichment & vuln data (app/enrich/)
cve.py:load_kev()caches the local CISA KEV set;_epss_scores()batches 100 CVEs/request toapi.first.org/data/v1/epss. KEV → CRITICAL + URGENT task; EPSS ≥ 0.5 → HIGH.version_intel.py: a version detection becomes a finding only if EOL or vulnerable. EOL viaendoflife.date/api/<slug>.json(longest-cycle-prefix match; IIS has a static map; Apache slug isapache-http-server). CVE-by-version = OSV.dev first (OSS libs by ecosystem/name) then NVD (server products by CPEvirtualMatchString, needsnvd_api_key).PRODUCT_MAPresolves product → eol slug / CPE / OSV coordinates.- Shared/third-party-IP down-rank (
finalize/ownership.py): tags CDN/cloud IPs (Shodan tags + ASN/owner keywords + AbuseIPDB usage type) and down-ranks their reputation findings to INFO (neighbour reputation, not the asset). Now applied inside the finalization pass (formerly a standaloneenrich/shared_ip.py).
9. Data model field reference (app/models/)
Shared: Tenant (tenants, schema shared): slug (unique), name,
schema_name, is_active. Per-tenant (tenant_<slug>):
- Scope / Seed — scope
name/description/is_active; seed(scope_id, kind, value)unique. - Asset —
type,value,parent_id(self-FK hierarchy),scope_id,state,discovered_by,attributes(JSONB),first_seen/last_seen; unique(type, value). - Check —
dedup_key(unique),label,issue,status(bool),category,source,evidence,asset_id/scope_id/run_id/finding_id. - ScanJob —
status,requested_scanners(JSONB),params,stats,error,started_at/finished_at. ScannerRun —scanner_key/type,status,raw_count/asset_count/finding_count. RawResult —payload. - Finding —
dedup_key(unique),module(easm/dast/cloud — scopes reconcile + dedup per module),title,description,severity,status(varchar 20),scans_seen,consecutive_clean,category,finding_class,risk,remediation,remediation_steps(JSONB),evidence,source_severity,severity_reason(JSONB),cve_id,cwe_id,cvss_score,references, FKs,first_seen/last_seen. - Task —
title,status,priority,assignee,due_date,finding_id. - FindingEvent —
event,old_status/new_status,actor,note,scan_job_id. SuppressionRule —match_type,match_value,action,scope_id(nullable = all scopes),enabled,expires_at.
Bases/mixins (db/base.py): UUIDPrimaryKey (uuid4 PK), Timestamped
(created_at/updated_at server-defaulted), SharedBase (schema=shared),
TenantBase (schema=None → translated). Engine (db/session.py): async engine,
NullPool, expire_on_commit=False.
Enums (models/enums.py): see ARCHITECTURE.md §13 and FindingStatus,
TaskStatus/Priority, CheckCategory, InputKind, ScannerType, AssetType.
10. REST API reference (app/api/v1/)
The public API is org-key authenticated (X-Master-Key + X-Admin-Key +
X-Org-Key) under /orgs and /org/... (domains, scans, findings, tasks,
feedback, DAST surface) — see the API Reference page. The table below is the
operator layer (legacy X-Tenant: <slug> header, admin-key gated; api/deps.py
get_tenant → 404 if unknown). The X-Tenant /scans and /findings routers
were removed — those are served only by the org-key API now.
| Method · Path | Body / filters |
|---|---|
POST /tenants · GET /tenants | slug (pattern), name → provisions schema |
GET /scanners | catalog (key/name/type/module/available) |
POST /scopes · GET /scopes · GET /scopes/{id} | name, seeds[] |
GET /assets | type, scope_id, search, limit/offset |
GET /checks · /checks/coverage | category, status, scope_id; coverage rollup |
GET/PATCH /tasks · /tasks/{id} | status, priority, assignee, due_date |
POST/GET/DELETE /suppressions | rule CRUD |
POST /imports/triples (201) | scope_id, source, ndjson | triples[] |
GET /health | liveness |
Default findings view hides inventory class and false_positive status —
the same discipline the report enforces.
11. Orchestration details
- Celery (
tasks/celery_app.py): broker = SQS in production (Redis for local/CI); no result backend (scan state lives in Postgres); acks-early (task_acks_late=False, doorbell pattern — a multi-day task never hits SQS's 12h visibility cap);worker_prefetch_multiplier=1(fair dispatch for long scans); no task time limit — a scan runs to completion, and a hung tool is caught by the per-tool inactivity watchdog (scanners/_proc.py), not a clock. Per-module queues (easm/dast/aws); start each worker with-Q <module>. - Task (
tasks/scan_tasks.py):scan.run_job,max_retries=0— a whole-job retry would duplicateScannerRunrows; per-scanner isolation + the PENDING idempotency guard make redelivery a safe no-op; unexpected errors callmark_job_failed. - Ingest (
ingest/triples.py): imports recon.sh NDJSON triples ({label, issue, status}) by routing issue text to asset/check shapes, thennormalize()— the same path scanners use.
12. Why the small things are the way they are (quick rationale index)
- Branded UA by default → authorized, attributable scanning; browser mode is opt-in for WAF-heavy targets.
- 16 MB tool buffer → nuclei JSONL lines with captured responses exceed asyncio's 64 KB default and would otherwise crash the reader.
- naabu consumes OWNED_HOSTS → never actively scan third-party/cloud IPs.
max_retries=0→ avoid duplicate scanner runs; isolate failures instead.- Open-port severity re-derived at finalize → tune the ports map without re-scanning.
- KB-authoritative + must-map → no template self-descriptions, no silent boilerplate; missing coverage is surfaced, not hidden.
- Secret validator → broad regexes are noisy; deny-list + entropy + proximity kill public-token false positives deterministically.
- Source precedence nuclei>httpx>naabu → the tool that actually probed the service wins over the one that only saw the port open.
See ARCHITECTURE.md for diagrams and the high-level flow.