Web Recon Pipeline — Banks Field Notes

This pipeline keeps each stage’s input separate from its richer or noisier output:

approved hosts
    │
    ▼
  httpx ───────────────> live_urls.txt ──────────┬─> fingerprint and screenshots
                                                 ├─> origins.txt ─> broad HTTP templates
                                                 └─> Katana ──────> endpoints.txt
                                                                   reviewed endpoint-specific tests

The important split is between endpoints.txt, which retains complete crawled URLs, and origins.txt, which contains one scheme://host:port value per web service.

File	Contents	Next consumer
`targets.txt`	Approved hostnames, addresses, or CIDRs	httpx port probing
`live_urls.txt`	Responsive HTTP or HTTPS origins	Fingerprinting, screenshots, and Katana
`katana_urls.txt`	Complete URLs from the standard crawl	Endpoint merge
`katana_headless_urls.txt`	Navigated and XHR URLs from selected browser-rendered targets	Endpoint merge
`endpoints.txt`	Deduplicated complete URLs	Manual review and endpoint-specific templates
`origins.txt`	Deduplicated `scheme://host:port` values	Broad Nuclei HTTP template directories

Run the workflow in a dedicated assessment directory. These files can contain internal names, query values, tokens, and scanner findings.

Start with the installed versions

Check the binaries and template set before relying on copied flags:

HTTPX_BIN="$(command -v httpx-toolkit || command -v httpx)"
printf 'httpx: %s\n' "${HTTPX_BIN:-not found}"
command -v katana
command -v nuclei
command -v unfurl
command -v jq

Do not continue until each command resolves the intended binary. Then record the versions used for the assessment:

"$HTTPX_BIN" -version
katana -version
nuclei -version
nuclei -tv

Kali often names the ProjectDiscovery binary httpx-toolkit; other installations commonly use httpx. Resolving it first avoids accidentally calling the unrelated Python HTTP client.

Update Nuclei templates before an assessment begins, then keep that template version fixed while testing so the results remain reproducible:

nuclei -ut
nuclei -tv

Prepare the approved input

Use one approved hostname, address, or CIDR per line. Remove blank lines and comments, deduplicate the list, and create owner-only working files on POSIX systems:

umask 077

awk 'NF && $1 !~ /^#/ { print $1 }' '<TARGETS_FILE>' \
  | sort -u \
  > targets.txt

wc -l targets.txt

Review targets.txt before probing. A clean input list is the scope boundary for every later stage; a URL discovered by crawling is not permission to leave the approved hosts.

Probe common web ports

This is the wider port list I use when the normal 80,443,8080,8443 pass is not enough. It is useful on both external and internal target lists, but the input still needs to contain only approved hosts.

"$HTTPX_BIN" -l targets.txt \
  -p 80,81,300,443,591,593,832,981,1010,1311,2082,2087,2095,2096,2480,3000,3128,3333,4243,4567,4711,4712,4993,5000,5104,5108,5800,6543,7000,7396,7474,8000,8001,8008,8014,8042,8069,8080,8081,8088,8090,8091,8118,8123,8172,8222,8243,8280,8281,8333,8443,8500,8834,8880,8888,8983,9000,9043,9060,9080,9090,9091,9200,9443,9800,9981,12443,16080,18091,18092,20720,28017 \
  -silent \
  -rl 50 \
  -t 25 \
  -o live_urls.txt

Keep the first output clean because Katana expects URLs, not the decorated status and title output. Run the richer fingerprint view separately:

"$HTTPX_BIN" -l live_urls.txt \
  -sc -title -td -server -ip \
  -rl 50 -t 25 \
  -o httpx_fingerprint.txt

Screenshots are also a separate pass. That keeps Chromium away from ports that did not return a web response:

"$HTTPX_BIN" -l live_urls.txt -ss -rl 20 -t 10 -o screenshot_results.txt

Crawl the live URLs

Start without a browser. Katana’s default scope follows the root domain; -fs fqdn keeps each crawl on the exact host supplied in the input.

katana -list live_urls.txt \
  -d 5 \
  -fs fqdn \
  -jc \
  -kf robotstxt,sitemapxml \
  -ef css,png,jpg,jpeg,svg,gif,ico,woff,woff2,eot,ttf \
  -c 10 -p 5 -rl 40 \
  -silent \
  -o katana_urls.txt

For applications that depend heavily on browser-rendered JavaScript, create headless_targets.txt as a reviewed subset of live_urls.txt, with one complete URL per line. Repeat the crawl only against that smaller set:

katana -list headless_targets.txt \
  -d 5 \
  -fs fqdn \
  -headless \
  -xhr \
  -ef css,png,jpg,jpeg,svg,gif,ico,woff,woff2,eot,ttf \
  -c 5 -p 2 -rl 20 \
  -silent \
  -jsonl \
  -eof raw,body \
  -o katana_headless.jsonl

-xhr records browser requests inside each JSONL result’s response.xhr_requests array; it does not add those URLs to normal text output. Extract both navigated pages and XHR endpoints before merging the crawl results:

jq -r '
  (.request.endpoint? // empty),
  (.response.xhr_requests[]?.endpoint? // empty)
' katana_headless.jsonl \
  | sort -u \
  > katana_headless_urls.txt

Do not add -no-sandbox by habit. It is intended for environments such as a container running as root where Chromium cannot otherwise start.

Preserve complete URLs

Nuclei needs complete URLs. Extracting only the path with unfurl paths loses the scheme and host, so a later nuclei -l run would no longer know where to send the request.

sort -u katana_urls.txt katana_headless_urls.txt > endpoints.txt

unfurl -u format '%s://%d%:%P' < live_urls.txt > origins.txt

If only the standard crawl was used:

sort -u katana_urls.txt > endpoints.txt
unfurl -u format '%s://%d%:%P' < live_urls.txt > origins.txt

Check the reduction before scanning:

wc -l targets.txt live_urls.txt endpoints.txt origins.txt
head -n 3 endpoints.txt
head -n 3 origins.txt

Every line in endpoints.txt should still have a scheme and host. Every line in origins.txt should end at the authority, without an application path.

Run a reviewed HTTP template set

The current Nuclei flag is -tags, not -as-tags. Broad template directories should run once per origin, not once for every path Katana found. Many templates append their own fixed path to {{BaseURL}}; using a crawled path as the base can create requests such as /api/users/login and repeat that mistake across the entire crawl.

Use origins.txt for the broad pass and keep endpoints.txt for templates intentionally designed to test a supplied endpoint. First list the templates selected by the directories and filters:

nuclei -tl \
  -t http/cves/ \
  -t http/exposed-panels/ \
  -t http/exposures/ \
  -t http/misconfiguration/ \
  -etags intrusive,dos,bruteforce,fuzz \
  -s low,medium,high,critical \
  > nuclei-http-templates.txt

Review that list and inspect unfamiliar or high-impact templates before execution. Tag exclusions reduce the set; they do not prove that every remaining request is safe for the target.

Display one selected template when its behavior needs closer review:

nuclei -td -t '<TEMPLATE_PATH>'

Run the reviewed set:

nuclei -l origins.txt \
  -t http/cves/ \
  -t http/exposed-panels/ \
  -t http/exposures/ \
  -t http/misconfiguration/ \
  -etags intrusive,dos,bruteforce,fuzz \
  -ni \
  -s low,medium,high,critical \
  -c 10 -rl 40 \
  -stats -jsonl -or \
  -o nuclei-http.jsonl

-ni disables Interactsh and excludes OAST-based templates. -or avoids placing full request and response bodies in JSONL. Findings can still contain matched or extracted values, so protect the output and treat each result as a lead to validate manually rather than a confirmed finding.

Do not substitute endpoints.txt into this command merely because it contains more lines. More inputs here usually mean duplicated or malformed template paths, not better coverage.

DNS, TLS, and network templates take hostnames or IPs rather than crawled paths, so keep that pass separate:

nuclei -l '<HOSTS_FILE>' \
  -t dns/ -t ssl/ -t network/ \
  -etags intrusive,dos,bruteforce,fuzz \
  -ni \
  -s low,medium,high,critical \
  -c 10 -rl 40 \
  -stats -jsonl -or \
  -o nuclei-hosts.jsonl

Check downloaded JavaScript separately

Most HTTP templates are designed for pages and API routes. A smaller token-exposure pass is more useful for JavaScript responses:

grep -Ei '\.(m?js)([?#].*)?$' endpoints.txt | sort -u > javascript_urls.txt

nuclei -l javascript_urls.txt \
  -t http/exposures/tokens/ \
  -ni \
  -c 5 -rl 20 \
  -jsonl -or \
  -o javascript_exposures.jsonl

Treat this output as sensitive because a successful extractor may print a live credential. Store it with the same care as other assessment secrets.

Keep heavier testing separate

Headless templates and DAST/fuzzing templates are not part of the broad pass. They create more traffic and can submit payloads to application inputs. Run them only against an explicitly approved endpoint list, with a target-specific scope expression and a reviewed template set.

Do not fold token-spray into this workflow. It performs authentication attempts and can create lockouts or noisy account activity; it needs separate authorization and a deliberately selected identity set, not a generic target list.