Runner: Httpx icon

Runner: Httpx

Interact with Httpx

Overview

This node integrates with the Httpx tool, a fast and multi-purpose HTTP toolkit for probing and analyzing HTTP servers. It allows users to send HTTP requests to multiple targets concurrently, gather detailed information about the responses, and optionally upload the results to an object storage bucket.

Common scenarios where this node is beneficial include:

  • Security testing and reconnaissance by probing hosts for HTTP response details.
  • Gathering metadata such as server headers, technologies in use, response times, and content hashes.
  • Automated scanning of multiple URLs or hosts with customizable probes and filters.
  • Capturing screenshots of web pages using a headless browser.
  • Uploading scan results to cloud storage for further analysis or archival.

Practical examples:

  • Scanning a list of domains to detect CDN providers, server types, and HTTP status codes.
  • Extracting specific data from HTTP responses using regex or preset extractors (e.g., IP addresses, emails).
  • Filtering out error pages or unwanted responses based on content length or response time.
  • Taking screenshots of web pages for visual verification.
  • Uploading JSON results and screenshots to a configured S3-compatible bucket for sharing or backup.

Properties

Name Meaning
Only Success Boolean flag to process only successful responses.
Upload Result Boolean flag to enable uploading the scan results to an object storage bucket.
Bucket The name of the target bucket where results will be uploaded (required if Upload Result is true).
URL Prefix The base URL prefix used to construct accessible URLs for uploaded results (required if Upload Result is true).
Path The HTTP path appended to each target host when making requests (default "/").
Method HTTP method to use for requests. Options: GET, POST, PUT, DELETE, HEAD, OPTIONS, PATCH, TRACE.
Options A collection of advanced options grouped into categories: Probes, Headless, Matchers, Extractor, Filters, Rate Limit, Miscellaneous, Output, Configurations, Optimizations. Each category contains specific flags and values to customize the HTTP probing behavior.
Advanced Config Additional environment variables, files, and file collections to configure the execution environment or input data.
Debug Mode Boolean flag to enable verbose debug output for troubleshooting.

Selected notable options within "Options" property:

  • Probes: Select what information to retrieve from the HTTP response, e.g., ASN, CDN, Content Length, Status Code, Title, Web Server, etc.
  • Headless: Enable headless browser features like taking screenshots or excluding screenshot bytes from output.
  • Matchers: Define conditions to match responses, such as matching specific status codes, strings, regex patterns, or CDNs.
  • Extractor: Extract data from responses using regex or predefined patterns (e.g., IP addresses, emails, URLs).
  • Filters: Filter out responses based on criteria like status code, content length, error page detection, or response time.
  • Rate Limit: Control concurrency and request rate limits.
  • Miscellaneous: Various flags for probing CSP domains, HTTP2 support, custom ports, TLS grabbing, etc.
  • Output: Configure inclusion of HTTP request/response data or redirect chains in the JSON output.
  • Configurations: Set HTTP headers, proxy settings, allowed/denied IPs, redirects, user agents, resolvers, and other HTTP client behaviors.
  • Optimizations: Tune delays, retries, max errors per host, response size limits, and timeout settings.

Output

The node outputs JSON objects representing the HTTP probe results for each input target. Each JSON object may contain fields such as:

  • input: The target host and port probed.
  • Various probe results depending on selected options, e.g., status code, content length, server info, technology detected, response time, favicon hash, etc.
  • If enabled, URLs pointing to uploaded stored response files and screenshots.
  • If headless mode is used, binary data fields may be included:
    • binary.response: Base64-encoded HTTP response body.
    • binary.screenshot: PNG image bytes of the webpage screenshot.

The output structure is flexible and depends on the selected probes, extractors, filters, and output options.

Dependencies

  • Requires an external installation of the Httpx command-line tool.
  • Optionally requires credentials for an S3-compatible object storage service to upload results.
  • Uses rclone CLI tool for uploading files to the specified bucket.
  • Node configuration must include appropriate API keys or authentication tokens for the object storage service.
  • Network access to target hosts and optionally to the object storage endpoint.

Troubleshooting

  • Common issues:
    • Missing or invalid credentials for the object storage service when upload is enabled.
    • Network connectivity problems to target hosts or storage endpoints.
    • Incorrectly formatted input targets causing probe failures.
    • Misconfiguration of advanced options leading to unexpected results or errors.
  • Error messages:
    • Errors related to failed command execution of Httpx or rclone indicate environment or permission issues.
    • JSON parsing errors suggest malformed output from the Httpx tool.
    • Authentication errors during upload point to invalid or missing API keys.
  • Resolutions:
    • Verify credentials and permissions for the storage bucket.
    • Ensure Httpx and rclone are installed and accessible in the environment.
    • Validate input data format and required parameters.
    • Use Debug Mode to get detailed logs for diagnosing issues.

Links and References

Discussion