Firecrawl icon

Firecrawl

Get data from Firecrawl API

Overview

This node integrates with the Firecrawl API to perform search queries across multiple sources such as web pages, images, and news. It allows users to specify a search query and customize parameters like sources, result limits, time-based filters, and request timeout. The node is useful for automating data retrieval from Firecrawl’s search engine, enabling workflows that require aggregated search results or scraping of search data.

Practical examples include:

  • Gathering recent news articles on a specific topic.
  • Collecting images related to a keyword for content creation.
  • Performing broad web searches to gather URLs and snippets for research or monitoring purposes.

Properties

Name Meaning
Query The search query string to look up in the selected sources.
Sources Specifies which sources to search from. Options: Web, Images, News. At least one must be chosen.
Timeout (Ms) Maximum time in milliseconds to wait for the API response before timing out.
Limit Maximum number of search results to return. Minimum value is 1.
Time Based Search Optional parameter to filter search results based on time (e.g., past hour, day).
Additional Fields A collection allowing addition of custom JSON properties to the request body when using a custom body.
Use Custom Body Boolean flag indicating whether to send a fully custom request body instead of standard parameters.

Output

The node outputs JSON data containing the search results returned by the Firecrawl API. This typically includes structured information about each search result such as titles, URLs, snippets, image links, or news metadata depending on the selected sources.

If binary data is present (not explicitly shown in the provided code), it would represent downloaded files or media associated with the search results, but this node primarily focuses on JSON search result data.

Dependencies

  • Requires an API key credential for authenticating with the Firecrawl API.
  • The base URL defaults to https://api.firecrawl.dev/v2 but can be overridden via credentials.
  • No other external dependencies are indicated.

Troubleshooting

  • Timeouts: If the request times out, consider increasing the "Timeout (Ms)" property or checking network connectivity.
  • Empty Results: Ensure the query string is not empty and that at least one source is selected.
  • Invalid Custom Body: When using the "Use Custom Body" option, ensure the JSON structure in "Additional Fields" is valid; otherwise, the API may reject the request.
  • Authentication Errors: Verify that the API key credential is correctly configured and has necessary permissions.
  • Limit Exceeded: Setting the limit too high might cause slow responses or API rate limiting.

Links and References

Discussion