Overview
This node is an AWS Kinesis Trigger designed to consume messages from an AWS Kinesis data stream. It listens to a specified stream and emits incoming records as workflow triggers in n8n. This is useful for real-time data processing scenarios such as monitoring event streams, processing logs, or integrating streaming data into automated workflows.
Practical examples include:
- Consuming clickstream data from a web application for analytics.
- Processing IoT sensor data streamed through Kinesis.
- Triggering workflows based on financial transaction events streamed in real time.
Properties
| Name | Meaning |
|---|---|
| Stream Name | The name of the Kinesis data stream to consume messages from. |
| Consumer Group | The name of the consumer group that manages the stream consumption state. |
| Allow Topic Creation | Whether to create the stream automatically if it does not already exist (true/false). |
| Read Messages From Beginning | Whether to start reading messages from the beginning of the stream (true) or only new ones (false). |
| Simplify Response | Controls the format of the emitted data: - Message: Emit the full message object. - Record: Emit individual records. - Data: Emit only the data payload from each record. |
Output
The node outputs JSON data representing the consumed Kinesis stream records. Depending on the "Simplify Response" property, the output structure varies:
- Message: Outputs the entire message object containing metadata and records.
- Record: Outputs an array of individual records from the stream.
- Data: Outputs an array of the
datafields extracted from each record, where each data item is parsed from base64 and JSON if possible.
If the data field is a string, the node attempts to parse it as JSON; if parsing fails, it returns the raw string wrapped inside an object with a data key.
The node does not output binary data directly but decodes base64-encoded data into strings or JSON objects.
Dependencies
- Requires valid AWS credentials with permissions to access Kinesis streams and DynamoDB (for managing consumer groups).
- The node uses an internal Kinesis client that requires AWS region, access key ID, and secret access key.
- No additional environment variables are explicitly required beyond standard AWS credential configuration.
Troubleshooting
- Stream Not Found: If the stream does not exist and "Allow Topic Creation" is false, the node will fail to connect. Enable stream creation or ensure the stream exists.
- Permission Errors: Insufficient AWS IAM permissions can cause failures. Ensure the credentials have rights for Kinesis and DynamoDB operations.
- Malformed Data: If the data cannot be parsed as JSON, the node returns the raw string. Verify the data format in your stream matches expectations.
- Consumer Group Issues: Using the same consumer group across multiple instances may cause conflicts or duplicate processing. Use unique consumer group names per workflow.
- Starting Position: Setting "Read Messages From Beginning" to false means only new messages after the trigger starts will be processed. To reprocess old data, set it to true.