GPT-Tokenizer icon

GPT-Tokenizer

Encode / decodes BPE Tokens or check Token Limits before working with the OpenAI GPT models.
Join our community

Actions5

Overview

The GPT-Tokenizer node's "Count Tokens" operation determines how many tokens a given input string would produce when encoded for use with OpenAI GPT models. This is particularly useful for users who need to ensure their text fits within model token limits, such as when preparing prompts for GPT-3/4 APIs or managing chunked data for large language model processing.

Practical examples:

  • Before sending a prompt to OpenAI, check if it exceeds the model's maximum token limit.
  • Analyze and optimize user-generated content length for AI chatbots.
  • Pre-process documents to fit within API constraints.

Properties

Name Meaning
Input String String to process. The text whose token count will be determined.
Destination Key The key to write the results to. Leave empty to use the default destination key (tokenCount).

Output

  • The output is a JSON object containing a single field:
    • Destination Key (default: tokenCount): The number of tokens in the provided input string.

Example output:

{
  "tokenCount": 42
}

If you specify a custom Destination Key, the result will appear under that key instead.

Dependencies

  • External library: gpt-tokenizer (used for encoding and counting tokens)
  • No external API keys or special n8n configurations are required for this operation.

Troubleshooting

Common issues:

  • "Input String is not a string": The provided input is not a valid string. Ensure you pass a text value.
  • "Input String field is empty": The input string property is empty. Provide a non-empty string to process.

How to resolve:

  • Double-check your workflow to ensure the "Input String" property receives a valid, non-empty string.
  • If using expressions, verify they evaluate to a string.

Links and References

Discussion