GPT-Tokenizer icon

GPT-Tokenizer

Encode / decodes BPE Tokens or check Token Limits before working with the OpenAI GPT models.
Join our community

Actions5

Overview

The GPT-Tokenizer node's "Check Token Limit" operation checks whether a given input string is within a specified maximum number of tokens, as defined by the OpenAI GPT tokenization rules. This is particularly useful when preparing prompts or messages for OpenAI models, which have strict token limits per request. For example, before sending user-generated content to an OpenAI API, you can use this node to ensure the content does not exceed the model's token limit, preventing errors and optimizing prompt management.

Practical scenarios:

  • Validating user input length before sending it to GPT-3/4.
  • Pre-processing text data in workflows that interact with OpenAI APIs.
  • Automating content moderation based on token count constraints.

Properties

Name Meaning
Input String String to process (the text whose token count will be checked).
Max Tokens The max number of tokens to allow (e.g., 2048).
Error When Exceeding Token Limit Whether to throw an error when the string exceeds the token limit (true or false).
Destination Key The key to write the results to. Leave empty to use the default destination key (isWithinTokenLimit).

Output

  • The output is a JSON object containing a single field (by default named isWithinTokenLimit, or as specified in "Destination Key"):
    • isWithinTokenLimit: Boolean value (true or false) indicating whether the input string is within the specified token limit.

Example output:

{
  "isWithinTokenLimit": true
}

or, if a custom destination key is set:

{
  "customKeyName": false
}

If "Error When Exceeding Token Limit" is enabled and the string exceeds the limit, the node will throw an error instead of returning a result.

Dependencies

  • External library: Uses the gpt-tokenizer package for token counting and validation.
  • No API keys or external service configuration required.

Troubleshooting

Common issues:

  • Input String is not a string: Occurs if the provided input is not a string. Ensure the "Input String" property receives valid text.
  • Input String field is empty: Happens if no text is provided. Always supply a non-empty string.
  • Provide Max Tokens. (bigger than 0): Triggered if "Max Tokens" is zero or negative. Set a positive integer for "Max Tokens".
  • String exceeds token limit: If "Error When Exceeding Token Limit" is enabled and the input exceeds the limit, the node throws this error. Either increase the limit or reduce the input size.

How to resolve:

  • Double-check all required fields are filled correctly.
  • Use appropriate values for "Max Tokens".
  • Adjust workflow logic if you want to handle over-limit cases gracefully (e.g., disable "Error When Exceeding Token Limit" to get a boolean result instead of an error).

Links and References

Discussion