GPT-Tokenizer
Encode / decodes BPE Tokens or check Token Limits before working with the OpenAI GPT models.
Overview
The GPT-Tokenizer node (Resource: Default, Operation: Decode) decodes an array of BPE (Byte Pair Encoding) tokens back into a string. This is particularly useful when working with OpenAI GPT models or other systems that use BPE tokenization, allowing you to convert tokenized data back into human-readable text.
Common scenarios:
- Decoding model outputs or intermediate results stored as token arrays.
- Integrating with APIs or workflows that require conversion from tokens to strings.
- Post-processing results from language models for display or further analysis.
Example:
If you have a list of tokens like [5661, 318, 1337], this node will output the corresponding decoded string.
Properties
| Name | Meaning |
|---|---|
| Input Tokens | Array of BPE tokens to decode. Example: [5661, 318, 1337] |
| Destination Key | The key in the output JSON where the decoded string will be stored. If left empty, defaults to data. |
Output
- The node outputs a JSON object for each input item.
- The decoded string is placed under the key specified by Destination Key (defaults to
dataif not set).
Example output:
{
"data": "Decoded string here"
}
Dependencies
- External library:
gpt-tokenizer(used for decoding tokens) - No API keys or external services are required for the decode operation.
Troubleshooting
Common issues:
- "Input Tokens is not an array":
The value provided for Input Tokens must be a valid array (e.g.,[123, 456]). Ensure you are passing a proper array and not a string or other type. - "Input Tokens field is empty":
You must provide at least one token in the array. An empty array is not allowed. - Output key missing:
If you do not specify a Destination Key, the result will be under the default keydata.
How to resolve:
- Double-check your Input Tokens property to ensure it is a non-empty array.
- Use the correct format for arrays in n8n expressions or static values.