JaaS AI icon

JaaS AI

Evaluate AI Agents with JaaS

Overview

This node integrates with the JaaS AI evaluation service to assess AI-generated answers based on customizable evaluation criteria. It is designed to evaluate the quality and accuracy of AI agents' responses by comparing them against ground truth answers, optionally within a given context.

Common scenarios include:

  • Automatically scoring chatbot or virtual assistant responses.
  • Benchmarking different AI models or configurations.
  • Conducting conversational AI evaluations with multiple criteria.
  • Verifying AI outputs in controlled cohorts for quality assurance.

For example, you can input a question, an AI-generated answer, and the correct answer, then specify criteria like "Accuracy" or other custom metrics. The node sends this data to the JaaS API and returns an evaluation summary.

Properties

Name Meaning
Evaluation Criteria #1 First criterion to evaluate the AI response (default: "Accuracy").
Evaluation Criteria #2 Second evaluation criterion (optional).
Evaluation Criteria #3 Third evaluation criterion (optional).
Evaluation Criteria #4 Fourth evaluation criterion (optional).
Evaluation Criteria #5 Fifth evaluation criterion (optional).
Type of Evaluation Type of evaluation to perform:
- Simple
- Conversational (S)
- Verified
Evaluation Fields: Question The question or prompt posed to the AI agent.
Answer The AI agent's answer to be evaluated.
Ground Truth Answer The correct or expected answer used as a reference for evaluation.
Context Additional context information relevant to the question and answer (optional).
Cohort Identifier for grouping evaluations, e.g., for tracking or segmentation (default: "N8N-NC").

Output

The node outputs a JSON object containing the evaluation results from the JaaS AI API:

  • status: The status of the evaluation request (e.g., success or failure).
  • criteria: The evaluation criteria that were applied.
  • type: The type of evaluation performed (matching the selected evaluation type).

The output does not include binary data; it strictly provides structured JSON summarizing the evaluation outcome.

Example output JSON structure:

{
  "status": "success",
  "criteria": ["Accuracy", "Fluency"],
  "type": "Simple"
}

Dependencies

  • Requires an active API key credential for the JaaS AI service.
  • Makes HTTP POST requests to https://api.jaas-ai.com/v1/evaluate.
  • The node expects the API key to be configured securely in n8n credentials.
  • No additional environment variables are required beyond the API key.

Troubleshooting

  • No API key provided!
    This error occurs if the API key credential is missing or not set up correctly. Ensure you have created and assigned the necessary API key credential in n8n before running the node.

  • HTTP request failures or timeouts:
    Check network connectivity and verify the API endpoint is reachable. Also confirm the API key is valid and has sufficient permissions.

  • Invalid or empty evaluation criteria:
    Make sure at least one non-empty evaluation criterion is provided. Empty strings are filtered out before sending the request.

  • Unexpected API response:
    If the API returns errors or unexpected data, review the input parameters for correctness, especially the question, answer, and ground truth fields.

Links and References

Discussion