Agent Evaluation

​

Overview

Agent Evaluation lets you measure how well an agent performs against a repeatable set of test cases, using criteria you define yourself. Instead of judging agent quality by spot-checking a few chat replies, you build a repeatable process. You create a Dataset of representative inputs, define Dimensions that describe what “good” looks like (tone, accuracy, following instructions, structure, etc.) and combine them into a Suite. Running the suite scores the agent against the dataset and produces consistent, comparable results. This turns agent quality from a subjective impression into something you can track over time:

• Catch regressions—re-run the same suite after changing agent instructions or tools, and immediately see whether scores improved or dropped.
• Combine automated and human judgment—score responses with an Artificial Intelligence (AI) judge model, a custom validation script, a human reviewer or any mix of the three.
• Compare instruction changes objectively—run the same dataset against different agent versions to see which configuration performs best before you adopt it.
• Build a record of quality—the platform keeps every run in a searchable history, so you can cite concrete evidence of how an agent behaves, rather than relying on anecdotes.

​

Prerequisites

• You need an existing agent to evaluate. See Agents if you have not created one yet.
• Your project role determines what you can do in Agent Evaluation:

​

Access Agent Evaluation

1. Open the agent you want to evaluate.
2. Click the three-dot menu (⋮) on the agent.
3. Select Evaluate (Beta).

This opens the Evaluation workspace for that agent, where you manage Suites, Dimensions, Datasets and the Results History.

​

Suites

A Suite is what you actually run: it pairs one agent version, one Dataset and one or more Dimensions (via per-suite bindings), plus a Judge Model used for any AI-scored dimensions. Create a Suite The first time you open Evaluate (Beta) for an agent, no suites exist yet—the Suites panel shows an empty state prompting you to create one, because you need a suite before you can run an evaluation.

1. In the Evaluation workspace, click New Suite.
2. In the Create Suite dialog, enter a Suite Name (required) and an optional Description, then click Save.
3. Saving creates the suite and opens it for configuration. Under General, set the remaining fields:

   Field	Description
   Judge Model	The LLM used to score any AI-evaluated dimensions in this suite. Defaults to Auto until you choose one.
   Version	The agent version this suite evaluates. Disabled while an evaluation run is in progress—otherwise you can change it at any time.

   
4. Click Save to store these changes. Use Discard to drop unsaved edits in this section.

​

Datasets

A Dataset is a named collection of test Cases that you run the agent against during an evaluation. Datasets belong to an owning agent but can be marked Available across the project so other agents’ suites can reuse them.

​

Create a Dataset

1. In the Evaluation workspace, open Manage Datasets.
2. Click + to create a dataset.
3. Enter a Dataset Name (required) and an optional Description.
4. If you want other agents’ suites to use this dataset, check Available across the project.
5. Click Save.

​

View the Dataset Page

Selecting a dataset from Manage Datasets opens its case list.

• Dataset list (left)—every dataset created for the agent, showing its case count and description. Click a dataset to view its cases. Click + to create another.
• Case table (right)—one row per case, with sortable Input, Expected Output, Source and Variables columns. Source shows how you created the case (for example, Chats or Import), and Variables shows how many key-value pairs the case defines, or — when it has none.
• + Case—opens the add-case menu.
• Delete icon (top right, next to + Case)—deletes the dataset itself.
• Row actions—hover a row to reveal its edit (pencil) and delete (trash) icons, for editing or removing that individual case.

​

Test Cases

A Case is a single test within a dataset: an input (and, optionally, an expected output and variables) that the agent responds to during a run. Each case represents one concrete scenario for evaluating the agent. The dataset stays meaningful only when its cases reflect the situations the agent needs to handle well. Every dimension bound to the suite scores each case independently, and a run breaks down results per case. Adding, editing or excluding a case changes what the suite evaluates and directly affects its overall score the next time it runs.

​

Add a Test Case

With the dataset selected, click + Case to open the add-case menu. Then choose one of the three methods below. From Chats and Runs Turns a real conversation or a past evaluation run into a repeatable test case, so you can capture actual agent interactions—including ones that previously went wrong—instead of writing an input from scratch.

1. From the add-case menu, select From chats and runs.
2. Switch between the Chats and History Runs tabs to find the conversation or run you want to reuse.
3. Select the entry to turn it into a repeatable test case in the dataset.

Each case in the dataset shows its Input, Expected Output, Source (marked Chats when pulled from a conversation) and Variables. Import a File Adds many test cases at once from a file, useful when you already maintain test data outside of ELITEA or want to bulk-load a large set of inputs instead of entering them one by one.

1. From the add-case menu, select Import file.
2. Choose a .csv or .json file containing your cases.
3. Upload the file. A result banner shows how many cases were imported and how many were rejected.

Create a Case Manually Builds a single test case field-by-field, giving you full control over the exact input, variables and expected output—the most direct option when you already know precisely what you want to test.

1. From the add-case menu, select Create manually.
2. Enter the Input (required)—the text sent to the agent.
3. Optionally add Variables as key-value pairs.
4. Optionally check the Expected Output option and enter the expected response.
5. Click Save to add the case to the dataset.

​

Dimensions

A Dimension defines one thing you want to measure—for example, “Response is polite and professional” or “Output is valid JSON.” Dimensions turn what would otherwise be a subjective impression of quality into a named, repeatable criterion that scores the same way on every case, every time the suite runs. One of three engines scores each dimension. You choose the engine when you create the dimension, and it stays fixed afterward: A dimension lives in a shared library, and you add it to one or more suites through a binding. Editing its evaluation instructions, target or scale affects every suite that includes it the next time that suite runs, not just the suite you were viewing when you made the change. Dimensions can be scoped at three levels:

• Agent—private to the agent you are evaluating.
• Project—shared and reusable across every agent in the project.
• Platform—shared platform-wide; read-only to regular users.

​

Create a Dimension

1. In the Evaluation workspace, open Manage Dimensions.
2. Switch between the Agent, Project and Platform tabs to choose the scope for the new dimension (Platform is read-only for dimension creation).
3. Click + to create a new dimension.
4. Fill in the following fields and click Save.

   Field	Description
   Name	The display name of the dimension.
   Available across the project	Lets all agents and agent versions in the current project use this dimension.
   Evaluator	AI, Human or Code—sets how you evaluate the dimension. Locks once you save the dimension.
   Evaluation Instructions (AI only)	Describes the qualities, behaviors or examples the judge model should identify.
   Evaluation Guidance (Human only)	Optional guidance to help reviewers score consistently.
   Validation Code (Python) (Code only)	Script written in the built-in editor. Must assign the result to a plain variable named result (a boolean for pass or fail, or a number for a numeric score) instead of using return. Depending on the evaluation target, the script can read input, output, expected and structure.
   Evaluation Target	Which parts of the case you show to the judge or script: Output, Input or Agent structure (select one or more).
   Scale Type	Score (1-100), Rating (1-5), Pass or Fail or Custom (define your own Min and Max).
   Success Criteria	At least (>=), At most (<=) or Exactly (=)—plus the Target Value the score must meet to pass.
   Importance	Low, Medium, High, Critical or Custom (with a custom weight value). Controls how much this dimension contributes to the overall score of the suite.

   

​

Build a Dimension with AI

If you would rather describe what you want in plain language than complete every field yourself, you can generate a dimension draft:

1. From the dimension creation menu, choose Build with AI.
2. Describe the quality you want to measure—for example, a description of a polite and professional tone, or a requirement that output be valid JSON.
3. Click Generate Draft.
4. Review the generated candidate dimensions and click one to select it.
5. The full dimension form opens pre-filled with the generated content. Adjust any field as needed, then click Create Dimension.

You can return to the prompt step or the candidate list at any point before creating the dimension.

​

Run an Evaluation

Use a completed suite to measure agent performance and review its results.

1. Open the suite you want to run.
2. Confirm that a dataset is attached under the Dataset section of the suite.
3. Confirm that at least one dimension is added under the Dimensions section of the suite.
4. Click Evaluate.
5. A progress indicator shows how many cases have completed (for example, “3 of 10 cases”) while the run is in progress. You can cancel a run in progress.
6. When the run finishes, the Results panel updates automatically with the new scores.

​

Review Results

The Results panel shows the outcome of the most recent run for the selected suite: The summary cards show the following information:

• Dimension table—for each dimension: its Scale, Average score, Target and whether the target was Met.
• Case list—every case in the dataset, expandable to show its per-dimension results. Each dimension card shows its state (score and whether the target was met, an error, or—for Human-scored dimensions—pending) along with the rationale from the judge, your own comments or an “Awaiting human evaluation” note.

When human-scored dimensions are still awaiting review, a banner above the case list reads: “Awaiting human evaluation — N result(s) still need a score. Totals update as each one is saved.” From the Results panel toolbar you can:

• Enhance with AI—analyze this run and generate suggested fixes to the agent’s instructions, scoring dimensions and dataset cases.
• Export to Excel—download the full results as a spreadsheet.
• Clear the results—remove the results for the current run (with a confirmation prompt).
• Results History—open the full run history for this agent in a separate panel.

​

Enhance with AI

Once a run has results, use Enhance with AI to get AI-generated suggestions for closing the gap between what the agent produced and what the suite expected—without manually re-reading every missed case yourself.

1. In the Results panel toolbar, click Enhance with AI
2. The platform analyzes the run and walks you through a few steps. Analysis always appears; the rest only appear if that run’s proposal actually has something to suggest there, so a run with only instruction fixes shows just Analysis and Agent Instructions:
   • Analysis—coverage statistics, a written diagnosis of why the run missed its targets, and a note on what the following steps contain. If nothing needs changing, this step says so instead (“No changes suggested—the agent and the evaluation look consistent for this run.”) and the wizard ends here.
   • Agent Instructions—shown only when the proposal includes instruction edits. A side-by-side diff of the agent’s current instructions against the proposed rewrite, with each individual suggested edit listed below as a checkbox you can accept or reject one at a time.
   • Dimensions—shown only when the proposal includes dimension fixes. Suggested rubric (Evaluation Instructions) or target changes for dimensions this run scored.
   • Dataset Cases—shown only when the proposal includes dataset fixes. Suggested Expected Output corrections for cases in the attached dataset.
3. Uncheck any individual suggestion you do not want applied; the Instructions diff updates live as you do.
4. Click Save, or Save as new version (shown only when at least one instruction edit is accepted):
   • Save—applies the accepted instruction edits directly to the agent version the suite is bound to, and applies the accepted dimension and dataset-case fixes immediately.
   • Save as new version—forks a new agent version containing only the accepted instruction edits, leaving the suite’s current version untouched. The accepted dimension and dataset-case fixes are still applied immediately either way.

​

Score a Human Dimension

Human-scored dimensions let a run continue without waiting. The cases appear as pending until someone enters a score.

1. In the case results, find a dimension card for a Human-evaluated dimension and open it.
2. Review the response shown, using any Evaluation Guidance written on the dimension.
3. Enter a score using the control shown for the scale of that dimension (a Pass or Fail choice or a numeric score or rating).
4. Optionally add a Comment (up to 2000 characters).
5. Click Save.

​

Review Run History

1. From the Results panel, click the Results History icon (or navigate to it directly from the Evaluation workspace).
2. The run history table lists every run for the agent, showing its Date, Suite, agent Version and Score. A badge highlights the score change compared to the previous scored run for the same suite.
3. Select a run to view its full results in the panel beside the table.
4. Use the row actions menu to Share, Export to Excel, open Analytics or Delete a run (deleting requires the appropriate permission).

​

Run Analytics

Analytics opens a dedicated page for that run, covering both test-case execution and LLM-as-judge scoring for that run only—no other evaluations or ordinary chats are mixed in. The header shows the run’s suite, run date and the agent version that was evaluated. Costs opens by default; use Refresh data or Export to Excel (one worksheet per tab) from the page toolbar.

​

Troubleshoot Evaluation Issues

Resolve a Dimension Error for a Case

This means the scoring engine could not produce a valid result for that specific case—for example, the AI judge returned a response that did not match the expected format. The error remains isolated to this dimension and case. Other dimensions and cases in the same run are unaffected. Review the evaluation instructions or evidence scope for the dimension and run the suite again if needed.

Resolve a Pending Human Dimension

Human-scored dimensions require someone to manually enter a score after the run completes—only a person can score them. Open the case and save a score to clear the pending state.

Create or Edit a Code Dimension

Only Admins can create or edit Code-based dimensions. Editors have view-only access. Also check that your validation script passes the automatic safety check—the check rejects dangerous imports, unsafe built-ins and dunder attribute access.

Keep an Active Case in a Shared Dataset

A suite must keep at least one active case from its attached dataset. Include a different case before excluding the one you want removed.

Resolve a Dataset Case Limit Error

A dataset can hold at most 10 cases. Adding one more—manually, via import, or from a chat/run—is rejected with a message naming the dataset’s current count and the limit. Remove an existing case first, or (for an import or promoted conversation) split the cases across multiple datasets instead of one.

Resolve Rejected Imported Cases

The platform rejects a .json file entirely when it is not valid JSON, and imports no cases from it. Otherwise, it validates each case on its own. Check the reasons listed next to the rejected count in the import result banner, fix those rows in your source file and re-import. Cases that already passed validation stay in the dataset unchanged.

Run a Suite

Before you can evaluate a suite, it needs a Dataset attached and at least one Dimension added. Open the suite and check both the Dataset and Dimensions sections. For instructions, see the Run the Suite section.

Enhance with AI Finds Nothing to Suggest

This means the analysis found the agent’s behavior and the evaluation setup already consistent for that run—there is nothing to fix. This is expected when the run scored well, or when prior fixes already addressed the gaps.

Enhance with AI Edit Cannot Be Applied

An instruction edit that no longer matches the agent’s current instructions (because they changed after the proposal was generated) cannot be applied. Uncheck that edit, or close the dialog and reopen Enhance with AI to generate a fresh proposal against the current instructions.

---

​

Best Practices

Start Small and Expand the Dataset

Begin with a handful of cases that cover the scenarios you care about most, confirm your dimensions score them sensibly, then add more cases—including known failure cases—once the setup is stable.

Write Specific, Concrete Evaluation Instructions

Vague instructions produce inconsistent AI scores. Naming exact qualities, thresholds, or examples such as those in the evaluation instructions example gives the judge model less room for inconsistent scoring between runs.

Reuse Project and Platform Dimensions

Reuse a shared dimension instead of recreating similar criteria for every agent, so scores stay comparable across agents that share the same standards.

Turn Real Failures into Cases

Use the From Chats and Runs method in the Add a Test Case section to capture an actual bad response as a permanent case, so the platform automatically catches the same mistake if it appears again.

Match the Engine to the Criterion

Use Code dimensions for objective, deterministic checks (format, required fields), AI dimensions for judged qualities (tone, relevance) and Human dimensions only for what genuinely needs human judgment. Human scoring lets a run continue without waiting, but a person must complete it.

Use Case: Catch a Regression After an Instruction Change

1. Run a suite once to establish a baseline score.
2. Edit the instructions or tools of the agent.
3. Re-run the same suite—same dataset, same dimensions—and compare the new score against the previous one in the Run History section.

A drop in score, especially on cases that previously passed, flags a regression before it reaches users.

Use Case: Compare Two Agent Versions

1. Create two suites that share the same Dataset and Dimensions but are each locked to a different agent Version.
2. Run both suites.
3. Compare their scores in the Run History section to decide which version performs better before deploying it.

Use Case: Build a Regression Set from Real Failures

1. When a chat or a run produces a bad response, open the From Chats and Runs method in the Add a Test Case section and add it to a dataset as a case.
2. If you have one, enter the correct Expected Output so future runs have something concrete to compare against.
3. Keep the case in the dataset used by your suites, so they automatically catch the same failure if it recurs.

---