Tokomono
Ai Suite
2. AI Testing

Introduction to AI Testing

Introduction to AI Testing

AI Testing in AI Suite allows you to evaluate your Agents against test questions and golden-truth answers using an LLM-as-a-Judge (JudgeAgent), and automatically generate prompt improvements when an agent fails a test case.

Try it live:

Configuration

You can configure your judge model, improver model, concurrency limits, and test queues in config/ai-suite.php:

'judge_model' => env('AI_JUDGE_MODEL', 'openai:gpt-4o'),
'improver_model' => env('AI_IMPROVER_MODEL', 'openai:gpt-4o'),
'max_iterations' => 100,
'max_generated_questions' => 50,

'queues' => [
    'agent_testing' => 'agent-testing',
    'question_generator' => 'ai-suite',
    'prompt_suggestion' => 'ai-suite',
],

Step-by-Step: Managing Test Questions & Golden Truths in the UI

Every agent has its own Test Suite consisting of Questions (the input prompt sent to the agent) and Golden Truths (the expected facts, criteria, or essence that JudgeAgent checks the agent's answer against).

1. Opening an Agent's Test Suite

  1. In the top navigation bar, click Agents ▾ and select Agents.
  2. Locate the agent you want to test and click the Test → button in the right-hand Actions column.
  3. You are taken to the Agent Detail & Test Suite screen (/ai-suite#/agent/{id}).

2. Adding a Test Question Manually

  1. On the Agent Detail screen (/ai-suite#/agent/{id}), click the blue + New question button in the toolbar above the Test Suite list.
  2. In the New question modal, fill in the fields:
    • Question to the agent: Enter the user input or scenario to send to the agent (e.g., "My order #12345 arrived with a cracked screen. What should I do?").
    • Golden truth (expected essence): Describe what a correct answer must contain or achieve (e.g., "The agent should apologize, ask for photos of the damaged screen and packaging, and reference order #12345.").
  3. Click Save. The question immediately appears in the Test Suite list.

3. Automatically Generating Test Questions with AI

Instead of writing every test case by hand, AI Suite can generate test questions and golden truths automatically from the agent's system prompt:

  1. On the Agent Detail screen (/ai-suite#/agent/{id}), click the green Generate Questions button on the right side of the toolbar.
  2. In the Generate Questions modal, enter the Amount of questions to generate (up to max_generated_questions).
  3. Click Generate.
  4. Background jobs (GenerateQuestionJob) are dispatched onto the configured question_generator queue using QuestionGeneratorAgent. Refresh the page once the queue jobs complete to see the newly generated questions and golden truths.

4. Editing or Deleting a Test Question

  • Edit: Click the blue pencil icon on the right side of any question row, update the Question to the agent or Golden truth (expected essence) in the Edit question modal, and click Save.
  • Delete: Click the red trash icon on the right side of any question row and click Delete in the confirmation modal.

Step-by-Step: Running Agent Tests in the UI

On the Agent Detail screen (/ai-suite#/agent/{id}), you can run the entire test suite, a selected subset of questions, or a single quick test.

1. Configure Model & Iterations

In the top-right configuration box of the Agent Detail header:

  1. Model Dropdown: Choose the LLM provider and model to evaluate the agent with (e.g., openai:gpt-4o).
  2. Amount of runs: Enter how many iterations (1 up to max_iterations) each question should be executed. Running multiple iterations per question measures reliability and consistency across non-deterministic LLM outputs.

2. Start the Test Run

Choose one of the three execution modes:

  • Run Full Test Suite: Click the purple Start Test Run button in the top-right box. All questions in the agent's test suite are queued for the configured number of iterations.
  • Run Selected Questions Only: Check the checkboxes on the left of specific questions in the Test Suite list, then click Run selected in the toolbar.
  • Run a Single Quick Test (1 Iteration): Click the purple play icon (▶) on the right side of an individual question row to immediately run 1 iteration of only that question.

After starting any test run, you are automatically redirected to the Run Detail page (/ai-suite#/run/{id}).

Step-by-Step: Inspecting Test Runs & Applying AI Prompt Fixes

1. Browsing & Filtering Historical Test Runs (/ai-suite#/runs)

  1. Click Tests in the top navigation bar (or click the clock icon on an agent's row in /ai-suite#/agents).
  2. To filter runs for a specific agent, use the Search agent to filter... input in the top-right corner of the table and select an agent from the dropdown. Click the × on the active filter pill to clear the filter.
  3. Click View on any row to open the Run Detail screen (/ai-suite#/run/{id}).

2. Understanding the Run Detail Screen (/ai-suite#/run/{id})

  • Live Auto-Updating: While background jobs (ExecuteTestQuestion) are still running in the queue, an Auto-updating... indicator is shown and the page polls automatically every second until all iterations finish.
  • Run Summary Header:
    • Shows the Run #ID, the Model used, the Success Percentage (%), and total Costs ($).
    • Clicking the underlined Agent name opens the Full Agent modal showing the exact system prompt used for the run.
    • Clicking Run Again immediately starts a new run with the exact same agent, model, questions, and iteration count.
  • Questions & Iterations Grid:
    • Each tested question displays its Golden Truth, its pass Score (%), and a row of colored squares representing each iteration:
      • Green square: Passed (JudgeAgent evaluated is_correct = true).
      • Red square: Failed (JudgeAgent evaluated is_correct = false).
      • Orange square: Execution error (is_correct = null).
      • Gray square: Pending in queue.
    • Inspecting an Iteration Answer: Click any colored square to open the Iteration Detail modal and read the exact Answer from model.

3. Generating & Applying AI Prompt Suggestions (Suggest fix)

When a question scores below 100%, AI Suite can analyze the failure and propose an improved system prompt using the configured improver_model:

  1. Request a Suggestion:
    • Click ⚡ Suggest fix directly below the iteration squares of any question with a score below 100%, or
    • Click a red (failed) iteration square to open its modal and click Request Suggestion.
  2. Wait for Analysis: A GeneratePromptSuggestion job is queued. While running, the button displays Generating....
  3. Review the Suggested Fix: Once complete, a green Suggested Fixes (New System Prompts) card appears at the top of the Run Detail screen (and inside the failed iteration modal) showing:
    • Question: The test question that triggered the analysis.
    • Explanation (Error Analysis): Why the model's answer failed the golden truth.
    • New Prompt: The complete improved system prompt generated by the improver model.
  4. Apply or Dismiss the Fix:
    • Apply Directly: Click Apply Directly and confirm in the Confirm Apply modal. This updates the agent's prompt in the database (and writes it back to the PHP class file if the agent is an imported code agent) so all future runs use the improved prompt.
    • Dismiss: Click the × button in the top-right corner of the suggestion card and confirm to discard the suggestion.