Index Xray Data

​

Overview

• Test Cases: Detailed test procedures, manual test steps with actions, data, and expected results
• Test Types: Support for Manual, Cucumber/Gherkin, and Generic test types
• Test Metadata: Test keys, summaries, descriptions, test types, and custom fields
• Attachments: Screenshots, test files, documentation, and media attached to test steps
• Preconditions: Associated preconditions for test case setup requirements

• Semantic Search: Find test cases and procedures across projects using natural language queries
• Context-Aware Chat: Get AI-generated answers from your test documentation with citations to specific test cases
• Cross-Project Discovery: Search across multiple Xray projects and test types
• Test Analysis: Analyze testing patterns, coverage, and procedures for quality improvement
• Knowledge Extraction: Transform test documentation into searchable organizational knowledge

• Finding similar test cases across projects to avoid duplication and ensure consistency
• Onboarding new QA team members by allowing them to ask questions about testing procedures and standards
• Analyzing test coverage gaps and identifying areas needing additional test cases
• Support teams searching for existing test procedures when investigating issues
• Test managers extracting insights from test documentation for reporting and process improvement

​

Prerequisites

1. Xray Credential: Xray Cloud API credentials with Client ID and Client Secret configured in ELITEA
2. Vector Storage: PgVector selected in Settings → AI Configuration
3. Embedding Model: Selected in AI Configuration (defaults available) → AI Configuration
4. Xray Toolkit: Configured with your Xray Cloud instance details and credentials

​

Required Permissions

• Read access to Xray Cloud projects and test cases
• Permission to view the specific projects you want to index

• Access to view test case attachments (if including attachments)
• Permission to view test steps and preconditions
• Access to both active and archived test cases (based on your requirements)

• Client Credentials: Client ID and Client Secret generated in Xray Cloud

​

Step-by-Step: Creating an Xray Credential

1. Generate Xray API Credentials in your Xray Cloud account (API Keys → Generate Client ID and Client Secret)
2. Create Credential in ELITEA: Navigate to Credentials → + Create → Xray → enter details and save

​

Step-by-Step: Configure Xray Toolkit

1. Create Toolkit: Navigate to Toolkits → + Create → XRAY Cloud
2. Configure Settings: Set Xray Cloud base URL and assign your Xray credential
3. Enable Tools: Select Index Data, List Collections, Search Index, Stepback Search Index, Stepback Summary Index, and Remove Index tools
4. Save Configuration

​

Tool Overview:

• Index Data: Creates searchable indexes from Xray test cases and documentation
• List Collections: Lists all available collections/indexes to verify what’s been indexed
• Search Index: Performs semantic search across indexed content using natural language queries
• Stepback Search Index: Advanced search that breaks down complex questions into simpler parts for better results
• Stepback Summary Index: Generates summaries and insights from search results across indexed content
• Remove Index: Deletes existing collections/indexes when you need to clean up or start fresh

​

Configuration Settings:

​

Step-by-Step: Index Xray Data

​

Step 1: Access the Interface

1. Navigate to Toolkits: Go to Toolkits in the main navigation
2. Select Your Xray Toolkit: Choose your configured Xray toolkit from the list
3. Open Indexes Tab: Click on the Indexes tab in the toolkit detail view

• PgVector and Embedding Model are configured in Settings → AI Configuration
• The Index Data tool is enabled in your toolkit configuration

​

Step 2: Create a New Index

1. Click Create New Index: In the Indexes sidebar, click the + Create New Index button
2. New Index Form: The center panel displays the new index creation form

​

Step 3: Configure Index Parameters

​

JQL Query Examples

project = "CALC" AND testType = "Manual"

project = "CALC" AND labels in ("Smoke", "Regression")

project = "CALC" AND summary ~ "login"

project = "CALC" AND testType = "Manual" AND labels in ("Smoke", "Critical")

​

GraphQL Query Examples

query { 
  getTests(jql: "project = \"CALC\"") { 
    results { 
      issueId 
      jira(fields: ["key"]) 
      testType { name } 
      steps { action result } 
    } 
  } 
}

query { 
  getTests(jql: "project = \"CALC\"") { 
    results { 
      issueId 
      jira(fields: ["key", "summary", "description", "created"]) 
      testType { name kind } 
      steps { 
        id 
        action 
        data 
        result 
        attachments { 
          id 
          filename 
          downloadLink 
        } 
      } 
    } 
  } 
}

​

Step 4: Start Indexing

1. Form Validation: The Index button remains inactive until all required fields are filled
2. Review Configuration: Verify all parameters are correct
3. Click Index Button: Start the indexing process
4. Monitor Progress: Watch real-time updates with visual indicators:
   • 🔄 In Progress: Indexing is currently running
   • ✅ Completed: Indexing finished successfully
   • ❌ Failed: Indexing encountered an error

​

Step 5: Verify Index Creation

1. Check Index Status: Visual indicators show completion status
2. Review Index Details: Click on the created index to see metadata and document count
3. Test Search: Use the Run tab to test search functionality with sample queries

​

Step 6: Search Your Indexed Data

1. Access Indexes Tab: Navigate to your Xray toolkit → Indexes tab
2. Select Index: Click on your created index from the sidebar
3. Open Run Tab: Click the Run tab in the center panel
4. Choose Search Tool: Select from available search tools:
   • Search Index: Basic semantic search
   • Stepback Search Index: Advanced search with question breakdown
   • Stepback Summary Index: Summarized insights from search results
5. Enter Query: Type your natural language question
6. View Results: See responses with citations to specific test cases

​

Real-Life Example: Indexing Calculator Project Tests

1. Configure Xray Toolkit:
   • Base URL: https://xray.cloud.getxray.app
   • Client ID: Generated from Xray Cloud API Keys
   • Client Secret: Generated from Xray Cloud API Keys
2. Index All Manual Tests:
   • JQL: project = "CALC" AND testType = "Manual"
   • Collection suffix: manual
   • Progress Step: 10 (report every 10 test cases)
   • Clean Index: ✓ (for fresh start)
   • Include Attachments: ✓ (for test screenshots and documentation)
   • Skip Attachment Extensions: [".exe", ".zip"]
   • Chunking Tool: json
3. Index Smoke Tests (Optional):
   • JQL: project = "CALC" AND labels in ("Smoke", "Critical")
   • Collection suffix: smoke
   • Include Attachments: ✓
4. Index Specific Test Type (Optional):
   • JQL: project = "CALC" AND testType = "Cucumber"
   • Collection suffix: bdd
   • Include Attachments: ✗ (Cucumber tests use text-based Gherkin)
5. Verify indexing:
   • Use “List Collections” tool to confirm collections exist
   • Expected collections: manual, smoke, bdd
   • Check indexing output for test case processing confirmation

• Feature-specific tests: “Find all test cases for addition functionality”
• Test type queries: “What are the steps to test division by zero?”
• Cross-functional testing: “Show me test cases that verify scientific calculator functions”
• Test coverage analysis: “Are there tests for negative number operations?”
• Precondition checks: “What preconditions are needed for advanced calculation tests?”

​

Search and Chat with Indexed Data

​

Using Toolkit in Conversations and Agents

1. In Conversations: Add the toolkit as a participant to ask questions and search your indexed Xray data
2. In Agents: Include the toolkit when creating AI agents to give them access to your Xray test data

• Start a New Conversation or Create an Agent
• Add Toolkit as Participant: Select your Xray toolkit from the available toolkits
• Ask Natural Language Questions: The toolkit will automatically search your indexed data and provide relevant answers with citations

​

Real-Life Example Workflow

• Navigate to Conversations → + New Conversation
• In the participants section click + to add Toolkit
• Select your Xray toolkit from the available toolkits list

• User Request in Chat:

  “Index all manual tests from project CALC with suffix ‘manual’. Include attachments but skip .exe and .zip files. Clean any existing index first.”

• User Request in Chat:

  “List my collections”

• Xray Toolkit Response:

• User: “What are the test steps for verifying addition operations?”
• Xray Toolkit: “Based on your indexed test data, here are the test steps for addition:
  1. CALC-123: Basic Addition Test
     • Action: Enter first number (e.g., 5)
     • Data: First operand
     • Result: Number displayed on screen
     • Action: Click addition operator (+)
     • Result: Operator registered
     • Action: Enter second number (e.g., 3)
     • Data: Second operand
     • Result: Number displayed
     • Action: Click equals (=)
     • Result: Result shows 8
  2. CALC-156: Negative Number Addition
     • Action: Enter negative number (e.g., -10)
     • Result: Negative sign displayed
     • Action: Click addition operator (+)
     • Action: Enter positive number (e.g., 5)
     • Result: Result shows -5

​

Attachment Handling

​

Attachment Processing

1. Extraction: Attachments are extracted from test step definitions
2. Filtering: Extensions in skip_attachment_extensions are skipped
3. Content Extraction: Supported file types have their content extracted and indexed
4. Linking: Attachments are linked to parent test cases via metadata

​

Supported Attachment Types

• Text files: .txt, .md, .csv
• Documents: .pdf, .doc, .docx
• Images: .png, .jpg, .gif (OCR if configured)
• Other: Various file formats based on available parsers

​

Example Configuration

include_attachments: true
skip_attachment_extensions: [".exe", ".zip", ".bin", ".iso", ".dmg"]

​

Metadata and Search Fields

​

Troubleshooting

​

Common Issues and Solutions

• Cause: Neither JQL nor GraphQL query was specified in indexing parameters
• Solution: Provide either a jql query (e.g., project = "CALC") or a graphql query in the index configuration

• Cause: Both JQL and GraphQL parameters were provided simultaneously
• Solution: Choose one query method and remove the other parameter

• Cause: Custom GraphQL query did not return test data in expected format
• Solution: Verify your GraphQL query returns test objects with fields like issueId, jira, testType, and steps

• Cause: Invalid or expired Client ID/Client Secret credentials
• Solution: Regenerate API credentials in Xray Cloud and update your ELITEA credential configuration

• Cause: include_attachments not enabled or all extensions are filtered
• Solution: Enable include_attachments: true and review skip_attachment_extensions list

• Cause: Large number of attachments or large attachment files
• Solution: Use skip_attachment_extensions to filter out large or unnecessary file types (e.g., videos, large images)

• Cause: Search Cut Off score is too high, filtering out potentially relevant results
• Solution: Adjust the Cut Off score in your search tool configuration. Lower values (e.g., 0.3-0.5) return more results, higher values (e.g., 0.7-0.9) return only highly relevant matches. Start with a lower Cut Off and gradually increase if too many irrelevant results appear.

​

Best Practices

​

Indexing Strategy

1. Start Small: Begin with a single project or test type to validate configuration
2. Filter Wisely: Use JQL to index only relevant test cases (active tests, specific labels)
3. Clean Indexes: Enable clean_index when re-indexing to avoid duplicates
4. Monitor Progress: Use appropriate progress_step values (5-20) for visibility

​

Query Optimization

1. Use JQL for Standard Queries: JQL is simpler and sufficient for most use cases
2. Use GraphQL for Complex Needs: GraphQL allows custom field selection and nested data
3. Combine Filters: Use multiple JQL conditions to narrow results (project + testType + labels)

​

Attachment Management

1. Skip Binary Files: Always exclude executables, archives, and system files
2. Skip Large Images: Consider skipping high-resolution images or videos
3. Target Useful Content: Index PDFs, docs, and screenshots that contain searchable content

​

Collection Naming

1. Use Descriptive Suffixes: Choose meaningful 7-character suffixes (manual, smoke, regress)
2. Separate by Purpose: Create different indexes for different test categories or projects
3. Document Conventions: Maintain a naming convention guide for your team

​

Additional Resources