Overview
Accessing Monitoring
To access the Monitoring section:- Click on the Settings icon in the main navigation sidebar.
- Select Monitoring from the settings menu.
- The Monitoring interface will display with two main sections available:
- Monitoring - Main landing page with project groups management
- Monitoring Data - Detailed analytics dashboard with filters, metrics, and charts
Managing Groups:
To manage project groups:- Click the Edit icon (pencil icon) on the right side of the Groups header.
- The tag editor will open, allowing you to add or remove group tags.
- Click Save to apply changes or Cancel to discard.
- Portfolio Organization: Group related projects together for unified monitoring and analysis
- Cross-Project Analytics: View aggregated metrics across all projects within a group using the treemap visualization
- Efficient Filtering: Quickly filter monitoring data by selecting a group instead of individual projects
- Hierarchical Navigation: Drill down from group-level insights to individual project details
- Flexible Categorization: Assign projects to multiple groups based on different criteria (e.g., department, product line, environment)
Monitoring Data Dashboard
To access the detailed analytics dashboard:- From the Monitoring landing page, click the Download button.
- The Monitoring Data dashboard will display with comprehensive filters, metrics, and charts.
Filter Options
Configure the data displayed using the following filters located at the top of the dashboard:
| Filter | Description | Options | Details |
|---|---|---|---|
| Projects | Select projects or project groups to monitor | • Single project selection • Project group selection (shows all projects in the group) • “All Projects” option when no selection is made | Choose specific projects or use groups to view aggregated metrics. Groups enable portfolio-level monitoring across multiple related projects |
| From | Start date for the monitoring period | Date picker | Defines the beginning of the time range for data analysis. Works in conjunction with the To date to set the monitoring window |
| To | End date for the monitoring period | Date picker | Defines the end of the time range, cannot exceed today’s date. Combined with From date to establish the data collection period |
| Aggregation | Time period aggregation | • Hour • Day • Week • Two weeks • Three weeks • Month | Determines how data points are grouped in charts. Smaller periods (Hour) show detailed trends, larger periods (Month) show long-term patterns |
| Type | Select entity types to monitor | • Agents (labeled as “applications” in data) • Conversations • “All Types” when no selection is made | Filters monitoring data by entity category. Selecting a specific type updates the Name dropdown and affects which metrics are displayed |
| Name | Select specific agents or conversations by name | • Dropdown shows items matching selected Type(s) • “All Items” when no selection is made | Narrows monitoring to specific entities(e.g.,Agents). Dynamically populated based on Type selection. Disabled until at least one project is selected |
| Users | Select specific users to monitor | ”All Users” when no selection is made | Filters data by user activity. Useful for analyzing individual or team performance. Disabled until at least one project is selected |
| Refresh | Apply filter changes and update all metrics and charts | Action button | Executes the current filter configuration and reloads all dashboard data including metrics and charts |
| Reset Filters | Clear all filters and return to default values | Action button | Restores all filters to their initial state, useful for starting a new analysis from scratch |
Key Metrics
Below the filter options, key metrics provide a snapshot of the current monitoring period:
| Metric | Description | Details |
|---|---|---|
| Users | Total number of users | Counts unique users who interacted with the application during the selected time period. Helps track user base growth and adoption |
| Tokens in | Number of tokens consumed (input to LLM) | Measures the total input tokens sent to language models. Useful for understanding request complexity and API usage costs |
| Tokens out | Number of tokens generated (output from LLM) | Measures the total output tokens generated by language models. Indicates response volume and helps track generation costs |
| Engagement | Percentage representing the total engagement rate | Shows the ratio of active users to total users who logged in. Higher percentages indicate stronger user interaction with the platform |
| Acceptance rate | Percentage of interactions where users accepted outputs | Reflects user satisfaction by measuring how often generated outputs are copied, downloaded, or saved. Key quality indicator |
| Agents | Total number of agents (applications) | Count of configured agents in the system. Only displayed when Type filter is set to “All Types” to provide comprehensive overview |
| Conversations | Total number of conversations | Count of conversation instances created. Only displayed when Type filter is set to “All Types” to provide comprehensive overview |
Available Charts
The Monitoring Data dashboard includes several chart sections to visualize different aspects of application usage:Project Treemap
When a project group is selected (or “All Projects”), a treemap visualization displays:- Visual representation of projects within the group
- Hierarchical view showing group and project levels
- Click on group or project elements to drill down into specific data
Adoption and Usage
This section displays two charts side by side:-
Active Users: Bar chart showing active and inactive users over time
- Stacked bars showing active users and inactive users
- Y-axis: Number of users
- X-axis: Date (formatted based on aggregation period)
-
Token Usage: Line chart showing token consumption and generation
- Displays “Tokens in” and “Tokens out” metrics over time
- X-axis: Date (formatted based on aggregation period)
Acceptance Rate
A comprehensive chart showing user acceptance of generated outputs:- Displays accepted vs. not accepted interactions over time
- Helps assess user satisfaction with generated outputs
- Shows trends in output acceptance
Some charts may be hidden based on system configuration. Additional chart sections such as Sentiments (Human Input and LLM Output sentiment analysis), Accuracy (Relevancy, Reliability, and Prompt Scatter quality metrics), and Topics (Topic Prompt and Topic Chart analysis) may be available depending on your system configuration.
Data Export
Two export buttons are located in the top-right corner of the Monitoring Data dashboard:Using Export Features
- Configure your monitoring filters (Projects, Date Range, Type, Name, Users, Aggregation)
- Click Refresh to apply filter settings
- Click either Export raw data or Export acceptance data
- Select your preferred format (JSON, Excel, or CSV)
- The file downloads automatically to your device
Export Raw Data
- JSON: Machine-readable format for data processing and integration
- Excel: Spreadsheet format for analysis and reporting
- All metrics data for the selected time period
- User activity information
- Token usage statistics
- Agent and conversation performance data
- Chart data points based on current filters
Export Acceptance Data
- JSON: Machine-readable format
- Excel: Spreadsheet format
- CSV: Comma-separated values format
- Acceptance rate statistics
- User interaction data (accepted vs. not accepted)
- Acceptance trends over time
- Performance indicators for user satisfaction
Best Practices
Use Project Groups for Portfolio Management
Use Project Groups for Portfolio Management
Organize related projects using group tags to enable portfolio-level monitoring. This allows you to view aggregated metrics across multiple projects and drill down into specific project details using the treemap visualization.
Configure Filters Before Refreshing
Configure Filters Before Refreshing
Set all desired filters (Projects, Date Range, Type, Name, Users, Aggregation) before clicking Refresh to minimize API calls and improve performance. Use Reset Filters to quickly return to default values.
Choose Appropriate Aggregation Periods
Choose Appropriate Aggregation Periods
Select aggregation periods based on your analysis needs:
- Hour: For detailed short-term analysis or debugging
- Day: For weekly or monthly trend analysis
- Week/Two weeks/Three weeks: For quarterly reviews
- Month: For long-term trends and annual reporting
Monitor Acceptance Rate Regularly
Monitor Acceptance Rate Regularly
Track the acceptance rate metric to gauge user satisfaction with generated outputs. A declining acceptance rate may indicate issues with output quality or relevance that require prompt attention.
Export Data for Historical Analysis
Export Data for Historical Analysis
Regularly export monitoring data to build a historical database. This enables long-term trend analysis, compliance reporting, and comparison of performance across different time periods.
Use Sentiment Analysis for Quality Insights
Use Sentiment Analysis for Quality Insights
Review the Sentiments charts to understand user input patterns and LLM output quality. Comparing Human Input sentiment with LLM Output sentiment can reveal opportunities for improving agent responses.
Leverage Session Storage for Workflow Efficiency
Leverage Session Storage for Workflow Efficiency
Your filter preferences are automatically saved in session storage, allowing you to maintain consistent monitoring views when navigating between different sections of the application.