Analytics: APIs for Analytics Dashboard

[Ayush8923]

Summary

Currently, we do not have a centralized analytics dashboard where we can track and analyze:

• provider-wise usage,
• provider-wise costs,
• LLM Call Usage
• organization-level usage,
• and evals-related consumption details.

Because of this, whenever organizations ask for their monthly usage or total evals cost, we currently need to manually calculate or fetch the data, which creates operational dependency and overhead.

To solve this, we need to build backend APIs for a centralized analytics dashboard that can provide usage and cost insights across providers, organizations, and different workflows.

Problem Statement

Right now:

• there is no single place to analyze provider usage,
• we cannot easily track which provider incurred the highest cost,
• there is no monthly cost comparison,
• org-level evals usage is not directly visible,
• and support/ops teams need to manually investigate usage-related queries.

This becomes difficult as more organizations start using Kaapi.

Goal

Build backend APIs that will power an analytics dashboard for:

• provider usage tracking,
• provider cost analysis,
• organization-level usage tracking,
• evals usage reporting,
• and monthly cost trends.

[Ayush8923]

@PritamSGB @kartpop For now, in the analytics dashboard, we can start by showing the following usage metrics:

1. Provider-wise usage
2. Evaluation cost usage
3. LLM call usage
4. LLM call cost

If you have any thoughts or suggestions on which stats should be prioritized and shown in the dashboard, please let me know.

[PritamSGB]

Sharing as per priority I am thinking

1. Total no of requests (llm call + llm chain) (grouped by monthly, modality, provider)
2. monthly Total costs (llm call + llm chain) (grouped by monthly, modality, provider)
3. monthly no of eval runs (grouped by monthly, modality, provider)
4. monthly eval run cost (grouped by monthly, modality, provider)

• Here adding additional dimension of modality e.g T-FS-T, S-FS-S, STT, TTS
• For now, we can avoid custom filters, or custom dimension selection (e.g provider, month or combo) if its too complex, but it can reduce need to create multiple charts.