From 81bd6c27d67521b3af4f3ea39e7c1f1afdc838fb Mon Sep 17 00:00:00 2001 From: ridhima-satam Date: Thu, 13 Aug 2026 14:24:35 -0700 Subject: [PATCH 1/4] Update migration tool README --- splunk-ao-migration-tool/README.md | 64 +++++++++++++++++++++++++++--- 1 file changed, 59 insertions(+), 5 deletions(-) diff --git a/splunk-ao-migration-tool/README.md b/splunk-ao-migration-tool/README.md index 250c0f95..d4c579bc 100644 --- a/splunk-ao-migration-tool/README.md +++ b/splunk-ao-migration-tool/README.md @@ -17,7 +17,7 @@ The migration touches four areas: | **Class / symbol names** | `Galileo*` prefix → `SplunkAO*` | | **Environment variables** | `GALILEO_*` → `SPLUNK_AO_*` | -Additionally there are a handful of **removed features** (Protect, `GalileoScorers`) and a **Python version floor bump** (3.10 → 3.11). +Additionally there are a handful of **removed features** (Protect, `GalileoScorers`), a **Python version floor bump** (3.10 → 3.11), and **domain entity renames**: "Metrics" → "Evaluators" and "Log Streams" → "Agent Streams". --- @@ -141,7 +141,7 @@ All sub-module paths follow the same rename pattern: |-----|-----| | `from galileo.decorator import …` | `from splunk_ao.decorator import …` | | `from galileo.logger import …` | `from splunk_ao.logger import …` | -| `from galileo.metric import …` | `from splunk_ao.metric import …` | +| `from galileo.metric import …` | `from splunk_ao.evaluator import …` | | `from galileo.schema.metrics import …` | `from splunk_ao.schema.metrics import …` | | `from galileo.shared.exceptions import …` | `from splunk_ao.shared.exceptions import …` | | `from galileo.exceptions import …` | `from splunk_ao.exceptions import …` | @@ -193,15 +193,63 @@ All sub-module paths follow the same rename pattern: ### 3.3 Metric Classes +> **Domain rename:** "Metrics" → "Evaluators". All classes, types, and enums containing "Metric" are renamed to use "Evaluator". + | Old | New | |-----|-----| -| `GalileoMetric` | `SplunkAOMetric` | +| `GalileoMetric` | `SplunkAOEvaluator` | | `GalileoMetrics` | `SplunkAOEvaluators` | | `GalileoScorers` | **Removed** (see §5.2) | +| `Metric` | `Evaluator` | +| `LlmMetric` | `LlmEvaluator` | +| `LocalMetric` | `LocalEvaluator` | +| `CodeMetric` | `CodeEvaluator` | +| `BuiltInMetrics` | `BuiltInEvaluators` | +| `Metrics` | `Evaluators` | +| `MetricSpec` | `EvaluatorSpec` | +| `LocalMetricConfig` | `LocalEvaluatorConfig` | ```diff - from galileo import GalileoMetric, GalileoMetrics -+ from splunk_ao import SplunkAOMetric, SplunkAOEvaluators ++ from splunk_ao import SplunkAOEvaluator, SplunkAOEvaluators + +- from splunk_ao.metric import Metric, LlmMetric, LocalMetric, CodeMetric ++ from splunk_ao.evaluator import Evaluator, LlmEvaluator, LocalEvaluator, CodeEvaluator +``` + +### 3.3a Log Stream / Agent Stream Classes + +> **Domain rename:** "Log Streams" → "Agent Streams". All classes and methods containing "LogStream" or "log_stream" are renamed to use "AgentStream" / "agent_stream". + +**Class renames:** + +| Old | New | +|-----|-----| +| `LogStream` | `AgentStream` | +| `LogStreams` | `AgentStreams` | + +**Method and property renames:** + +| Class | Old method / property | New method / property | +|-------|-----------------------|-----------------------| +| `AgentStream` (was `LogStream`) | `enable_metrics()` | `enable_evaluators()` | +| `AgentStream` (was `LogStream`) | `get_metrics()` | `get_evaluators()` | +| `Project` | `create_log_stream()` | `create_agent_stream()` | +| `Project` | `list_log_streams()` | `list_agent_streams()` | +| `Project` | `.logstreams` | `.agent_streams` | + +```diff +- from splunk_ao.log_stream import LogStream ++ from splunk_ao.agent_stream import AgentStream + +- project.create_log_stream("prod") ++ project.create_agent_stream("prod") + +- streams = project.list_log_streams() ++ streams = project.list_agent_streams() + +- project.logstreams ++ project.agent_streams ``` ### 3.4 Handlers & Middleware @@ -449,9 +497,15 @@ The following are **unchanged** between galileo and splunk-ao and require no mig - [ ] Rename `GalileoAPIError` → `SplunkAOAPIError` - [ ] Rename `GalileoLoggerException` → `SplunkAOLoggerException` - [ ] Rename `GalileoFutureError` → `SplunkAOFutureError` -- [ ] Rename `GalileoMetric` → `SplunkAOMetric` +- [ ] Rename `GalileoMetric` → `SplunkAOEvaluator` - [ ] Rename `GalileoMetrics` → `SplunkAOEvaluators` - [ ] Replace `GalileoScorers` with `SplunkAOEvaluators` +- [ ] Rename domain evaluator classes: `Metric` → `Evaluator`, `LlmMetric` → `LlmEvaluator`, `LocalMetric` → `LocalEvaluator`, `CodeMetric` → `CodeEvaluator` +- [ ] Rename `BuiltInMetrics` → `BuiltInEvaluators`, `MetricSpec` → `EvaluatorSpec`, `LocalMetricConfig` → `LocalEvaluatorConfig` +- [ ] Update evaluator module imports: `splunk_ao.metric` → `splunk_ao.evaluator` +- [ ] Rename `LogStream` → `AgentStream`, `LogStreams` → `AgentStreams` +- [ ] Update `Project` method calls: `create_log_stream()` → `create_agent_stream()`, `list_log_streams()` → `list_agent_streams()`, `.logstreams` → `.agent_streams` +- [ ] Rename `AgentStream` methods: `enable_metrics()` → `enable_evaluators()`, `get_metrics()` → `get_evaluators()` - [ ] Rename `GalileoAgentControlBridge` → `SplunkAOAgentControlBridge` - [ ] Rename all `GALILEO_*` environment variables to `SPLUNK_AO_*` - [ ] Update `.env`, `.env.example`, CI/CD secrets, and deployment configs From 1c9738e54b3ac1103854e176245d4ba5e4598ba9 Mon Sep 17 00:00:00 2001 From: ridhima-splunk Date: Tue, 18 Aug 2026 13:40:55 -0700 Subject: [PATCH 2/4] Applied review comments --- splunk-ao-migration-tool/README.md | 45 +++++++++++++------ .../examples/after_splunk_ao.py | 2 +- 2 files changed, 33 insertions(+), 14 deletions(-) diff --git a/splunk-ao-migration-tool/README.md b/splunk-ao-migration-tool/README.md index d4c579bc..7d7e28b8 100644 --- a/splunk-ao-migration-tool/README.md +++ b/splunk-ao-migration-tool/README.md @@ -142,7 +142,10 @@ All sub-module paths follow the same rename pattern: | `from galileo.decorator import …` | `from splunk_ao.decorator import …` | | `from galileo.logger import …` | `from splunk_ao.logger import …` | | `from galileo.metric import …` | `from splunk_ao.evaluator import …` | -| `from galileo.schema.metrics import …` | `from splunk_ao.schema.metrics import …` | +| `from galileo.metrics import …` | `from splunk_ao.evaluators import …` | +| `from galileo.log_stream import …` | `from splunk_ao.agent_stream import …` | +| `from galileo.log_streams import …` | `from splunk_ao.agent_streams import …` | +| `from galileo.schema.metrics import …` | `from splunk_ao.schema.metrics import …` (module path unchanged) | | `from galileo.shared.exceptions import …` | `from splunk_ao.shared.exceptions import …` | | `from galileo.exceptions import …` | `from splunk_ao.exceptions import …` | | `from galileo.configuration import …` | `from splunk_ao.configuration import …` | @@ -193,27 +196,27 @@ All sub-module paths follow the same rename pattern: ### 3.3 Metric Classes -> **Domain rename:** "Metrics" → "Evaluators". All classes, types, and enums containing "Metric" are renamed to use "Evaluator". +> **Domain rename:** "Metrics" → "Evaluators". The object-API classes (from `galileo.metric`) and the built-in scorer accessor are renamed. The Pydantic schema models in `splunk_ao.schema.metrics` (`Metric`, `LocalMetricConfig`) and the `MetricSpec` type alias keep their existing names — do **not** blanket find-and-replace "Metric" → "Evaluator". | Old | New | |-----|-----| | `GalileoMetric` | `SplunkAOEvaluator` | | `GalileoMetrics` | `SplunkAOEvaluators` | | `GalileoScorers` | **Removed** (see §5.2) | -| `Metric` | `Evaluator` | +| `Metric` (OO base class from `galileo.metric`) | `Evaluator` | | `LlmMetric` | `LlmEvaluator` | | `LocalMetric` | `LocalEvaluator` | | `CodeMetric` | `CodeEvaluator` | | `BuiltInMetrics` | `BuiltInEvaluators` | | `Metrics` | `Evaluators` | -| `MetricSpec` | `EvaluatorSpec` | -| `LocalMetricConfig` | `LocalEvaluatorConfig` | +| `MetricSpec` | `MetricSpec` — **not renamed** (still `from splunk_ao import MetricSpec`) | +| `LocalMetricConfig` | `LocalMetricConfig` — **not renamed** (still `from splunk_ao.schema.metrics import LocalMetricConfig`) | ```diff - from galileo import GalileoMetric, GalileoMetrics + from splunk_ao import SplunkAOEvaluator, SplunkAOEvaluators -- from splunk_ao.metric import Metric, LlmMetric, LocalMetric, CodeMetric +- from galileo.metric import Metric, LlmMetric, LocalMetric, CodeMetric + from splunk_ao.evaluator import Evaluator, LlmEvaluator, LocalEvaluator, CodeEvaluator ``` @@ -232,14 +235,17 @@ All sub-module paths follow the same rename pattern: | Class | Old method / property | New method / property | |-------|-----------------------|-----------------------| -| `AgentStream` (was `LogStream`) | `enable_metrics()` | `enable_evaluators()` | -| `AgentStream` (was `LogStream`) | `get_metrics()` | `get_evaluators()` | +| `AgentStream` (was `LogStream`) | `enable_metrics()` | `set_metrics()` | +| `AgentStreams` service / `splunk_ao.agent_streams` module-level helper | `enable_metrics()` | `enable_evaluators()` | +| `splunk_ao.evaluators` module-level helper (was `galileo.metrics`) | `get_metrics()` | `get_evaluators()` | | `Project` | `create_log_stream()` | `create_agent_stream()` | | `Project` | `list_log_streams()` | `list_agent_streams()` | | `Project` | `.logstreams` | `.agent_streams` | +> `AgentStream.get_metrics()` is **unchanged** — it still returns the names of the evaluators enabled on the stream. + ```diff -- from splunk_ao.log_stream import LogStream +- from galileo.log_stream import LogStream + from splunk_ao.agent_stream import AgentStream - project.create_log_stream("prod") @@ -252,6 +258,18 @@ All sub-module paths follow the same rename pattern: + project.agent_streams ``` +**Module-level convenience function renames:** + +| Old (`galileo`) | New (`splunk_ao`) | +|-----------------|-------------------| +| `get_log_stream()` | `get_agent_stream()` (`splunk_ao.agent_streams`) | +| `list_log_streams()` | `list_agent_streams()` (`splunk_ao.agent_streams`) | +| `create_log_stream()` | `create_agent_stream()` (`splunk_ao.agent_streams`) | +| `enable_metrics()` | `enable_evaluators()` (`splunk_ao.agent_streams`) | +| `get_metrics()` | `get_evaluators()` (`splunk_ao.evaluators`) | +| `delete_metric()` | `delete_evaluator()` (`splunk_ao.evaluators`) | +| `create_custom_llm_metric()` | `create_custom_llm_evaluator()` (`splunk_ao.evaluators`) | + ### 3.4 Handlers & Middleware | Old | New | @@ -461,7 +479,7 @@ with splunk_ao_context(project="my-project", agent_stream="production"): result = call_llm("Hello") # Direct logger approach -# project/log_stream are constructor args, not start_session args +# project/agent_stream are constructor args, not start_session args logger = SplunkAOLogger(project="my-project", agent_stream="production") logger.start_session(name="my-session") logger.add_llm_span(input="Hello", output="Hi", model="gpt-4") @@ -500,12 +518,13 @@ The following are **unchanged** between galileo and splunk-ao and require no mig - [ ] Rename `GalileoMetric` → `SplunkAOEvaluator` - [ ] Rename `GalileoMetrics` → `SplunkAOEvaluators` - [ ] Replace `GalileoScorers` with `SplunkAOEvaluators` -- [ ] Rename domain evaluator classes: `Metric` → `Evaluator`, `LlmMetric` → `LlmEvaluator`, `LocalMetric` → `LocalEvaluator`, `CodeMetric` → `CodeEvaluator` -- [ ] Rename `BuiltInMetrics` → `BuiltInEvaluators`, `MetricSpec` → `EvaluatorSpec`, `LocalMetricConfig` → `LocalEvaluatorConfig` +- [ ] Rename domain evaluator classes: `Metric` → `Evaluator` (OO class only, **not** `splunk_ao.schema.metrics.Metric`), `LlmMetric` → `LlmEvaluator`, `LocalMetric` → `LocalEvaluator`, `CodeMetric` → `CodeEvaluator` +- [ ] Rename `BuiltInMetrics` → `BuiltInEvaluators` (note: `MetricSpec` and `LocalMetricConfig` are **not** renamed) - [ ] Update evaluator module imports: `splunk_ao.metric` → `splunk_ao.evaluator` - [ ] Rename `LogStream` → `AgentStream`, `LogStreams` → `AgentStreams` - [ ] Update `Project` method calls: `create_log_stream()` → `create_agent_stream()`, `list_log_streams()` → `list_agent_streams()`, `.logstreams` → `.agent_streams` -- [ ] Rename `AgentStream` methods: `enable_metrics()` → `enable_evaluators()`, `get_metrics()` → `get_evaluators()` +- [ ] Replace `log_stream.enable_metrics()` with `agent_stream.set_metrics()` (use `enable_evaluators()` only on the `AgentStreams` service / module-level helper); `AgentStream.get_metrics()` is unchanged +- [ ] Rename module-level helpers: `get_metrics()` → `get_evaluators()`, `delete_metric()` → `delete_evaluator()`, `create_custom_llm_metric()` → `create_custom_llm_evaluator()`, `get_log_stream()`/`list_log_streams()`/`create_log_stream()` → `get_agent_stream()`/`list_agent_streams()`/`create_agent_stream()` - [ ] Rename `GalileoAgentControlBridge` → `SplunkAOAgentControlBridge` - [ ] Rename all `GALILEO_*` environment variables to `SPLUNK_AO_*` - [ ] Update `.env`, `.env.example`, CI/CD secrets, and deployment configs diff --git a/splunk-ao-migration-tool/examples/after_splunk_ao.py b/splunk-ao-migration-tool/examples/after_splunk_ao.py index 6ee59db7..75efd064 100644 --- a/splunk-ao-migration-tool/examples/after_splunk_ao.py +++ b/splunk-ao-migration-tool/examples/after_splunk_ao.py @@ -16,7 +16,7 @@ def call_llm(prompt: str) -> str: result = call_llm("Hello") # Direct logger approach -# project/log_stream are constructor args, not start_session args +# project/agent_stream are constructor args, not start_session args logger = SplunkAOLogger(project="my-project", agent_stream="production") logger.start_session(name="my-session") logger.add_llm_span(input="Hello", output="Hi", model="gpt-4") From f6a756a213425b42fb910ae85f4ddb077b0621ce Mon Sep 17 00:00:00 2001 From: ridhima-splunk Date: Tue, 18 Aug 2026 14:38:56 -0700 Subject: [PATCH 3/4] Added BuiltInEvaluators --- splunk-ao-migration-tool/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/splunk-ao-migration-tool/README.md b/splunk-ao-migration-tool/README.md index 7d7e28b8..48f4a252 100644 --- a/splunk-ao-migration-tool/README.md +++ b/splunk-ao-migration-tool/README.md @@ -207,7 +207,7 @@ All sub-module paths follow the same rename pattern: | `LlmMetric` | `LlmEvaluator` | | `LocalMetric` | `LocalEvaluator` | | `CodeMetric` | `CodeEvaluator` | -| `BuiltInMetrics` | `BuiltInEvaluators` | +| `BuiltInMetrics` | `BuiltInEvaluators` — `from splunk_ao.evaluator import BuiltInEvaluators` (not re-exported at top level; typically accessed via `Evaluator.metrics`) | | `Metrics` | `Evaluators` | | `MetricSpec` | `MetricSpec` — **not renamed** (still `from splunk_ao import MetricSpec`) | | `LocalMetricConfig` | `LocalMetricConfig` — **not renamed** (still `from splunk_ao.schema.metrics import LocalMetricConfig`) | @@ -519,7 +519,7 @@ The following are **unchanged** between galileo and splunk-ao and require no mig - [ ] Rename `GalileoMetrics` → `SplunkAOEvaluators` - [ ] Replace `GalileoScorers` with `SplunkAOEvaluators` - [ ] Rename domain evaluator classes: `Metric` → `Evaluator` (OO class only, **not** `splunk_ao.schema.metrics.Metric`), `LlmMetric` → `LlmEvaluator`, `LocalMetric` → `LocalEvaluator`, `CodeMetric` → `CodeEvaluator` -- [ ] Rename `BuiltInMetrics` → `BuiltInEvaluators` (note: `MetricSpec` and `LocalMetricConfig` are **not** renamed) +- [ ] Rename `BuiltInMetrics` → `BuiltInEvaluators` (`from splunk_ao.evaluator import BuiltInEvaluators`; not re-exported at top level — note: `MetricSpec` and `LocalMetricConfig` are **not** renamed) - [ ] Update evaluator module imports: `splunk_ao.metric` → `splunk_ao.evaluator` - [ ] Rename `LogStream` → `AgentStream`, `LogStreams` → `AgentStreams` - [ ] Update `Project` method calls: `create_log_stream()` → `create_agent_stream()`, `list_log_streams()` → `list_agent_streams()`, `.logstreams` → `.agent_streams` From 74e7ec48f9f126833d8c0e8461c8ce6a92c71e1c Mon Sep 17 00:00:00 2001 From: ridhima-splunk Date: Wed, 19 Aug 2026 11:14:28 -0700 Subject: [PATCH 4/4] applied new comment fixes --- splunk-ao-migration-tool/README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/splunk-ao-migration-tool/README.md b/splunk-ao-migration-tool/README.md index 48f4a252..93d164f0 100644 --- a/splunk-ao-migration-tool/README.md +++ b/splunk-ao-migration-tool/README.md @@ -235,14 +235,15 @@ All sub-module paths follow the same rename pattern: | Class | Old method / property | New method / property | |-------|-----------------------|-----------------------| -| `AgentStream` (was `LogStream`) | `enable_metrics()` | `set_metrics()` | +| `AgentStream` (`from splunk_ao import AgentStream`) | `set_metrics()` / `get_metrics()` | **unchanged** | +| `AgentStream` returned by `get_agent_stream()` / `create_agent_stream()` | `enable_metrics()` | `enable_evaluators()` | | `AgentStreams` service / `splunk_ao.agent_streams` module-level helper | `enable_metrics()` | `enable_evaluators()` | | `splunk_ao.evaluators` module-level helper (was `galileo.metrics`) | `get_metrics()` | `get_evaluators()` | | `Project` | `create_log_stream()` | `create_agent_stream()` | | `Project` | `list_log_streams()` | `list_agent_streams()` | | `Project` | `.logstreams` | `.agent_streams` | -> `AgentStream.get_metrics()` is **unchanged** — it still returns the names of the evaluators enabled on the stream. +> `AgentStream.set_metrics()` and `AgentStream.get_metrics()` are **unchanged** — they still set and return the names of the evaluators enabled on the stream. ```diff - from galileo.log_stream import LogStream @@ -520,10 +521,10 @@ The following are **unchanged** between galileo and splunk-ao and require no mig - [ ] Replace `GalileoScorers` with `SplunkAOEvaluators` - [ ] Rename domain evaluator classes: `Metric` → `Evaluator` (OO class only, **not** `splunk_ao.schema.metrics.Metric`), `LlmMetric` → `LlmEvaluator`, `LocalMetric` → `LocalEvaluator`, `CodeMetric` → `CodeEvaluator` - [ ] Rename `BuiltInMetrics` → `BuiltInEvaluators` (`from splunk_ao.evaluator import BuiltInEvaluators`; not re-exported at top level — note: `MetricSpec` and `LocalMetricConfig` are **not** renamed) -- [ ] Update evaluator module imports: `splunk_ao.metric` → `splunk_ao.evaluator` +- [ ] Update evaluator module imports: `galileo.metric` → `splunk_ao.evaluator` - [ ] Rename `LogStream` → `AgentStream`, `LogStreams` → `AgentStreams` - [ ] Update `Project` method calls: `create_log_stream()` → `create_agent_stream()`, `list_log_streams()` → `list_agent_streams()`, `.logstreams` → `.agent_streams` -- [ ] Replace `log_stream.enable_metrics()` with `agent_stream.set_metrics()` (use `enable_evaluators()` only on the `AgentStreams` service / module-level helper); `AgentStream.get_metrics()` is unchanged +- [ ] `AgentStream.set_metrics()` and `AgentStream.get_metrics()` are **unchanged**; replace `enable_metrics()` with `enable_evaluators()` on the `AgentStream` returned by `get_agent_stream()`/`create_agent_stream()`, on the `AgentStreams` service, and on the module-level helper - [ ] Rename module-level helpers: `get_metrics()` → `get_evaluators()`, `delete_metric()` → `delete_evaluator()`, `create_custom_llm_metric()` → `create_custom_llm_evaluator()`, `get_log_stream()`/`list_log_streams()`/`create_log_stream()` → `get_agent_stream()`/`list_agent_streams()`/`create_agent_stream()` - [ ] Rename `GalileoAgentControlBridge` → `SplunkAOAgentControlBridge` - [ ] Rename all `GALILEO_*` environment variables to `SPLUNK_AO_*`