Skip to content

HYBIM-898 Update migration tool README(New PR, DO NOT REVIEW) #229

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ridhima-splunk wants to merge 1 commit into
base: main
Choose a base branch
from
+81 −8
Open

HYBIM-898 Update migration tool README(New PR, DO NOT REVIEW) #229

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 80 additions & 7 deletions splunk-ao-migration-tool/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ The migration touches four areas:
| **Class / symbol names** | `Galileo*` prefix → `SplunkAO*` |
| **Environment variables** | `GALILEO_*` → `SPLUNK_AO_*` |

Additionally there are a handful of **removed features** (Protect, `GalileoScorers`) and a **Python version floor bump** (3.10 → 3.11).
Additionally there are a handful of **removed features** (Protect, `GalileoScorers`), a **Python version floor bump** (3.10 → 3.11), and **domain entity renames**: "Metrics" → "Evaluators" and "Log Streams" → "Agent Streams".

---

Expand Down Expand Up @@ -141,8 +141,11 @@ All sub-module paths follow the same rename pattern:
|-----|-----|
| `from galileo.decorator import …` | `from splunk_ao.decorator import …` |
| `from galileo.logger import …` | `from splunk_ao.logger import …` |
| `from galileo.metric import …` | `from splunk_ao.metric import …` |
| `from galileo.schema.metrics import …` | `from splunk_ao.schema.metrics import …` |
| `from galileo.metric import …` | `from splunk_ao.evaluator import …` |
| `from galileo.metrics import …` | `from splunk_ao.evaluators import …` |
| `from galileo.log_stream import …` | `from splunk_ao.agent_stream import …` |
| `from galileo.log_streams import …` | `from splunk_ao.agent_streams import …` |
| `from galileo.schema.metrics import …` | `from splunk_ao.schema.metrics import …` (module path unchanged) |
| `from galileo.shared.exceptions import …` | `from splunk_ao.shared.exceptions import …` |
| `from galileo.exceptions import …` | `from splunk_ao.exceptions import …` |
| `from galileo.configuration import …` | `from splunk_ao.configuration import …` |
Expand Down Expand Up @@ -193,17 +196,80 @@ All sub-module paths follow the same rename pattern:

### 3.3 Metric Classes

> **Domain rename:** "Metrics" → "Evaluators". The object-API classes (from `galileo.metric`) and the built-in scorer accessor are renamed. The Pydantic schema models in `splunk_ao.schema.metrics` (`Metric`, `LocalMetricConfig`) and the `MetricSpec` type alias keep their existing names — do **not** blanket find-and-replace "Metric" → "Evaluator".

| Old | New |
|-----|-----|
| `GalileoMetric` | `SplunkAOMetric` |
| `GalileoMetric` | `SplunkAOEvaluator` |
| `GalileoMetrics` | `SplunkAOEvaluators` |
| `GalileoScorers` | **Removed** (see §5.2) |
| `Metric` (OO base class from `galileo.metric`) | `Evaluator` |
| `LlmMetric` | `LlmEvaluator` |
| `LocalMetric` | `LocalEvaluator` |
| `CodeMetric` | `CodeEvaluator` |
| `BuiltInMetrics` | `BuiltInEvaluators` — `from splunk_ao.evaluator import BuiltInEvaluators` (not re-exported at top level; typically accessed via `Evaluator.metrics`) |
| `Metrics` | `Evaluators` |
| `MetricSpec` | `MetricSpec` — **not renamed** (still `from splunk_ao import MetricSpec`) |
| `LocalMetricConfig` | `LocalMetricConfig` — **not renamed** (still `from splunk_ao.schema.metrics import LocalMetricConfig`) |

```diff
- from galileo import GalileoMetric, GalileoMetrics
+ from splunk_ao import SplunkAOMetric, SplunkAOEvaluators
+ from splunk_ao import SplunkAOEvaluator, SplunkAOEvaluators

- from galileo.metric import Metric, LlmMetric, LocalMetric, CodeMetric
+ from splunk_ao.evaluator import Evaluator, LlmEvaluator, LocalEvaluator, CodeEvaluator
```

### 3.3a Log Stream / Agent Stream Classes

> **Domain rename:** "Log Streams" → "Agent Streams". All classes and methods containing "LogStream" or "log_stream" are renamed to use "AgentStream" / "agent_stream".

@fercor-cisco fercor-cisco Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 minor (documentation): "All classes and methods containing LogStream or log_stream are renamed" is too strong and will send readers looking for renames that don't exist. The generated transport layer deliberately keeps the old naming: splunk_ao.resources.api.log_stream, LogStreamResponse, LogStreamCreateRequest (imported at src/splunk_ao/agent_streams.py:6-13), and the wire format is unchanged too — the OTLP header is still logstream and the resource attribute still splunk_ao.logstream.name (src/splunk_ao/exporter/config.py:136,154). Scoping the claim to the public object/service APIs avoids the confusion.

Suggested change
> **Domain rename:** "Log Streams" → "Agent Streams". All classes and methods containing "LogStream" or "log_stream" are renamed to use "AgentStream" / "agent_stream".
> **Domain rename:** "Log Streams" → "Agent Streams". Public classes, methods, and module paths containing `LogStream` / `log_stream` are renamed to `AgentStream` / `agent_stream`. The generated transport layer (`splunk_ao.resources.*`, e.g. `LogStreamResponse`) and the on-the-wire OTLP header / resource attribute names are intentionally **unchanged** — server-side renaming is tracked separately.

🤖 Generated by the Astra agent


**Class renames:**

| Old | New |
|-----|-----|
| `LogStream` | `AgentStream` |
| `LogStreams` | `AgentStreams` |

**Method and property renames:**

| Class | Old method / property | New method / property |
|-------|-----------------------|-----------------------|
| `AgentStream` (was `LogStream`) | `enable_metrics()` | `set_metrics()` |

@fercor-cisco fercor-cisco Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major (bug): This row is incorrect in both columns, and following it will break migrating code.

There are two distinct classes named AgentStream (as there were two named LogStream in galileo), and they have disjoint metric APIs:

  • splunk_ao.agent_stream.AgentStream (the object API, exported as from splunk_ao import AgentStream) has get_metrics() and set_metrics() (src/splunk_ao/agent_stream.py:409,443). In galileo, galileo.log_stream.LogStream had exactly the same two methods (f129877:src/galileo/log_stream.py:407,441) — set_metrics() is not a rename of anything; it is unchanged, and enable_metrics() never existed on this class.
  • splunk_ao.agent_streams.AgentStream (the LogStreamResponse subclass returned by AgentStreams().get()/list()/create() and by get_agent_stream() / list_agent_streams() / create_agent_stream()) has only enable_evaluators() (src/splunk_ao/agent_streams.py:125) — no set_metrics().

So the customers who actually wrote log_stream.enable_metrics([...]) are holding the second kind of object, and this row tells them to call .set_metrics(...) on it → AttributeError.

Suggested replacement rows below (the get_metrics() note at line 245 then becomes redundant and can be dropped).

Suggested change
| `AgentStream` (was `LogStream`) | `enable_metrics()` | `set_metrics()` |
| `AgentStream` (object API, `splunk_ao.agent_stream`) | `get_metrics()` / `set_metrics()` | **unchanged** — same names, no migration needed |
| `AgentStream` (service result, `splunk_ao.agent_streams` — returned by `AgentStreams()` and the module-level helpers) | `enable_metrics()` | `enable_evaluators()` |

🤖 Generated by the Astra agent

| `AgentStreams` service / `splunk_ao.agent_streams` module-level helper | `enable_metrics()` | `enable_evaluators()` |
| `splunk_ao.evaluators` module-level helper (was `galileo.metrics`) | `get_metrics()` | `get_evaluators()` |
| `Project` | `create_log_stream()` | `create_agent_stream()` |
| `Project` | `list_log_streams()` | `list_agent_streams()` |
| `Project` | `.logstreams` | `.agent_streams` |

> `AgentStream.get_metrics()` is **unchanged** — it still returns the names of the evaluators enabled on the stream.

```diff
- from galileo.log_stream import LogStream
+ from splunk_ao.agent_stream import AgentStream

- project.create_log_stream("prod")
+ project.create_agent_stream("prod")

- streams = project.list_log_streams()
+ streams = project.list_agent_streams()

- project.logstreams
+ project.agent_streams
```

**Module-level convenience function renames:**

| Old (`galileo`) | New (`splunk_ao`) |
|-----------------|-------------------|
| `get_log_stream()` | `get_agent_stream()` (`splunk_ao.agent_streams`) |
| `list_log_streams()` | `list_agent_streams()` (`splunk_ao.agent_streams`) |
| `create_log_stream()` | `create_agent_stream()` (`splunk_ao.agent_streams`) |
| `enable_metrics()` | `enable_evaluators()` (`splunk_ao.agent_streams`) |
| `get_metrics()` | `get_evaluators()` (`splunk_ao.evaluators`) |
| `delete_metric()` | `delete_evaluator()` (`splunk_ao.evaluators`) |
| `create_custom_llm_metric()` | `create_custom_llm_evaluator()` (`splunk_ao.evaluators`) |
Comment on lines +261 to +271

@fercor-cisco fercor-cisco Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major (documentation): The rename also changed public keyword-argument names, which this guide never mentions. Since these are keyword-only or commonly passed by keyword, migrated code that only applies the documented class/function renames still fails with TypeError at runtime. Verified renames (galileo → splunk-ao):

Callable Old keyword New keyword
enable_evaluators() (module-level and AgentStreams method) log_stream_name= agent_stream_name= (src/splunk_ao/agent_streams.py:690,483)
get_evaluators() log_stream_id= agent_stream_id= (src/splunk_ao/evaluators.py:207)
export_records() / Export.records() log_stream_id= agent_stream_id= (src/splunk_ao/export.py:39)
SplunkAOLogger(...) log_stream_id= agent_stream_id= (src/splunk_ao/logger/logger.py:239)
SplunkAOOTLPExporter(...) / SplunkAOSpanProcessor(...) logstream=, log_stream_id= agentstream=, agent_stream_id=
agent_streams.AgentStream(...) ctor log_stream= agent_stream= (src/splunk_ao/agent_streams.py:103)

The OTel pair is worth calling out explicitly: src/splunk_ao/otel.py:58-64 rejects logstream/log_stream_id with an explicit TypeError, so it is a hard failure rather than a silently-ignored kwarg. Suggest adding a short "keyword argument renames" subsection here (plus a checklist item in §9) with the rule any log_stream/log_stream_id/log_stream_name keyword becomes agent_stream* and the OTel logstream= → agentstream= exception to the underscore convention.

🤖 Generated by the Astra agent


### 3.4 Handlers & Middleware

| Old | New |
Expand Down Expand Up @@ -413,7 +479,7 @@ with splunk_ao_context(project="my-project", agent_stream="production"):
result = call_llm("Hello")

# Direct logger approach
# project/log_stream are constructor args, not start_session args
# project/agent_stream are constructor args, not start_session args
logger = SplunkAOLogger(project="my-project", agent_stream="production")
logger.start_session(name="my-session")
logger.add_llm_span(input="Hello", output="Hi", model="gpt-4")
Expand Down Expand Up @@ -449,9 +515,16 @@ The following are **unchanged** between galileo and splunk-ao and require no mig
- [ ] Rename `GalileoAPIError` → `SplunkAOAPIError`
- [ ] Rename `GalileoLoggerException` → `SplunkAOLoggerException`
- [ ] Rename `GalileoFutureError` → `SplunkAOFutureError`
- [ ] Rename `GalileoMetric` → `SplunkAOMetric`
- [ ] Rename `GalileoMetric` → `SplunkAOEvaluator`
- [ ] Rename `GalileoMetrics` → `SplunkAOEvaluators`
- [ ] Replace `GalileoScorers` with `SplunkAOEvaluators`
- [ ] Rename domain evaluator classes: `Metric` → `Evaluator` (OO class only, **not** `splunk_ao.schema.metrics.Metric`), `LlmMetric` → `LlmEvaluator`, `LocalMetric` → `LocalEvaluator`, `CodeMetric` → `CodeEvaluator`
- [ ] Rename `BuiltInMetrics` → `BuiltInEvaluators` (`from splunk_ao.evaluator import BuiltInEvaluators`; not re-exported at top level — note: `MetricSpec` and `LocalMetricConfig` are **not** renamed)
- [ ] Update evaluator module imports: `splunk_ao.metric` → `splunk_ao.evaluator`
- [ ] Rename `LogStream` → `AgentStream`, `LogStreams` → `AgentStreams`
- [ ] Update `Project` method calls: `create_log_stream()` → `create_agent_stream()`, `list_log_streams()` → `list_agent_streams()`, `.logstreams` → `.agent_streams`
- [ ] Replace `log_stream.enable_metrics()` with `agent_stream.set_metrics()` (use `enable_evaluators()` only on the `AgentStreams` service / module-level helper); `AgentStream.get_metrics()` is unchanged

@fercor-cisco fercor-cisco Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major (bug): Same defect as the table row at line 238, and the parenthetical makes it worse: enable_evaluators() is not restricted to the service class and module-level helper — it is also the method on the AgentStream instances those helpers return (src/splunk_ao/agent_streams.py:125). Conversely set_metrics() exists only on the object-API splunk_ao.agent_stream.AgentStream, where it was already called set_metrics() in galileo, so it is not a migration step at all.

Suggested change
- [ ] Replace `log_stream.enable_metrics()` with `agent_stream.set_metrics()` (use `enable_evaluators()` only on the `AgentStreams` service / module-level helper); `AgentStream.get_metrics()` is unchanged
- [ ] Replace `enable_metrics()` with `enable_evaluators()` on the `AgentStreams` service, on `AgentStream` objects returned by `AgentStreams()` / `get_agent_stream()` / `list_agent_streams()` / `create_agent_stream()`, and on the module-level helper — and update the keyword `log_stream_name=` → `agent_stream_name=`
- [ ] Note: `AgentStream.get_metrics()` and `AgentStream.set_metrics()` on the object API (`splunk_ao.agent_stream`) keep their existing names

🤖 Generated by the Astra agent

- [ ] Rename module-level helpers: `get_metrics()` → `get_evaluators()`, `delete_metric()` → `delete_evaluator()`, `create_custom_llm_metric()` → `create_custom_llm_evaluator()`, `get_log_stream()`/`list_log_streams()`/`create_log_stream()` → `get_agent_stream()`/`list_agent_streams()`/`create_agent_stream()`
- [ ] Rename `GalileoAgentControlBridge` → `SplunkAOAgentControlBridge`
- [ ] Rename all `GALILEO_*` environment variables to `SPLUNK_AO_*`
- [ ] Update `.env`, `.env.example`, CI/CD secrets, and deployment configs
Expand Down
2 changes: 1 addition & 1 deletion splunk-ao-migration-tool/examples/after_splunk_ao.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ def call_llm(prompt: str) -> str:
result = call_llm("Hello")

# Direct logger approach
# project/log_stream are constructor args, not start_session args
# project/agent_stream are constructor args, not start_session args
logger = SplunkAOLogger(project="my-project", agent_stream="production")
logger.start_session(name="my-session")
logger.add_llm_span(input="Hello", output="Hi", model="gpt-4")
Expand Down
Loading