Add CI/CD gating notebook using list_experiments - #87
Open
gsvigruha wants to merge 4 commits into
Open
Conversation
Adds 04-ci-cd-gating.ipynb, demonstrating how to gate a change on experiment scores using LLMObs.list_experiments() (dd-trace-py#19329). The flow: run an experiment tagged with the current git.commit.sha, pin it as the baseline, change the prompt and commit it, re-run, then fetch both runs back from Datadog by SHA and fail if any evaluator regresses. Datadog holds the state between runs, so no artifacts are passed between CI jobs. The prompt under test lives in prompts/capital_prompt.txt rather than in a notebook cell, so that a commit can actually change the behavior being measured. capitals.csv seeds the dataset on first run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Experiment.__init__ calls resolve_llmobs_git_metadata() per experiment, so the commit SHA is re-resolved on every LLMObs.experiment() call rather than cached at enable() time. Tagging it by hand was unnecessary, and the note claiming otherwise was wrong. The notebook now sets only its own ci.stage tag and lets the SDK supply git.commit.sha / git.repository_url, reading the SHA locally just to know what to look up. The CI snippet sets DD_GIT_COMMIT_SHA, which is more reliable than a shallow CI checkout. Also drop the private _tags access in favor of the tags returned by list_experiments(), and note that the status filtering in find_run() is client-side because list_experiments() exposes no status filter. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
git_is_dirty() was only ever printed, and misleadingly so: step 5 writes the prompt file before committing it, so the tree is legitimately dirty partway through. The caveat it hinted at is now stated in prose instead. REPO_ROOT only existed to supply cwd= to the step 5 git calls, which work as-is from the notebook's directory with an absolute PROMPT_PATH. git_sha() stays — find_run() needs a SHA to look runs up by, and the SDK offers no public way to read the tag back off an Experiment. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_to_score() was 20 lines of guessing between avg/mean/value keys and boolean distributions, hedging over a payload shape I hadn't confirmed. None of it explained anything about CI/CD. Checking a real experiment's aggregates shows evaluators report a distribution keyed by mean/p50/p90/min_value/max_value, so eval_scores() is now a dict comprehension over one SCORE_KEY. Picking a statistic to gate on is a real decision; enumerating key spellings was not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds 04-ci-cd-gating.ipynb, demonstrating how to gate a change on experiment scores using LLMObs.list_experiments() (dd-trace-py#19329).
The flow: run an experiment tagged with the current git.commit.sha, pin it as the baseline, change the prompt and commit it, re-run, then fetch both runs back from Datadog by SHA and fail if any evaluator regresses. Datadog holds the state between runs, so no artifacts are passed between CI jobs.
The prompt under test lives in prompts/capital_prompt.txt rather than in a notebook cell, so that a commit can actually change the behavior being measured. capitals.csv seeds the dataset on first run.