Add 2026-06-llmops-quickstart blog code - #91
Open
CEDipEngineering wants to merge 3 commits into
Open
Conversation
End-to-end LLMOps quickstart on Databricks (customer support ticket classifier): MLflow ChatAgent on a Foundation Model API endpoint, an evaluation gate that promotes a Champion in Unity Catalog, and a Databricks Asset Bundle that deploys schema, experiment, and jobs with batch + real-time inference. - Folder follows the YYYY-MM-[name] convention - README documents setup, structure, data (30 synthetic tickets, no PII), and licenses - LICENSE.md is an unmodified copy of the repo Databricks license - CODEOWNERS entry added for the new folder I have read the contribution guidelines. No sensitive info, no PII, no external dataset. Pending: SME code review approval and internal approval. Co-authored-by: Isaac
CEDipEngineering
requested review from
QuentinAmbard,
alanreese-dbrx,
alexott,
anupkalburgi,
kwulffert23,
matthewmoorcroft and
srinivasadmala
as code owners
June 5, 2026 19:07
Co-authored-by: Isaac
…al, AI Gateway logging Bring the quickstart current (2026) and align it with the MLOps quickstart's step-by-step, best-practices structure. Tested end-to-end across three workspaces (deploy → 4 jobs → approval → governed endpoint → batch + realtime). - Evaluation: replace the hand-rolled accuracy loop with mlflow.genai.evaluate() using a deterministic exact_match gate scorer plus the built-in Correctness LLM judge; every row is captured as an MLflow Trace. - Lifecycle: passing versions register as Challenger; new model_approval.py promotes Challenger→Champion only when run with --params approved=true, wired as a predecessor task to deployment. - Governance: after agents.deploy(), enable AI Gateway inference-table payload logging on the agent endpoint (idempotent). Guardrails/rate limits documented as an FM-endpoint pattern (not supported on custom agent endpoints). - Agent: normalize response content so reasoning models (Claude Sonnet 5, GPT-5) that return structured content blocks work, not just plain-string responses. - Default LLM endpoint → databricks-claude-sonnet-5; pin mlflow>=3.4.0 in the logged model. - Harden stale-deployment cleanup; README updated with the new flow, approval gate, governance notes, UC-privilege prerequisite, and a "before you call it done" checklist.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
End-to-end LLMOps quickstart on Databricks — a customer support ticket classifier
that carries an LLM agent through the full lifecycle: data ingestion → agent build →
evaluation → approval → governed deployment → batch + real-time inference.
Accompanies an upcoming Databricks Community blog post (JIRA TLC-1077).
Contents
2026-06-llmops-quickstart/(follows theYYYY-MM-[name]convention)ChatAgenton a Foundation Model API endpoint (defaultdatabricks-claude-sonnet-5)README.md(setup, structure, prerequisites, governance notes, "before you call it done" checklist, blog-link placeholder)LICENSE.md— unmodified copy of the repo's Databricks licenseCODEOWNERSentry added for the new folder2026 LLMOps building blocks
This quickstart mirrors the pragmatic, step-by-step shape of the MLOps Quickstart, adapted for LLM apps with current (2026) tooling:
mlflow.genai.evaluate()with a deterministicexact_matchgate scorer plus the built-inCorrectnessLLM judge; every prediction captured as an MLflow Trace.Challenger;model_approval.pypromotes toChampiononly when run with--params approved=true, wired as a predecessor to deployment.Testing
Deployed and run end-to-end from scratch on three separate workspaces (two AWS, one Azure): bundle deploy → all four jobs → approval → governed endpoint → batch + real-time inference all verified green. Portability notes (e.g. the serverless runtime identity's Unity Catalog privileges) are captured in the README prerequisites.
Guidelines checklist
YYYY-MM-[folder_name]Blog post link will be added to the README once published.
This pull request and its description were written by Isaac.