From b59b01db6169a57a0b4df3de2e9d3c0b8f7abe76 Mon Sep 17 00:00:00 2001 From: Andres Garcia Date: Wed, 21 Oct 2026 22:13:16 +0000 Subject: [PATCH 1/6] knowledge_base: add model_serving_endpoint example Add a bundle that registers a Unity Catalog model, trains it with a serverless job (environment_version 4), and serves one version through a model_serving_endpoints resource with AI Gateway rate limits, an inference table, usage tracking, and CAN_QUERY permissions. Dev uses scale to zero and a Small workload; prod uses Medium without scale to zero. The README documents the two-step first deploy with --select. Closes #15 Co-authored-by: Isaac Co-authored-by: omnigent --- .../model_serving_endpoint/.gitignore | 1 + .../model_serving_endpoint/README.md | 307 ++++++++++++++++++ .../model_serving_endpoint/databricks.yml | 60 ++++ .../model.model_serving_endpoint.yml | 40 +++ .../resources/model.registered_model.yml | 7 + .../resources/train_model.experiment.yml | 4 + .../resources/train_model.job.yml | 36 ++ .../model_serving_endpoint/src/train_model.py | 42 +++ 8 files changed, 497 insertions(+) create mode 100644 knowledge_base/model_serving_endpoint/.gitignore create mode 100644 knowledge_base/model_serving_endpoint/README.md create mode 100644 knowledge_base/model_serving_endpoint/databricks.yml create mode 100644 knowledge_base/model_serving_endpoint/resources/model.model_serving_endpoint.yml create mode 100644 knowledge_base/model_serving_endpoint/resources/model.registered_model.yml create mode 100644 knowledge_base/model_serving_endpoint/resources/train_model.experiment.yml create mode 100644 knowledge_base/model_serving_endpoint/resources/train_model.job.yml create mode 100644 knowledge_base/model_serving_endpoint/src/train_model.py diff --git a/knowledge_base/model_serving_endpoint/.gitignore b/knowledge_base/model_serving_endpoint/.gitignore new file mode 100644 index 00000000..15bcc6dd --- /dev/null +++ b/knowledge_base/model_serving_endpoint/.gitignore @@ -0,0 +1 @@ +.databricks diff --git a/knowledge_base/model_serving_endpoint/README.md b/knowledge_base/model_serving_endpoint/README.md new file mode 100644 index 00000000..1ba4c063 --- /dev/null +++ b/knowledge_base/model_serving_endpoint/README.md @@ -0,0 +1,307 @@ +# Model serving endpoint with AI Gateway + +This example deploys a custom model to a Databricks Model Serving endpoint. +It uses a Unity Catalog registered model, a serverless training job, and AI Gateway. + +## What this example shows + +- A Unity Catalog `registered_models` resource. +- A serverless job that trains a small scikit-learn model. +- The job logs the model with MLflow and registers a new version in Unity Catalog. +- The job sets the alias `champion` on the new version. +- A `model_serving_endpoints` resource that serves one version of the model. +- AI Gateway rate limits, an inference table, and usage tracking. +- Scale to zero and a `Small` workload in `dev`. +- No scale to zero and a `Medium` workload in `prod`. +- Endpoint `permissions` that give `CAN_QUERY` to a consumers group. + +``` +src/train_model.py (serverless job train_model) + ↓ log_model + set alias "champion" +.. version N + ↓ entity_version: ${var.model_version} +model_endpoint (AI Gateway: rate limits, inference table, usage tracking) + ↓ +databricks serving-endpoints query +``` + +### Project structure + +``` +. +├── databricks.yml # Bundle name, variables, and the dev and prod targets +├── resources/ +│ ├── model.registered_model.yml # Unity Catalog registered model +│ ├── model.model_serving_endpoint.yml # Serving endpoint with AI Gateway and permissions +│ ├── train_model.experiment.yml # MLflow experiment for the training runs +│ └── train_model.job.yml # Serverless job that trains and registers the model +└── src/ + └── train_model.py # Trains the model, registers a version, sets the alias +``` + +### Why the endpoint uses a version number + +A served entity takes `entity_name` and `entity_version`. +The `entity_version` field takes a version number. +It does not take an alias such as `champion`. +Thus, the variable `model_version` holds the version that the endpoint serves. +Set `model_version` to the version that has the `champion` alias. +The bundle does not move the alias and the endpoint version together. +You keep them aligned when you deploy. + +## Prerequisites + +- A Databricks workspace with Unity Catalog enabled. +- Model Serving available in the workspace region. +- Serverless compute for jobs enabled in the workspace. +- Databricks CLI version 1.20.0 or above. The `bundle validate` check for this example used that version. +- An existing Unity Catalog catalog and schema. The defaults are `main` and `default`. +- These privileges for the user who deploys the bundle: + - `USE CATALOG` on the catalog. + - `USE SCHEMA`, `CREATE MODEL`, and `CREATE TABLE` on the schema. + - Permission to create serving endpoints in the workspace. +- A workspace group for the endpoint consumers. The default is `users`. + +The bundle does not create the catalog or the schema. +The bundle does not delete them when you destroy it. + +## Configure + +Set the variables in `databricks.yml`, or use `--var` on the command line. + +| Variable | Default | Purpose | +| --- | --- | --- | +| `catalog` | `main` | Catalog for the model and the inference table. | +| `schema` | `default` | Schema for the model and the inference table. | +| `model_name` | `iris_classifier` | Name of the registered model, without catalog and schema. | +| `model_version` | `1` | Model version that the endpoint serves. | +| `endpoint_name` | `iris-classifier` | Name of the serving endpoint. | +| `consumers_group` | `users` | Group that gets `CAN_QUERY` on the endpoint. | +| `workload_size` | `Small` (`prod`: `Medium`) | Provisioned concurrency range of the served model. | +| `scale_to_zero` | `true` (`prod`: `false`) | Scale the served model to zero when it gets no traffic. | +| `rate_limit_calls` | `100` (`prod`: `1000`) | Maximum calls per minute for the endpoint. | +| `rate_limit_user_calls` | `20` (`prod`: `100`) | Maximum calls per minute for each user. | + +The `dev` target uses `mode: development`. +In `dev`, the CLI adds the prefix `dev__` to the model name and the endpoint name. +Thus, each developer gets a different model and endpoint. + +The bundle has no `workspace.host`. +The CLI uses the host of your profile. +Use `-p ` to select a profile. + +## First deploy + +The endpoint needs a model version before Databricks can create it. +A new registered model has no versions. +Thus, the first deploy has two steps. + +1. Authenticate the CLI: + + ```sh + databricks auth login --host + ``` + +2. Deploy only the registered model, the experiment, and the training job: + + ```sh + databricks bundle deploy -t dev --select registered_models.model,experiments.train_model_experiment,jobs.train_model + ``` + +3. Run the training job: + + ```sh + databricks bundle run -t dev train_model + ``` + +4. Read the version number in the job output. + The first run creates version `1`. + The output shows a line such as `Deploy this version with: --var model_version=1`. + +5. Optional: make sure that the `champion` alias points to that version: + + ```sh + databricks model-versions get-by-alias main.default.dev__iris_classifier champion + ``` + +6. Deploy the full bundle with that version: + + ```sh + databricks bundle deploy -t dev --var model_version=1 + ``` + +7. Wait until the endpoint is ready. + The first deploy of an endpoint can take 10 minutes or more. + + ```sh + databricks serving-endpoints get dev__iris-classifier + ``` + + The endpoint is ready when `state.ready` is `READY` and `state.config_update` is `NOT_UPDATING`. + +For `prod`, do the same steps with `-t prod`. +In `prod`, the names have no prefix. + +If step 6 fails, correct the cause and run the same command again. +The deploy is safe to run again. +The CLI compares the bundle with the workspace and changes only the differences. + +## Query the endpoint + +Use `databricks serving-endpoints query` with the endpoint name. +The model takes the four iris measurements as input. + +```sh +databricks serving-endpoints query dev__iris-classifier --json '{ + "dataframe_records": [ + {"sepal length (cm)": 5.1, "sepal width (cm)": 3.5, "petal length (cm)": 1.4, "petal width (cm)": 0.2}, + {"sepal length (cm)": 6.7, "sepal width (cm)": 3.0, "petal length (cm)": 5.2, "petal width (cm)": 2.3} + ] +}' +``` + +The response has one prediction for each input record. +Each prediction is a class number: `0`, `1`, or `2`. + +```json +{ + "predictions": [0, 2] +} +``` + +The exact values depend on the trained model. +The response can also contain other fields, such as `served-model-name`. + +The caller needs `CAN_QUERY` on the endpoint. +Members of `consumers_group` have `CAN_QUERY`. + +## AI Gateway settings + +The `ai_gateway` block in `resources/model.model_serving_endpoint.yml` sets these keys. + +| Key | Purpose | `dev` | `prod` | +| --- | --- | --- | --- | +| `rate_limits` with `key: endpoint` | Limits the calls per minute for the full endpoint. | `100` | `1000` | +| `rate_limits` with `key: user` | Limits the calls per minute for each user. | `20` | `100` | +| `rate_limits.renewal_period` | Sets the period of the limit. The API supports only `minute`. | `minute` | `minute` | +| `inference_table_config.enabled` | Logs requests and responses to a Unity Catalog table. | `true` | `true` | +| `inference_table_config.catalog_name` and `schema_name` | Set the location of the inference table. | `catalog`, `schema` | `catalog`, `schema` | +| `inference_table_config.table_name_prefix` | Sets the prefix of the table name. The value is the registered model name. | `dev__iris_classifier` | `iris_classifier` | +| `usage_tracking_config.enabled` | Writes usage data to system tables. | `true` | `true` | + +The inference table name is `.._payload`. + +The example does not set `guardrails` or `fallback_config`. +Those keys apply to external models and foundation model endpoints. + +## Update the model + +1. Run the training job again: + + ```sh + databricks bundle run -t dev train_model + ``` + +2. Read the new version number in the job output. + The job also moves the `champion` alias to the new version. + +3. Deploy the bundle with the new version: + + ```sh + databricks bundle deploy -t dev --var model_version=2 + ``` + +To keep the version in the bundle, change the `model_version` default in `databricks.yml`. +Commit that change. + +### Zero-downtime notes + +- A config update does not stop the endpoint. +- The endpoint serves the old version until the new version is ready. +- Then the endpoint sends all traffic to the new version. +- During the update, `state.config_update` is `IN_PROGRESS`. +- If the new version fails to start, the endpoint continues to serve the old version. +- Do not start a second deploy while an update is `IN_PROGRESS`. + +### Roll back to a prior version + +1. Deploy the bundle with the prior version: + + ```sh + databricks bundle deploy -t dev --var model_version=1 + ``` + +2. Move the `champion` alias back to the prior version: + + ```sh + databricks registered-models set-alias main.default.dev__iris_classifier champion 1 + ``` + +Do not delete the prior model version before the rollback is complete. + +## Edge cases and limits + +- **Cold start with scale to zero.** + When scale to zero is on, the endpoint stops compute after a period with no traffic. + The next request starts the compute again. + That request can take minutes, and it can time out. + Use scale to zero only for `dev` and tests. + The `prod` target sets `scale_to_zero: false`. +- **Endpoint name.** + The name must be unique in the workspace. + The name can contain only letters, numbers, dashes, and underscores. + Databricks limits the length of the name. + The `dev` prefix makes the name longer, so keep `endpoint_name` short. + Two bundles with the same `endpoint_name` and target conflict. +- **Inference table requirements.** + The catalog and the schema must exist before the deploy. + The user who deploys needs `CREATE TABLE` on the schema. + Databricks creates the table when it enables the inference table. + Rows arrive in the table after a delay. They are not available in real time. + To change `catalog_name`, `schema_name`, or `table_name_prefix`, first deploy with `enabled: false`. + Then deploy the new values with `enabled: true`. +- **Cloud and region availability.** + Model Serving is available on AWS, Azure, and Google Cloud. + Some regions do not support Model Serving, AI Gateway, or inference tables. + Some regions do not support GPU workloads. + Before you deploy, examine the Model Serving region list for your cloud. +- **GPU vs CPU.** + This example uses the default `workload_type`, which is `CPU`. + A scikit-learn model does not need a GPU. + For a deep learning model, set `workload_type` on the served entity, for example `GPU_SMALL`. + GPU workloads cost more and take longer to start. +- **Serverless vs classic compute.** + The training job uses serverless compute with `environment_version: "4"`. + This example does not include a classic cluster configuration. + To use a classic cluster, replace `environment_key` with a `job_cluster_key` or `new_cluster`. + Then install `mlflow` and `scikit-learn` as task libraries. +- **No schedules.** + The training job has no schedule or trigger. + It runs only when you start it. +- **No secrets.** + The bundle has no tokens, hosts, or IDs. + The CLI gets credentials from your profile. + +## Clean up + +1. Destroy the bundle: + + ```sh + databricks bundle destroy -t dev + ``` + + The CLI deletes the endpoint, the job, the experiment, and the registered model. + The CLI deletes the endpoint before the registered model, because the endpoint refers to the model. + Unity Catalog deletes all model versions together with the registered model. + +2. Drop the inference table, because the bundle does not own it: + + ```sql + DROP TABLE IF EXISTS main.default.dev__iris_classifier_payload; + ``` + +The bundle does not delete the catalog, the schema, or the usage data in system tables. + +In `prod`, the registered model can have versions that other systems use. +Before you destroy `prod`, make sure that no other endpoint or job uses the model. +To keep the model, add `lifecycle: { prevent_destroy: true }` to the registered model. diff --git a/knowledge_base/model_serving_endpoint/databricks.yml b/knowledge_base/model_serving_endpoint/databricks.yml new file mode 100644 index 00000000..47c48b29 --- /dev/null +++ b/knowledge_base/model_serving_endpoint/databricks.yml @@ -0,0 +1,60 @@ +# This is a Declarative Automation Bundle definition for model_serving_endpoint. +# See https://docs.databricks.com/dev-tools/bundles/index.html for documentation. +bundle: + name: model_serving_endpoint + engine: direct + +variables: + catalog: + description: Existing Unity Catalog catalog for the model and the inference table + default: main + schema: + description: Existing Unity Catalog schema for the model and the inference table + default: default + model_name: + description: Name of the Unity Catalog registered model (without catalog and schema) + default: iris_classifier + model_version: + description: >- + Model version that the endpoint serves. Serving endpoints take a version number, + not an alias. Set it to the version that holds the `champion` alias. + default: "1" + endpoint_name: + description: Name of the serving endpoint. Must be unique in the workspace. + default: iris-classifier + consumers_group: + description: Workspace group that gets CAN_QUERY on the endpoint + default: users + workload_size: + description: Provisioned concurrency range of the served model (Small, Medium, Large) + default: Small + scale_to_zero: + description: Scale the served model down to zero when it gets no traffic + default: true + rate_limit_calls: + description: Maximum calls per minute for the full endpoint + default: 100 + rate_limit_user_calls: + description: Maximum calls per minute for each user + default: 20 + +include: + - resources/*.yml + +targets: + dev: + mode: development + default: true + + prod: + mode: production + workspace: + root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target} + permissions: + - user_name: ${workspace.current_user.userName} + level: CAN_MANAGE + variables: + workload_size: Medium + scale_to_zero: false + rate_limit_calls: 1000 + rate_limit_user_calls: 100 diff --git a/knowledge_base/model_serving_endpoint/resources/model.model_serving_endpoint.yml b/knowledge_base/model_serving_endpoint/resources/model.model_serving_endpoint.yml new file mode 100644 index 00000000..c864e2ff --- /dev/null +++ b/knowledge_base/model_serving_endpoint/resources/model.model_serving_endpoint.yml @@ -0,0 +1,40 @@ +resources: + model_serving_endpoints: + model_endpoint: + name: ${var.endpoint_name} + + config: + served_entities: + - name: iris-classifier + # A Unity Catalog model needs the full three-level name. + entity_name: ${var.catalog}.${var.schema}.${resources.registered_models.model.name} + # The endpoint takes a version number, not an alias such as `champion`. + entity_version: ${var.model_version} + workload_size: ${var.workload_size} + scale_to_zero_enabled: ${var.scale_to_zero} + + ai_gateway: + rate_limits: + - key: endpoint + calls: ${var.rate_limit_calls} + renewal_period: minute + - key: user + calls: ${var.rate_limit_user_calls} + renewal_period: minute + # Log requests and responses to .._payload. + inference_table_config: + enabled: true + catalog_name: ${var.catalog} + schema_name: ${var.schema} + table_name_prefix: ${resources.registered_models.model.name} + # Write usage to the system.serving system tables. + usage_tracking_config: + enabled: true + + tags: + - key: example + value: model_serving_endpoint + + permissions: + - group_name: ${var.consumers_group} + level: CAN_QUERY diff --git a/knowledge_base/model_serving_endpoint/resources/model.registered_model.yml b/knowledge_base/model_serving_endpoint/resources/model.registered_model.yml new file mode 100644 index 00000000..aa2cbe91 --- /dev/null +++ b/knowledge_base/model_serving_endpoint/resources/model.registered_model.yml @@ -0,0 +1,7 @@ +resources: + registered_models: + model: + catalog_name: ${var.catalog} + schema_name: ${var.schema} + name: ${var.model_name} + comment: Iris classifier served by the model_serving_endpoint example (${bundle.target} target) diff --git a/knowledge_base/model_serving_endpoint/resources/train_model.experiment.yml b/knowledge_base/model_serving_endpoint/resources/train_model.experiment.yml new file mode 100644 index 00000000..76546e00 --- /dev/null +++ b/knowledge_base/model_serving_endpoint/resources/train_model.experiment.yml @@ -0,0 +1,4 @@ +resources: + experiments: + train_model_experiment: + name: ${workspace.root_path}/train_model diff --git a/knowledge_base/model_serving_endpoint/resources/train_model.job.yml b/knowledge_base/model_serving_endpoint/resources/train_model.job.yml new file mode 100644 index 00000000..8745b6eb --- /dev/null +++ b/knowledge_base/model_serving_endpoint/resources/train_model.job.yml @@ -0,0 +1,36 @@ +resources: + jobs: + train_model: + name: train_model + + parameters: + - name: model_full_name + default: ${var.catalog}.${var.schema}.${resources.registered_models.model.name} + - name: alias + default: champion + - name: experiment_name + default: ${resources.experiments.train_model_experiment.name} + + tasks: + - task_key: train + description: Train a small scikit-learn model, register a new version in Unity Catalog, and set the alias + environment_key: serverless_env + spark_python_task: + python_file: ../src/train_model.py + parameters: + - --model-full-name + - "{{job.parameters.model_full_name}}" + - --alias + - "{{job.parameters.alias}}" + - --experiment-name + - "{{job.parameters.experiment_name}}" + + environments: + - environment_key: serverless_env + spec: + environment_version: "4" + dependencies: + - mlflow>=3.1 + - scikit-learn + + max_concurrent_runs: 1 diff --git a/knowledge_base/model_serving_endpoint/src/train_model.py b/knowledge_base/model_serving_endpoint/src/train_model.py new file mode 100644 index 00000000..a81e3831 --- /dev/null +++ b/knowledge_base/model_serving_endpoint/src/train_model.py @@ -0,0 +1,42 @@ +"""Train a small scikit-learn model and register it in Unity Catalog. + +Each run creates a new model version and moves the alias to that version. +""" + +import argparse + +import mlflow +from mlflow import MlflowClient +from mlflow.models import infer_signature +from sklearn.datasets import load_iris +from sklearn.linear_model import LogisticRegression + +parser = argparse.ArgumentParser() +parser.add_argument("--model-full-name", required=True) +parser.add_argument("--alias", default="champion") +parser.add_argument("--experiment-name", required=True) +args = parser.parse_args() + +mlflow.set_registry_uri("databricks-uc") +mlflow.set_experiment(args.experiment_name) + +X, y = load_iris(return_X_y=True, as_frame=True) +model = LogisticRegression(max_iter=200).fit(X, y) + +# Unity Catalog requires a model signature. +signature = infer_signature(X, model.predict(X)) + +with mlflow.start_run(): + model_info = mlflow.sklearn.log_model( + sk_model=model, + name="model", + signature=signature, + input_example=X.head(2), + registered_model_name=args.model_full_name, + ) + +version = model_info.registered_model_version +MlflowClient().set_registered_model_alias(args.model_full_name, args.alias, version) + +print(f"Registered {args.model_full_name} version {version} with alias '{args.alias}'.") +print(f"Deploy this version with: --var model_version={version}") From cec5efcd17259218ac95c170732a8b14b70fd58f Mon Sep 17 00:00:00 2001 From: Andres Garcia Date: Wed, 21 Oct 2026 22:18:17 +0000 Subject: [PATCH 2/6] knowledge_base/model_serving_endpoint: shorten README to sibling shape Cut the README from 307 to 65 lines: intro, Prerequisites, Usage with the two-step first deploy and one query command, and four Notes. No config changes. Co-authored-by: Isaac Co-authored-by: omnigent --- .../model_serving_endpoint/README.md | 311 ++---------------- 1 file changed, 33 insertions(+), 278 deletions(-) diff --git a/knowledge_base/model_serving_endpoint/README.md b/knowledge_base/model_serving_endpoint/README.md index 1ba4c063..4ab5672d 100644 --- a/knowledge_base/model_serving_endpoint/README.md +++ b/knowledge_base/model_serving_endpoint/README.md @@ -1,307 +1,62 @@ -# Model serving endpoint with AI Gateway +# Model serving endpoint -This example deploys a custom model to a Databricks Model Serving endpoint. -It uses a Unity Catalog registered model, a serverless training job, and AI Gateway. +This Declarative Automation Bundles example shows how to serve a Unity Catalog model on a Model Serving endpoint. +A serverless job trains a small scikit-learn model, registers a version, and sets the alias `champion`. +The endpoint uses AI Gateway rate limits, an inference table, usage tracking, and `CAN_QUERY` permissions. -## What this example shows - -- A Unity Catalog `registered_models` resource. -- A serverless job that trains a small scikit-learn model. -- The job logs the model with MLflow and registers a new version in Unity Catalog. -- The job sets the alias `champion` on the new version. -- A `model_serving_endpoints` resource that serves one version of the model. -- AI Gateway rate limits, an inference table, and usage tracking. -- Scale to zero and a `Small` workload in `dev`. -- No scale to zero and a `Medium` workload in `prod`. -- Endpoint `permissions` that give `CAN_QUERY` to a consumers group. - -``` -src/train_model.py (serverless job train_model) - ↓ log_model + set alias "champion" -.. version N - ↓ entity_version: ${var.model_version} -model_endpoint (AI Gateway: rate limits, inference table, usage tracking) - ↓ -databricks serving-endpoints query -``` - -### Project structure - -``` -. -├── databricks.yml # Bundle name, variables, and the dev and prod targets -├── resources/ -│ ├── model.registered_model.yml # Unity Catalog registered model -│ ├── model.model_serving_endpoint.yml # Serving endpoint with AI Gateway and permissions -│ ├── train_model.experiment.yml # MLflow experiment for the training runs -│ └── train_model.job.yml # Serverless job that trains and registers the model -└── src/ - └── train_model.py # Trains the model, registers a version, sets the alias -``` - -### Why the endpoint uses a version number - -A served entity takes `entity_name` and `entity_version`. -The `entity_version` field takes a version number. -It does not take an alias such as `champion`. -Thus, the variable `model_version` holds the version that the endpoint serves. -Set `model_version` to the version that has the `champion` alias. -The bundle does not move the alias and the endpoint version together. -You keep them aligned when you deploy. +For more information, refer to the [Model Serving documentation](https://docs.databricks.com/aws/en/machine-learning/model-serving/). ## Prerequisites -- A Databricks workspace with Unity Catalog enabled. -- Model Serving available in the workspace region. -- Serverless compute for jobs enabled in the workspace. -- Databricks CLI version 1.20.0 or above. The `bundle validate` check for this example used that version. -- An existing Unity Catalog catalog and schema. The defaults are `main` and `default`. -- These privileges for the user who deploys the bundle: - - `USE CATALOG` on the catalog. - - `USE SCHEMA`, `CREATE MODEL`, and `CREATE TABLE` on the schema. - - Permission to create serving endpoints in the workspace. -- A workspace group for the endpoint consumers. The default is `users`. - -The bundle does not create the catalog or the schema. -The bundle does not delete them when you destroy it. - -## Configure - -Set the variables in `databricks.yml`, or use `--var` on the command line. +* Databricks CLI v1.20.0 or above +* A workspace with Unity Catalog, Model Serving, and serverless jobs enabled +* An existing catalog and schema (default: `main.default`) +* `USE CATALOG`, plus `USE SCHEMA`, `CREATE MODEL`, and `CREATE TABLE` on the schema -| Variable | Default | Purpose | -| --- | --- | --- | -| `catalog` | `main` | Catalog for the model and the inference table. | -| `schema` | `default` | Schema for the model and the inference table. | -| `model_name` | `iris_classifier` | Name of the registered model, without catalog and schema. | -| `model_version` | `1` | Model version that the endpoint serves. | -| `endpoint_name` | `iris-classifier` | Name of the serving endpoint. | -| `consumers_group` | `users` | Group that gets `CAN_QUERY` on the endpoint. | -| `workload_size` | `Small` (`prod`: `Medium`) | Provisioned concurrency range of the served model. | -| `scale_to_zero` | `true` (`prod`: `false`) | Scale the served model to zero when it gets no traffic. | -| `rate_limit_calls` | `100` (`prod`: `1000`) | Maximum calls per minute for the endpoint. | -| `rate_limit_user_calls` | `20` (`prod`: `100`) | Maximum calls per minute for each user. | +## Usage -The `dev` target uses `mode: development`. -In `dev`, the CLI adds the prefix `dev__` to the model name and the endpoint name. -Thus, each developer gets a different model and endpoint. - -The bundle has no `workspace.host`. -The CLI uses the host of your profile. -Use `-p ` to select a profile. - -## First deploy - -The endpoint needs a model version before Databricks can create it. -A new registered model has no versions. +The endpoint needs a model version, but a new model has no versions. Thus, the first deploy has two steps. -1. Authenticate the CLI: - - ```sh - databricks auth login --host - ``` - -2. Deploy only the registered model, the experiment, and the training job: +1. Deploy only the model, the experiment, and the training job: ```sh - databricks bundle deploy -t dev --select registered_models.model,experiments.train_model_experiment,jobs.train_model + databricks bundle deploy --select registered_models.model,experiments.train_model_experiment,jobs.train_model ``` -3. Run the training job: +2. Run the training job: ```sh - databricks bundle run -t dev train_model + databricks bundle run train_model ``` -4. Read the version number in the job output. - The first run creates version `1`. - The output shows a line such as `Deploy this version with: --var model_version=1`. - -5. Optional: make sure that the `champion` alias points to that version: +3. Read the version number in the job output, for example `--var model_version=1`. - ```sh - databricks model-versions get-by-alias main.default.dev__iris_classifier champion - ``` - -6. Deploy the full bundle with that version: +4. Deploy the full bundle with that version: ```sh - databricks bundle deploy -t dev --var model_version=1 + databricks bundle deploy --var model_version=1 ``` -7. Wait until the endpoint is ready. - The first deploy of an endpoint can take 10 minutes or more. +5. Query the endpoint. In `dev`, the endpoint name has the prefix `dev__`: ```sh - databricks serving-endpoints get dev__iris-classifier + databricks serving-endpoints query dev__iris-classifier --json '{ + "dataframe_records": [ + {"sepal length (cm)": 5.1, "sepal width (cm)": 3.5, "petal length (cm)": 1.4, "petal width (cm)": 0.2} + ] + }' ``` - The endpoint is ready when `state.ready` is `READY` and `state.config_update` is `NOT_UPDATING`. - -For `prod`, do the same steps with `-t prod`. -In `prod`, the names have no prefix. - -If step 6 fails, correct the cause and run the same command again. -The deploy is safe to run again. -The CLI compares the bundle with the workspace and changes only the differences. - -## Query the endpoint - -Use `databricks serving-endpoints query` with the endpoint name. -The model takes the four iris measurements as input. - -```sh -databricks serving-endpoints query dev__iris-classifier --json '{ - "dataframe_records": [ - {"sepal length (cm)": 5.1, "sepal width (cm)": 3.5, "petal length (cm)": 1.4, "petal width (cm)": 0.2}, - {"sepal length (cm)": 6.7, "sepal width (cm)": 3.0, "petal length (cm)": 5.2, "petal width (cm)": 2.3} - ] -}' -``` - -The response has one prediction for each input record. -Each prediction is a class number: `0`, `1`, or `2`. - -```json -{ - "predictions": [0, 2] -} -``` - -The exact values depend on the trained model. -The response can also contain other fields, such as `served-model-name`. - -The caller needs `CAN_QUERY` on the endpoint. -Members of `consumers_group` have `CAN_QUERY`. - -## AI Gateway settings - -The `ai_gateway` block in `resources/model.model_serving_endpoint.yml` sets these keys. - -| Key | Purpose | `dev` | `prod` | -| --- | --- | --- | --- | -| `rate_limits` with `key: endpoint` | Limits the calls per minute for the full endpoint. | `100` | `1000` | -| `rate_limits` with `key: user` | Limits the calls per minute for each user. | `20` | `100` | -| `rate_limits.renewal_period` | Sets the period of the limit. The API supports only `minute`. | `minute` | `minute` | -| `inference_table_config.enabled` | Logs requests and responses to a Unity Catalog table. | `true` | `true` | -| `inference_table_config.catalog_name` and `schema_name` | Set the location of the inference table. | `catalog`, `schema` | `catalog`, `schema` | -| `inference_table_config.table_name_prefix` | Sets the prefix of the table name. The value is the registered model name. | `dev__iris_classifier` | `iris_classifier` | -| `usage_tracking_config.enabled` | Writes usage data to system tables. | `true` | `true` | - -The inference table name is `.._payload`. - -The example does not set `guardrails` or `fallback_config`. -Those keys apply to external models and foundation model endpoints. + The response has one class number for each record, for example `{"predictions": [0]}`. -## Update the model - -1. Run the training job again: - - ```sh - databricks bundle run -t dev train_model - ``` - -2. Read the new version number in the job output. - The job also moves the `champion` alias to the new version. - -3. Deploy the bundle with the new version: - - ```sh - databricks bundle deploy -t dev --var model_version=2 - ``` - -To keep the version in the bundle, change the `model_version` default in `databricks.yml`. -Commit that change. - -### Zero-downtime notes - -- A config update does not stop the endpoint. -- The endpoint serves the old version until the new version is ready. -- Then the endpoint sends all traffic to the new version. -- During the update, `state.config_update` is `IN_PROGRESS`. -- If the new version fails to start, the endpoint continues to serve the old version. -- Do not start a second deploy while an update is `IN_PROGRESS`. - -### Roll back to a prior version - -1. Deploy the bundle with the prior version: - - ```sh - databricks bundle deploy -t dev --var model_version=1 - ``` - -2. Move the `champion` alias back to the prior version: - - ```sh - databricks registered-models set-alias main.default.dev__iris_classifier champion 1 - ``` - -Do not delete the prior model version before the rollback is complete. - -## Edge cases and limits - -- **Cold start with scale to zero.** - When scale to zero is on, the endpoint stops compute after a period with no traffic. - The next request starts the compute again. - That request can take minutes, and it can time out. - Use scale to zero only for `dev` and tests. - The `prod` target sets `scale_to_zero: false`. -- **Endpoint name.** - The name must be unique in the workspace. - The name can contain only letters, numbers, dashes, and underscores. - Databricks limits the length of the name. - The `dev` prefix makes the name longer, so keep `endpoint_name` short. - Two bundles with the same `endpoint_name` and target conflict. -- **Inference table requirements.** - The catalog and the schema must exist before the deploy. - The user who deploys needs `CREATE TABLE` on the schema. - Databricks creates the table when it enables the inference table. - Rows arrive in the table after a delay. They are not available in real time. - To change `catalog_name`, `schema_name`, or `table_name_prefix`, first deploy with `enabled: false`. - Then deploy the new values with `enabled: true`. -- **Cloud and region availability.** - Model Serving is available on AWS, Azure, and Google Cloud. - Some regions do not support Model Serving, AI Gateway, or inference tables. - Some regions do not support GPU workloads. - Before you deploy, examine the Model Serving region list for your cloud. -- **GPU vs CPU.** - This example uses the default `workload_type`, which is `CPU`. - A scikit-learn model does not need a GPU. - For a deep learning model, set `workload_type` on the served entity, for example `GPU_SMALL`. - GPU workloads cost more and take longer to start. -- **Serverless vs classic compute.** - The training job uses serverless compute with `environment_version: "4"`. - This example does not include a classic cluster configuration. - To use a classic cluster, replace `environment_key` with a `job_cluster_key` or `new_cluster`. - Then install `mlflow` and `scikit-learn` as task libraries. -- **No schedules.** - The training job has no schedule or trigger. - It runs only when you start it. -- **No secrets.** - The bundle has no tokens, hosts, or IDs. - The CLI gets credentials from your profile. - -## Clean up - -1. Destroy the bundle: - - ```sh - databricks bundle destroy -t dev - ``` - - The CLI deletes the endpoint, the job, the experiment, and the registered model. - The CLI deletes the endpoint before the registered model, because the endpoint refers to the model. - Unity Catalog deletes all model versions together with the registered model. - -2. Drop the inference table, because the bundle does not own it: - - ```sql - DROP TABLE IF EXISTS main.default.dev__iris_classifier_payload; - ``` +To serve a new model version, do steps 2 to 4 again with the new version number. +To roll back, deploy again with the prior version number. +Use `-t prod` for the `prod` target. -The bundle does not delete the catalog, the schema, or the usage data in system tables. +## Notes -In `prod`, the registered model can have versions that other systems use. -Before you destroy `prod`, make sure that no other endpoint or job uses the model. -To keep the model, add `lifecycle: { prevent_destroy: true }` to the registered model. +* `entity_version` takes a version number, not an alias. Keep `model_version` equal to the `champion` version. +* The workspace can reject AI Gateway `rate_limits` on a custom model endpoint. If the deploy fails, remove `rate_limits`. +* In `dev`, the endpoint scales to zero. The first request after an idle period can take minutes. +* `databricks bundle destroy` removes the endpoint, the job, the experiment, and the model with all its versions. Drop the `_payload` inference table manually. From 9c4c30866c9e9fa6b16ee41932a28b8bbd442cf3 Mon Sep 17 00:00:00 2001 From: Andres Garcia Date: Wed, 21 Oct 2026 22:22:09 +0000 Subject: [PATCH 3/6] knowledge_base/model_serving_endpoint: clarify README deploy phases and rate limits Say that the first deploy has two deploy phases (steps 1 and 4), and state the AI Gateway rate limit constraints for custom model endpoints. Docs only. Co-authored-by: Isaac Co-authored-by: omnigent --- knowledge_base/model_serving_endpoint/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/knowledge_base/model_serving_endpoint/README.md b/knowledge_base/model_serving_endpoint/README.md index 4ab5672d..7be52c61 100644 --- a/knowledge_base/model_serving_endpoint/README.md +++ b/knowledge_base/model_serving_endpoint/README.md @@ -16,7 +16,7 @@ For more information, refer to the [Model Serving documentation](https://docs.da ## Usage The endpoint needs a model version, but a new model has no versions. -Thus, the first deploy has two steps. +Thus, the first deploy has two deploy phases: steps 1 and 4. 1. Deploy only the model, the experiment, and the training job: @@ -57,6 +57,6 @@ Use `-t prod` for the `prod` target. ## Notes * `entity_version` takes a version number, not an alias. Keep `model_version` equal to the `champion` version. -* The workspace can reject AI Gateway `rate_limits` on a custom model endpoint. If the deploy fails, remove `rate_limits`. +* AI Gateway rate limits on a custom model endpoint support only queries per minute (QPM). The endpoint must not be route-optimized. If the deploy rejects `rate_limits`, remove them. * In `dev`, the endpoint scales to zero. The first request after an idle period can take minutes. * `databricks bundle destroy` removes the endpoint, the job, the experiment, and the model with all its versions. Drop the `_payload` inference table manually. From ccf804be0cf6e009e735d6fdf7464082f8ded723 Mon Sep 17 00:00:00 2001 From: Andres Garcia Date: Wed, 21 Oct 2026 23:10:48 +0000 Subject: [PATCH 4/6] knowledge_base/model_serving_endpoint: deploy under /Workspace/bundle-examples, trim README Set workspace.root_path to /Workspace/bundle-examples/${bundle.name}/ ${bundle.target} (dev adds ${workspace.current_user.short_name}). Remove the README Notes section and fold the champion-version hint into the deploy step, to match the sibling knowledge_base READMEs. Tested end to end in dev: deploy --select, bundle run train_model, full deploy with model_version=1, endpoint READY, query OK, destroy. Co-authored-by: Isaac Co-authored-by: omnigent --- knowledge_base/model_serving_endpoint/README.md | 10 ++-------- knowledge_base/model_serving_endpoint/databricks.yml | 4 +++- 2 files changed, 5 insertions(+), 9 deletions(-) diff --git a/knowledge_base/model_serving_endpoint/README.md b/knowledge_base/model_serving_endpoint/README.md index 7be52c61..365e041a 100644 --- a/knowledge_base/model_serving_endpoint/README.md +++ b/knowledge_base/model_serving_endpoint/README.md @@ -31,8 +31,9 @@ Thus, the first deploy has two deploy phases: steps 1 and 4. ``` 3. Read the version number in the job output, for example `--var model_version=1`. + The job sets the `champion` alias on this version. -4. Deploy the full bundle with that version: +4. Deploy the full bundle. Set `model_version` to the `champion` version: ```sh databricks bundle deploy --var model_version=1 @@ -53,10 +54,3 @@ Thus, the first deploy has two deploy phases: steps 1 and 4. To serve a new model version, do steps 2 to 4 again with the new version number. To roll back, deploy again with the prior version number. Use `-t prod` for the `prod` target. - -## Notes - -* `entity_version` takes a version number, not an alias. Keep `model_version` equal to the `champion` version. -* AI Gateway rate limits on a custom model endpoint support only queries per minute (QPM). The endpoint must not be route-optimized. If the deploy rejects `rate_limits`, remove them. -* In `dev`, the endpoint scales to zero. The first request after an idle period can take minutes. -* `databricks bundle destroy` removes the endpoint, the job, the experiment, and the model with all its versions. Drop the `_payload` inference table manually. diff --git a/knowledge_base/model_serving_endpoint/databricks.yml b/knowledge_base/model_serving_endpoint/databricks.yml index 47c48b29..fc85f804 100644 --- a/knowledge_base/model_serving_endpoint/databricks.yml +++ b/knowledge_base/model_serving_endpoint/databricks.yml @@ -45,11 +45,13 @@ targets: dev: mode: development default: true + workspace: + root_path: /Workspace/bundle-examples/${bundle.name}/${bundle.target}/${workspace.current_user.short_name} prod: mode: production workspace: - root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target} + root_path: /Workspace/bundle-examples/${bundle.name}/${bundle.target} permissions: - user_name: ${workspace.current_user.userName} level: CAN_MANAGE From b25ab1ae6415fd01fb97b52d2a9818d5516fc84c Mon Sep 17 00:00:00 2001 From: Andres Garcia Date: Wed, 21 Oct 2026 23:13:02 +0000 Subject: [PATCH 5/6] Add cleanup step to model serving README Co-authored-by: omnigent --- knowledge_base/model_serving_endpoint/README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/knowledge_base/model_serving_endpoint/README.md b/knowledge_base/model_serving_endpoint/README.md index 365e041a..e641ec4a 100644 --- a/knowledge_base/model_serving_endpoint/README.md +++ b/knowledge_base/model_serving_endpoint/README.md @@ -51,6 +51,16 @@ Thus, the first deploy has two deploy phases: steps 1 and 4. The response has one class number for each record, for example `{"predictions": [0]}`. +6. Remove the resources. `bundle destroy` keeps the registered model while it has versions, so delete the versions and the model: + + ```sh + databricks bundle destroy + databricks model-versions delete .. + databricks registered-models delete .. + ``` + + In `dev`, `` has the prefix `dev__`. Also drop the `_payload` inference table. + To serve a new model version, do steps 2 to 4 again with the new version number. To roll back, deploy again with the prior version number. Use `-t prod` for the `prod` target. From d3f54e67c57e07ccd0e2c860121d01aba3b32c44 Mon Sep 17 00:00:00 2001 From: Andres Garcia Date: Thu, 22 Oct 2026 02:41:03 +0000 Subject: [PATCH 6/6] Use serverless environment version 6 Move the serverless job environment from version 4 to version 6, the newest GA version. Co-authored-by: Isaac Co-authored-by: omnigent --- .../model_serving_endpoint/resources/train_model.job.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/knowledge_base/model_serving_endpoint/resources/train_model.job.yml b/knowledge_base/model_serving_endpoint/resources/train_model.job.yml index 8745b6eb..6ba2e11e 100644 --- a/knowledge_base/model_serving_endpoint/resources/train_model.job.yml +++ b/knowledge_base/model_serving_endpoint/resources/train_model.job.yml @@ -28,7 +28,7 @@ resources: environments: - environment_key: serverless_env spec: - environment_version: "4" + environment_version: "6" dependencies: - mlflow>=3.1 - scikit-learn