feat(studio): replace the NAT with the Fabric in Create-Example - #1223
Merged
Conversation
Adds the Fabric (nemo-agents-spec-v1) email-phishing agent as a canned example in Studio's Create Example Agent modal, alongside the NAT email-security-analyst. - New static asset public/sample-agents/email-phishing-agent/agent.yml: an independent copy of the plugin example's agent.yaml (source of truth) with a Studio-sample header. Body byte-identical, so the two stay diffable. - New SAMPLE_AGENTS entry with configFormat: nemo-agents-spec-v1, so the create API validates it as Fabric and loadSampleAgentConfig injects the selected model at models.default.model (the existing Fabric branch — no loader change needed). - Reworked the INVARIANT comment: it previously only covered NAT _type -> Python package deps; added the Fabric shape (mcp.servers.<n>.url console script must be on PATH — here email-phishing-iocs) and the hand-sync note for the copies. Verified: loadSampleAgentConfig tests 7/7; the asset parses with config_format, models.default.model, and the console-script MCP url the loader/deploy path need. NOTE: the asset is a hand-maintained copy. #1222 (specialist sub-agents) changes the plugin agent.yaml; this sample will need a one-line re-sync once that lands. Pre-existing (NOT from this change): studio typecheck fails on src/api/evaluation/agent-evaluations.ts (missing AggregateScalarScore in the generated web SDK) — reproduces on pristine main; needs make update-web-sdk. Signed-off-by: Nathan Walston <nwalston@nvidia.com>
Contributor
📝 WalkthroughWalkthroughAdds a Studio Fabric sample agent for email-phishing analysis. The configuration defines DeepAgents delegation, IOC extraction through MCP, model and telemetry settings, and deployment metadata. ChangesEmail phishing agent
Sequence Diagram(s)sequenceDiagram
participant User
participant DeepAgentsOrchestrator
participant PhishingAnalyzer
participant EmailPhishingIOCs
User->>DeepAgentsOrchestrator: Submit complete email
DeepAgentsOrchestrator->>PhishingAnalyzer: Delegate phishing analysis
PhishingAnalyzer->>EmailPhishingIOCs: Extract indicators when needed
EmailPhishingIOCs-->>PhishingAnalyzer: Return extracted IOCs
PhishingAnalyzer-->>DeepAgentsOrchestrator: Return YAML verdict
DeepAgentsOrchestrator-->>User: Return subagent verdict
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
NAT is being deprecated, so remove the NAT email-security-analyst sample from Studio's Create Example Agent dropdown, leaving the Fabric email-phishing agent (added in this PR) as the example. - Remove the email_security_analyst SAMPLE_AGENTS entry + its INVARIANT _type lines. - Delete its static agent.yml. - KEEP the eval-config assets (eval-config.*.json, dataset.jsonl, READMEs): the eval-config picker (EVAL_CONFIG_SAMPLES) is a separate registry and, per its own contract, configs are not owned by an agent, so the picker keeps working. Only agent.yml is removed from that directory. DEFAULT_SAMPLE_AGENT_KEY now resolves to email_phishing_agent (only entry). Scope: Studio dropdown only. The NAT plugin example (plugins/nemo-agents/examples/email-security-analyst/) is intentionally left in place; removing it is a broader NAT-deprecation call for a separate, cross-team ticket. loadSampleAgentConfig tests 7/7. Signed-off-by: Nathan Walston <nwalston@nvidia.com>
walston
changed the title
Contributor
|
15 tasks
marcusds
approved these changes
Aug 11, 2026
marcusds
changed the title
marcusds
enabled auto-merge
August 11, 2026 04:53
15 tasks
marcusds
added a commit
that referenced
this pull request
Aug 11, 2026
Every evaluation Studio submitted targeted the agent proxy's /generate, which
only NAT's FastAPI front end serves. A `nemo-agents-spec-v1` agent is served by
the Platform-owned Fabric server, which exposes /health, /v1/chat/completions
and DELETE /v1/sessions/{id} and nothing else, so the request 404s and the run
fails with nothing to show for it.
That is the format `nemo-build-agent` produces by default, and since #1223 it is
also what Studio's Create Example Agent produces — SAMPLE_AGENTS now holds a
single Fabric entry — so the documented way to build an agent yields one that
cannot be evaluated.
The target now posts OpenAI chat completions for every agent rather than
branching on config_format. Both formats serve that path: Fabric natively, and
NAT through its FastAPI front end, whose default workflow endpoint sets
openai_api_v1_path to /v1/chat/completions. No agent config in this repo
overrides general.front_end, and both launchers (`nat start fastapi` in the
subprocess and container backends) take those defaults. Studio's own chat
playground already relies on this, posting chat completions to every deployment
without regard to format.
Branching on the agent entity would have meant resolving it before submit, and
an unresolved or failed lookup would have to pick a wire format anyway —
defaulting to the one that 404s for the agents this fixes. One shape for both
formats removes the lookup, the fallback, and the race between them.
The generic agent target needed no evaluator or SDK change: it already takes a
URL, a Jinja body template and a JSONPath. render_template recurses through
dicts and lists, so `{{ instruction }}` substitutes inside the nested messages
entry, and _extract_jsonpath returns matches[0].value, so
$.choices[0].message.content resolves to the text.
DatasetEvalRowResultsPanel read the rendered prompt out of the request body at
`input_message`, a key the new body does not have; it would have silently fallen
back to dumping the raw dataset row. It now reads the last chat message and
keeps `input_message` as a fallback so jobs submitted before this still render.
Also drops AgentEvaluationsRoute/components/submitEvaluationSpec.ts, an
unreferenced second copy of the submission logic still building the /generate
target, and updates the route's AGENTS.md, which prescribed /generate as the
eval target.
ASTD-410
Signed-off-by: mschwab <mschwab@nvidia.com>
marcusds
added a commit
that referenced
this pull request
Aug 11, 2026
Every evaluation Studio submitted targeted the agent proxy's /generate, which
only NAT's FastAPI front end serves. A `nemo-agents-spec-v1` agent is served by
the Platform-owned Fabric server, which exposes /health, /v1/chat/completions
and DELETE /v1/sessions/{id} and nothing else, so the request 404s and the run
fails with nothing to show for it.
That is the format `nemo-build-agent` produces by default, and since #1223 it is
also what Studio's Create Example Agent produces — SAMPLE_AGENTS now holds a
single Fabric entry — so the documented way to build an agent yields one that
cannot be evaluated.
The target now posts OpenAI chat completions for every agent rather than
branching on config_format. Both formats serve that path: Fabric natively, and
NAT through its FastAPI front end, whose default workflow endpoint sets
openai_api_v1_path to /v1/chat/completions. No agent config in this repo
overrides general.front_end, and both launchers (`nat start fastapi` in the
subprocess and container backends) take those defaults. Studio's own chat
playground already relies on this, posting chat completions to every deployment
without regard to format.
Branching on the agent entity would have meant resolving it before submit, and
an unresolved or failed lookup would have to pick a wire format anyway —
defaulting to the one that 404s for the agents this fixes. One shape for both
formats removes the lookup, the fallback, and the race between them.
The generic agent target needed no evaluator or SDK change: it already takes a
URL, a Jinja body template and a JSONPath. render_template recurses through
dicts and lists, so `{{ instruction }}` substitutes inside the nested messages
entry, and _extract_jsonpath returns matches[0].value, so
$.choices[0].message.content resolves to the text.
DatasetEvalRowResultsPanel read the rendered prompt out of the request body at
`input_message`, a key the new body does not have; it would have silently fallen
back to dumping the raw dataset row. It now reads the last chat message and
keeps `input_message` as a fallback so jobs submitted before this still render.
Also drops AgentEvaluationsRoute/components/submitEvaluationSpec.ts, an
unreferenced second copy of the submission logic still building the /generate
target, and updates the route's AGENTS.md, which prescribed /generate as the
eval target.
ASTD-410
Signed-off-by: mschwab <mschwab@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Screen.Recording.2026-08-10.at.15.04.20.mov
Summary
Replaces the NAT
email-security-analystsample with the Fabric (nemo-agents-spec-v1) email-phishing agent in Studio's Create Example Agent dropdown. NAT is being deprecated; this makes the modern Fabric example the one users land on. The Fabric example itself shipped via #1117.No new plumbing — the create flow already supports Fabric samples:
loadSampleAgentConfigbranches onconfig_formatand injects the model atmodels.default.model, andCreateExampleAgentModalenumeratesSAMPLE_AGENTSand forwardsconfig_format.Changes
Add the Fabric example:
public/sample-agents/email-phishing-agent/agent.yml— static asset, an independent (byte-identical) copy of the plugin example'sagent.yamlwith a Studio-sample header.SAMPLE_AGENTSentry (email_phishing_agent,configFormat: 'nemo-agents-spec-v1').email-phishing-iocson PATH).Remove the NAT example (dropdown only):
email_security_analystSAMPLE_AGENTSentry + itsagent.yml.DEFAULT_SAMPLE_AGENT_KEYnow resolves toemail_phishing_agent.eval-config.*.json,dataset.jsonl, READMEs):EVAL_CONFIG_SAMPLESis a separate registry and, per its own contract, configs aren't owned by an agent — so the eval-config picker keeps working unchanged. Onlyagent.ymlwas removed from that directory.Scope
Studio dropdown only. The NAT plugin example
plugins/nemo-agents/examples/email-security-analyst/(Octavian's code) is intentionally left in place — removing it is a broader NAT-deprecation decision (there are other NAT examples too) and warrants its own cross-team ticket, not a silent fold into this one.Verification
loadSampleAgentConfigtests 7/7; dropdown now lists only the Fabric agent; eval-config picker assets all still resolve on disk.Notes for review
agent.ymlasset duplicates the pluginagent.yaml. feat(nemo-agents): add specialist subagents + rename to Email Security Triage #1222 (specialist sub-agents) changes that file, so this sample needs a one-line re-sync once feat(nemo-agents): add specialist subagents + rename to Email Security Triage #1222 merges.src/api/evaluation/agent-evaluations.ts(AggregateScalarScoremissing from the generated evaluator web SDK) fails on pristinemaintoo — evaluator web-SDK drift needingmake update-web-sdk.Tickets
Resolves ASTD-407 (expanded from "add" to "replace" per the NAT-deprecation direction).