Skip to content

feat: add llm:chat-all for batched per-agent calls - #83

Merged
JNK234 merged 1 commit into
mainfrom
feat/chat-all-53
Sep 24, 2026
Merged

JNK234 merged 1 commit into
mainfrom
feat/chat-all-53

Conversation

@JNK234

@JNK234 JNK234 commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator

Closes #53.

What it does

Every agent in a set asks the LLM its own question at the same time, and the model waits once for all the replies. ask turtles [ llm:chat ... ] waits for each turtle in turn; llm:chat-all takes about as long as the slowest single reply.

let results llm:chat-all turtles [ -> (word "You have " energy " energy. Graze or rest?") ]
;=> [[(turtle 0) "Rest." true] [(turtle 1) "Graze." true] ...]

let decisions (llm:chat-all turtles [ -> (word "Size " size ". A predator appears.") ] schema)
;=> [[(turtle 0) [["action" "hide"] ["confidence" 0.8]] true] ...]
  • The prompt reporter runs as each agent, like ask, so it reads that agent's own variables.
  • Each entry is [agent reply ok?]. A failed call only marks its own entry false, carrying the error message; the rest of the batch still completes.
  • The optional schema reuses the structured-output path (llm:chat-with-schema), so each reply arrives as NetLogo values.
  • Built on what is already there: each agent's profile, the provider throttle, 429 retry, per-agent history and token usage all apply, with no new configuration.

Design notes

  • Prompts first. Every prompt is built before any request goes out, so an error in the prompt reporter costs no API calls.
  • Waiting. The batch keeps waiting while replies keep arriving, so later agents queued behind a throttle cap still finish. It gives up only after a full timeout budget with no reply at all, so a hung provider costs one budget, not one per agent.
  • History safety. A reply that arrives after the batch gave up on it, or after the wait was halted, never writes history.
  • Refactor. Schema validation moved into a shared parseObjectSchema helper; llm:chat-with-schema's error messages are unchanged.

Verification

  • sbt test: 268/268, including 14 new tests.txt cases. They cover triples, prompts running as each agent, per-agent history and usage, profiles, failure isolation, schema parsing and bad schemas, non-string prompts, an empty agentset, concurrency, a hung batch costing one budget, progress extending the wait, and rejecting extra schema arguments.
  • Live end-to-end harness against Groq (openai/gpt-oss-20b): 48/48, including the new T11 (6 turtles through a throttle cap of 3, plus a schema batch).
  • Live timing with 8 turtles: 4.83 s with ask turtles [ llm:chat ] versus 1.22 s with llm:chat-all.
  • The throttle is not asserted by a dedicated Scala spec. chat-all calls the same provider method that ThrottleIntegrationSpec covers, and T11 exercises a cap live.

Docs

docs/API-REFERENCE.md: a quick-reference row, a ### llm:chat-all section, and a "Batched Calls" usage pattern.

One request per agent, sent together, one wait. A model that runs
`ask turtles [ llm:chat ... ]` waits for each turtle in turn; chat-all
takes about as long as the slowest single reply.

- `llm:chat-all agentset [ -> prompt ]` reports [agent reply ok?] triples;
  the prompt reporter runs as each agent, like ask
- optional schema argument reuses the structured-output path and returns
  parsed [key value] lists per agent
- a failed call is reported in that agent's triple and never aborts the
  batch; no history is committed for it
- every prompt is built before any request is sent, so a prompt error
  costs no requests
- requests go through each agent's provider, so throttling, retry,
  profiles, history and usage all apply per agent
- the batch waits while replies keep arriving and gives up only after a
  full budget with no reply, so a hung provider costs one budget, not
  one per agent; late replies and halted waits never write history

Closes #53
@JNK234
JNK234 merged commit ac1a2d7 into main Sep 24, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: batched tick-level inference (llm:chat-all)

1 participant