Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions demos/e2e-tests/e2e-tests.nlogox
Original file line number Diff line number Diff line change
Expand Up @@ -345,6 +345,73 @@ to test-profiles
report-totals
end

;; ---------------------------------------------------------------------------
;; T11 — batched calls (#53)
;;
;; llm:chat-all sends one request per turtle at once and waits once. Run
;; through a throttled copy of the config (cap 3) so the batch also proves it
;; composes with the throttle. Every turtle must get its own reply, its own
;; history and its own usage; a schema batch must come back as parsed lists.
;; ---------------------------------------------------------------------------

to test-chat-all
log-line "T11 batched calls"
let throttled (word "config-" provider-under-test "-batch.txt")
write-throttled-config active-config throttled 3 0
llm:load-config throttled

crt 6 [ set label "batch-turtle" ]
let batch turtles with [label = "batch-turtle"]
let results []
reset-timer
carefully
[ set results llm:chat-all batch [ ->
(word "Reply with exactly the number " who) ] ]
[ log-line (word " chat-all failed: " error-message) ]
let elapsed timer
log-triples results
log-line (word " 6 calls, cap 3 -> " precision elapsed 2 "s")

assert "one triple per turtle" (length results = 6)
assert "every call succeeded" (all-ok? results)
assert "each reply names its own turtle"
(all-ok? results and reduce and map [ r ->
member? (word [who] of item 0 r) item 1 r ] results)
assert "each turtle has its own history" (all? batch [ length llm:history = 2 ])
assert "each turtle was credited one call"
(all? batch [ llm:get llm:usage "calls" = 1 ])

let schema (word "{\"type\":\"object\",\"properties\":"
"{\"n\":{\"type\":\"integer\"}},"
"\"required\":[\"n\"],\"additionalProperties\":false}")
let pair (turtle-set min-one-of batch [who] max-one-of batch [who])
let parsed []
carefully
[ set parsed (llm:chat-all pair [ -> (word "Return n = " who) ] schema) ]
[ log-line (word " schema chat-all failed: " error-message) ]
log-triples parsed
assert "schema batch succeeded" (length parsed = 2 and all-ok? parsed)
assert "schema replies are parsed per turtle"
(all-ok? parsed and reduce and map [ r ->
llm:get item 1 r "n" = [who] of item 0 r ] parsed)

ask batch [ die ]
llm:load-config active-config
carefully [ file-delete throttled ] [ ]
report-totals
end

;; True when a llm:chat-all result is non-empty and every call succeeded.
to-report all-ok? [results]
report not empty? results and reduce and map [ r -> item 2 r ] results
end

;; Log each [agent reply ok?] triple on its own line.
to log-triples [results]
foreach results [ r ->
log-line (word " " item 0 r " ok? " item 2 r " -> " item 1 r) ]
end

;; A second model on the provider under test, so a profile can be told apart
;; from the default. Ollama has no guaranteed second model pulled, so it
;; reuses the active one and the test still proves routing and bookkeeping.
Expand Down Expand Up @@ -391,6 +458,7 @@ to test-all
test-throttling
test-usage
test-profiles
test-chat-all
output-print ""
log-line (word "TOTAL passed " pass-count " failed " fail-count)
end
Expand Down
81 changes: 81 additions & 0 deletions docs/API-REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ The NetLogo Multi-LLM Extension provides a unified interface for multiple Large
| ------------------------------------ | ------------- | ---------------------------------------------------------- |
| `llm:chat text` | Chat | Send synchronous chat message, returns response |
| `llm:chat-async text` | Chat | Send asynchronous chat message, returns awaitable reporter |
| `llm:chat-all agents [ -> prompt ]` | Chat | One request per agent, sent together; returns `[agent reply ok?]` triples |
| `llm:chat-with-template file vars` | Chat | Send templated prompt with variable substitution |
| `llm:chat-with-thinking text` | Chat | Returns `[answer thinking]` for reasoning-capable models |
| `llm:choose prompt choices` | Chat | Force selection from provided options |
Expand Down Expand Up @@ -215,6 +216,73 @@ print response
- Use `runresult` to retrieve the actual response
- Still maintains conversation history per agent

### llm:chat-all

**Syntax**: `llm:chat-all agentset prompt-reporter` or `(llm:chat-all agentset prompt-reporter schema)`

**Description**: Sends one chat request per agent, all at once, and waits once for every
reply. A model that does `ask turtles [ llm:chat ... ]` waits for each call in turn — 100
turtles at 2 seconds each is over 3 minutes per tick. `llm:chat-all` takes about as long as
the slowest single call.

**Parameters**:

- `agentset` (agentset): The agents that each send one request
- `prompt-reporter` (anonymous reporter): Run as each agent, like `ask`, to build that
agent's prompt. It can read the agent's own variables, and `myself` is the caller.
- `schema` (string, optional): A JSON Schema, as in `llm:chat-with-schema`. Each reply is then
a parsed `[key value]` list instead of a string.

**Returns**: A list of `[agent reply ok?]` triples, one per agent.

- `ok?` is `true` when the call succeeded; `reply` is the response (or the parsed list).
- `ok?` is `false` when that agent's call failed; `reply` is the error message. The other
agents are unaffected — one failure never aborts the batch.

**Example**:

```netlogo
turtles-own [ energy decision ]

to go
let results llm:chat-all turtles [ -> (word "You have " energy " energy. Move or rest?") ]
foreach results [ r ->
let who-asked item 0 r
ifelse item 2 r
[ ask who-asked [ set decision item 1 r ] ]
[ ask who-asked [ set decision "rest" ] ] ; fallback when the call failed
]
tick
end
```

With a schema, each reply arrives as NetLogo values:

```netlogo
let schema "{\"type\":\"object\",\"properties\":{\"action\":{\"type\":\"string\",\"enum\":[\"move\",\"rest\"]}},\"required\":[\"action\"]}"
let results (llm:chat-all turtles [ -> (word "Energy: " energy) ] schema)
foreach results [ r ->
if item 2 r [ ask item 0 r [ set decision llm:get item 1 r "action" ] ]
]
```

**Notes**:

- Every prompt is built before any request is sent. A prompt reporter that errors or does not
report a string stops the primitive with an error and costs no requests.
- Requests go through each agent's own provider, so everything that applies to `llm:chat`
applies per agent: its profile (`llm:use-profile`), the provider's throttle
(`max_concurrent_requests`, `min_request_interval_ms`), 429 retry, history and usage.
- With throttling on, "send all at once" means "queue all, at most N in flight". Set
`max_concurrent_requests` when running large agentsets on a rate-limited plan.
- Each agent's history gets its own exchange only when its call succeeds.
- The batch keeps waiting as long as replies keep arriving, so a throttled batch whose later
agents queue behind the cap still completes. An agent is reported as failed only when no reply
from any agent has arrived for its whole timeout budget (see Timeout Tuning), so a hung provider
costs one budget, not one per agent. A reply that arrives after that is not added to history.
- The order of the list follows the agentset; read the agent from `item 0` rather than relying
on position.

### llm:choose

**Syntax**: `llm:choose prompt choices`
Expand Down Expand Up @@ -1195,6 +1263,19 @@ to check-responses
end
```

### Batched Calls

For one call per agent per tick, prefer `llm:chat-all` over `llm:chat-async` + `runresult`: it
sends every request at once, waits once, and hands back each agent's reply with a success flag.
Use `llm:chat-async` when the model should keep ticking while a reply is pending.

```netlogo
let results llm:chat-all turtles [ -> (word "Neighbors: " count turtles-on neighbors) ]
foreach filter [ r -> item 2 r ] results [ r ->
ask item 0 r [ set label item 1 r ]
]
```

### Decision Making

```netlogo
Expand Down
Loading
Loading