Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 18 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,10 @@ conclave ask "Name three sorting algorithms." -c grok,perplexity --mode raw
conclave ask "Is a service mesh worth it for 8 services?" \
-c grok,gemini,claude --mode debate --rounds 3

# Debate with early-stop: stop before --rounds once answers stop changing
conclave ask "Is a service mesh worth it for 8 services?" \
-c grok,gemini,claude --mode debate --rounds 5 --converge-threshold 0.95

# Adversarial: one model proposes, the rest refute, the synthesizer judges
conclave ask "Defend event sourcing for this ledger." \
-c grok,gemini,perplexity --mode adversarial --proposer grok
Expand All @@ -89,9 +93,11 @@ conclave ask "..." -c grok,perplexity --mode debate --json
```

Mode flags at a glance: `--mode synthesize|raw|debate|adversarial`. `--rounds N`
(default 2) applies to `debate`; `--proposer NAME` (default: first member) applies
to `adversarial`. `--synthesizer/-s` overrides the synthesizer *and* the adversarial
judge.
(default 2) is the *maximum* round count for `debate`; `--converge-threshold FLOAT`
(or `--converge`/`--no-converge`) optionally stops a debate early once answers
stabilize round-over-round (off by default — `--rounds` runs in full). `--proposer
NAME` (default: first member) applies to `adversarial`. `--synthesizer/-s` overrides
the synthesizer *and* the adversarial judge.

`--council` accepts either a comma-separated list of friendly names or the name
of a council defined in your config (see below). The built-in `default` council
Expand Down Expand Up @@ -127,6 +133,10 @@ for rnd in debate.rounds:
print("round", rnd.round_number, [a.name for a in rnd.successful_answers])
print("FINAL:\n", debate.synthesis)

# debate with optional early-stop: stop before `rounds` once answers converge
quick = council.debate_sync("Is P=NP likely false?", rounds=5, converge_threshold=0.95)
print("ran", len(quick.rounds), "rounds; converged:", quick.converged, quick.convergence_score)

# adversarial: propose -> refute -> verdict
adv = council.adversarial_sync("Defend CRDTs for offline-first apps.", proposer="grok")
print("PROPOSAL by", adv.adversarial.proposer, "->", adv.adversarial.proposal.answer)
Expand All @@ -138,7 +148,8 @@ print("VERDICT:\n", adv.adversarial.verdict) # also mirrored to adv.synthesis
`CouncilResult` exposes `mode`, `answers` (per-model `ModelAnswer` with `model_id`,
`latency_s`, `usage`, `error`), `synthesis`, `synthesizer`, `skipped`, plus
`successful_answers` / `failed_answers` helpers. For `debate` it also carries
`rounds` (a list of `DebateRound`, each with per-member `answers`); for
`rounds` (a list of `DebateRound`, each with per-member `answers`) plus
`converged`/`convergence_score` (set when an early-stop fired); for
`adversarial` it carries `adversarial` (an `AdversarialResult` with `proposer`,
`proposal`, `critiques`, `verdict`). For debate the final round is mirrored into
`answers` and the synthesis into `synthesis`; for adversarial the proposal +
Expand Down Expand Up @@ -168,8 +179,9 @@ Repeated or eval runs can be served from an on-disk cache instead of re-calling
the providers. It is **off by default** and **never persists API keys** — the
cache key is a SHA-256 over the normalized prompt, the ordered council members
(friendly name + resolved model id), the mode, the synthesizer/judge identity,
and the mode params (temperature, debate `rounds`, adversarial `proposer`). No
key value or env-var name ever reaches the key or the stored payload.
and the mode params (temperature, debate `rounds` + `converge_threshold`,
adversarial `proposer`). No key value or env-var name ever reaches the key or the
stored payload.

Enable it per run with `--cache` (or disable a config default with `--no-cache`):

Expand Down
9 changes: 9 additions & 0 deletions config.example.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,12 @@ synthesizer: claude
# The cache is keyed on (prompt, council, mode, model ids) and NEVER stores keys.
# Stored under $XDG_CACHE_HOME/conclave (else ~/.cache/conclave).
cache: false

# Optional debate early-stop on convergence (OFF by default, unset). When set to
# a value in [0.0, 1.0], a `--mode debate` run stops before `--rounds` is
# exhausted once answers stabilize round-over-round (a difflib similarity ratio
# averaged across members reaches this threshold). Leave unset for the historic
# fixed-rounds behavior. Override per invocation with `--converge-threshold` or
# `--converge` / `--no-converge`. A high value (e.g. 0.95) only stops on
# near-identical successive answers.
# converge_threshold: 0.95
11 changes: 9 additions & 2 deletions docs/PRODUCT_DESIGN_DOCUMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -345,8 +345,15 @@ Ordered roughly by strategic value to the origin use case and to mcp-warden.

1. **`vote` mode** — structured majority with reported split. Needs a constrained
answer schema so votes are comparable.
2. **Debate convergence/stop criteria** — today debate runs a fixed `--rounds`; add optional
early-stop when answers converge (and a configurable convergence signal).
2. ~~**Debate convergence/stop criteria** — today debate runs a fixed `--rounds`; add optional
early-stop when answers converge (and a configurable convergence signal).~~ **LANDED**
(issue #4): opt-in early-stop via `converge_threshold` (config field, `Council.debate`
param, and `--converge-threshold` / `--converge`/`--no-converge` CLI flags). The signal is
round-over-round answer stability — per-member `difflib.SequenceMatcher` ratio averaged
across members — deterministic, stdlib-only, offline-testable. Off by default (`None`),
preserving fixed-rounds behavior exactly. Recorded on `CouncilResult.converged` +
`convergence_score`; `converge_threshold` is part of the debate cache key so a converged
run and a fixed run never collide. Kept in the list, struck through, for traceability.
3. **More first-class providers** — additional friendly-name defaults (e.g. more OpenAI,
Anthropic, Google, and open-weights endpoints). New OpenAI-compatible vendors are already
config-only via `endpoints:`; this item is about promoting common ones to typed defaults
Expand Down
11 changes: 9 additions & 2 deletions src/conclave/cache.py
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ def make_key(
temperature: float,
rounds: int | None = None,
proposer: str | None = None,
converge_threshold: float | None = None,
) -> str:
"""Build the stable cache key (sha256 hex) for a council run.

Expand All @@ -95,8 +96,8 @@ def make_key(
* ordered ``(friendly_name, resolved_model_id)`` member pairs (order matters --
see module docstring),
* synthesizer/judge friendly name + resolved model id,
* temperature, and mode params (``rounds`` for debate, ``proposer`` for
adversarial) when they apply.
* temperature, and mode params (``rounds`` + ``converge_threshold`` for
debate, ``proposer`` for adversarial) when they apply.

Args:
prompt: The raw user prompt (normalized internally).
Expand All @@ -108,6 +109,9 @@ def make_key(
rounds: Debate round count (included only for ``debate``).
proposer: Adversarial proposer friendly name (included only for
``adversarial``).
converge_threshold: Debate early-stop threshold (included only for
``debate``). A converged run and a fixed-rounds run over otherwise
identical inputs must not collide, so this is part of the key.

Returns:
A 64-char lowercase hex SHA-256 digest. Contains zero key material.
Expand All @@ -124,6 +128,9 @@ def make_key(
}
if mode == "debate":
payload["rounds"] = rounds
# Early-stop changes how many rounds actually run and thus the output;
# it must distinguish a converged run from a fixed-rounds run.
payload["converge_threshold"] = converge_threshold
if mode == "adversarial":
payload["proposer"] = proposer

Expand Down
67 changes: 65 additions & 2 deletions src/conclave/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,12 @@ def _render_debate(result: CouncilResult) -> None:
console.rule(f"[bold]Round {rnd.round_number}[/bold]")
for ans in rnd.answers:
console.print(_answer_panel(ans, border="magenta"))
if result.converged:
score = result.convergence_score if result.convergence_score is not None else 0.0
err_console.print(
f"[green]Converged after {len(result.rounds)} round(s) "
f"(score {score:.3f}); stopped early.[/green]"
)
_print_synthesis(result, title="FINAL SYNTHESIS")


Expand Down Expand Up @@ -134,6 +140,39 @@ def _render_adversarial(result: CouncilResult) -> None:

_VALID_MODES = {"synthesize", "raw", "debate", "adversarial"}

# Threshold used when --converge is passed without an explicit --converge-threshold.
# High by design: an early stop should require answers that are nearly stable
# round-over-round, so the default is conservative and rarely fires spuriously.
_DEFAULT_CONVERGE_THRESHOLD = 0.95


def _resolve_converge_threshold(
converge: bool | None,
converge_threshold: float | None,
config_threshold: float | None,
) -> float | None:
"""Resolve the CLI convergence flags + config into a single threshold.

Returns the threshold to run with, or ``None`` for early-stop off. Precedence,
mirroring how ``--cache/--no-cache`` overrides config per invocation:

* an explicit ``--converge-threshold`` value wins (and implies on);
* else ``--no-converge`` forces off regardless of config;
* else ``--converge`` turns on at the default threshold;
* else (both flags unset) defer to the config value.

Resolution happens here (not in the council) so ``--no-converge`` can force
off even when config enables it; the resulting concrete threshold is then
passed to :meth:`Council.debate_sync`.
"""
if converge_threshold is not None:
return converge_threshold
if converge is False:
return None
if converge is True:
return _DEFAULT_CONVERGE_THRESHOLD
return config_threshold


@app.command()
def ask(
Expand All @@ -154,7 +193,28 @@ def ask(
None, "--synthesizer", "-s", help="Override the synthesizer/judge model name."
),
rounds: int = typer.Option(
2, "--rounds", "-r", help="Number of debate rounds (debate mode only).", min=1
2, "--rounds", "-r", help="Maximum number of debate rounds (debate mode only).", min=1
),
converge: bool | None = typer.Option(
None,
"--converge/--no-converge",
help=(
"Enable/disable debate early-stop on answer convergence (debate mode "
"only; off by default, defers to config when unset). --converge with "
"no --converge-threshold uses a default threshold of "
f"{_DEFAULT_CONVERGE_THRESHOLD}; --no-converge forces it off."
),
),
converge_threshold: float | None = typer.Option(
None,
"--converge-threshold",
help=(
"Debate early-stop threshold in [0.0, 1.0] (debate mode only). When "
"set, the debate stops once round-over-round answer stability reaches "
"this value. Implies --converge. Overrides config; unset defers to it."
),
min=0.0,
max=1.0,
),
proposer: str | None = typer.Option(
None,
Expand Down Expand Up @@ -201,7 +261,10 @@ def ask(

c = Council(models=members, synthesizer=synthesizer, config=cfg, cache=cache)
if mode_lower == "debate":
result = c.debate_sync(prompt, rounds=rounds)
threshold = _resolve_converge_threshold(
converge, converge_threshold, cfg.converge_threshold
)
result = c.debate_sync(prompt, rounds=rounds, converge_threshold=threshold)
elif mode_lower == "adversarial":
result = c.adversarial_sync(prompt, proposer=proposer)
else:
Expand Down
36 changes: 36 additions & 0 deletions src/conclave/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,13 +66,21 @@ class ConclaveConfig(BaseModel):
repeat run is served from the on-disk cache (see :mod:`conclave.cache`)
instead of re-calling the providers. The cache never persists keys.
A ``--cache/--no-cache`` CLI flag overrides this per invocation.
converge_threshold: opt-in debate early-stop threshold (off by default,
``None``). When set to a value in ``[0.0, 1.0]``, a ``debate`` run
stops before ``rounds`` is exhausted once the round-over-round answer
stability (a :mod:`difflib` similarity ratio averaged across members)
reaches the threshold. ``None`` keeps the historic fixed-rounds
behavior exactly. A ``--converge-threshold`` / ``--converge/--no-converge``
CLI flag overrides this per invocation. See :func:`conclave.modes.run_debate`.
"""

models: dict[str, str] = Field(default_factory=dict)
councils: dict[str, list[str]] = Field(default_factory=dict)
synthesizer: str = DEFAULT_SYNTHESIZER
endpoints: dict[str, CustomEndpoint] = Field(default_factory=dict)
cache: bool = False
converge_threshold: float | None = None

def resolve_model_id(self, name: str) -> str:
"""Map a friendly name to a provider-prefixed model id.
Expand Down Expand Up @@ -191,10 +199,38 @@ def _load_config_uncached(path: Path) -> ConclaveConfig:
# Off by default; only an explicit truthy ``cache: true`` in the file enables it.
cache = bool(raw.get("cache", False))

# Off by default (None). A malformed value degrades to off rather than raising,
# keeping config loading resilient like the rest of this module.
converge_threshold = _coerce_threshold(raw.get("converge_threshold"))

return ConclaveConfig(
models=merged_models,
councils=councils,
synthesizer=synthesizer,
endpoints=endpoints,
cache=cache,
converge_threshold=converge_threshold,
)


def _coerce_threshold(value: Any) -> float | None:
"""Coerce a config ``converge_threshold`` value to a float in ``[0.0, 1.0]``.

Returns ``None`` (early-stop off) for an absent, non-numeric, or out-of-range
value, logging a warning in the latter cases. This mirrors the module's
resilient loading: a bad config field never crashes a run, it just disables
the optional feature.
"""
if value is None:
return None
try:
threshold = float(value)
except (TypeError, ValueError):
logger.warning("converge_threshold %r is not a number; disabling early-stop", value)
return None
if not 0.0 <= threshold <= 1.0:
logger.warning(
"converge_threshold %s is outside [0.0, 1.0]; disabling early-stop", threshold
)
return None
return threshold
45 changes: 39 additions & 6 deletions src/conclave/council.py
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ def _cache_key(
*,
rounds: int | None = None,
proposer: str | None = None,
converge_threshold: float | None = None,
) -> str:
"""Build the cache key for a run from the resolved, secret-free identity.

Expand All @@ -125,6 +126,7 @@ def _cache_key(
temperature=self.temperature,
rounds=rounds,
proposer=proposer,
converge_threshold=converge_threshold,
)

async def _cached_run(
Expand All @@ -135,6 +137,7 @@ async def _cached_run(
*,
rounds: int | None = None,
proposer: str | None = None,
converge_threshold: float | None = None,
) -> CouncilResult:
"""Serve ``run`` from the result cache when caching is enabled.

Expand All @@ -146,7 +149,13 @@ async def _cached_run(
if not self.cache_enabled:
return await run()

key = self._cache_key(prompt, mode, rounds=rounds, proposer=proposer)
key = self._cache_key(
prompt,
mode,
rounds=rounds,
proposer=proposer,
converge_threshold=converge_threshold,
)
hit = cache_mod.load(key)
if hit is not None:
logger.info("cache hit for %s run (%s)", mode, key[:12])
Expand Down Expand Up @@ -307,20 +316,39 @@ async def synthesize_blocks(self, system_prompt: str, user_content: str) -> Mode
timeout=self.timeout,
)

async def debate(self, prompt: str, rounds: int = 2) -> CouncilResult:
async def debate(
self, prompt: str, rounds: int = 2, converge_threshold: float | None = None
) -> CouncilResult:
"""Run a multi-round debate. See :func:`conclave.modes.run_debate`.

Round 1 is an independent fan-out; rounds 2..N show each member its
peers' anonymized prior answers; the synthesizer consolidates survivors.
Cache-served when caching is enabled (``rounds`` is part of the key).
Cache-served when caching is enabled (``rounds`` and the resolved
``converge_threshold`` are part of the key).

Args:
prompt: The user prompt.
rounds: Maximum number of rounds (the historic fixed count).
converge_threshold: Opt-in early-stop threshold. ``None`` (default)
defers to ``config.converge_threshold`` (off unless set in
``~/.conclave/config.yml``); an explicit value overrides it for
this call. With early-stop off the debate runs exactly ``rounds``,
identical to the historic behavior. Mirrors the ``cache``
None-defers-to-config convention.
"""
from .modes import run_debate

# Resolve the opt-in here (mirrors ``cache``: explicit arg wins, else
# config) so the cache key reflects what will actually run.
threshold = (
self.config.converge_threshold if converge_threshold is None else converge_threshold
)
return await self._cached_run(
prompt,
"debate",
lambda: run_debate(self, prompt, rounds=rounds),
lambda: run_debate(self, prompt, rounds=rounds, converge_threshold=threshold),
rounds=rounds,
converge_threshold=threshold,
)

async def adversarial(self, prompt: str, proposer: str | None = None) -> CouncilResult:
Expand Down Expand Up @@ -398,9 +426,14 @@ def ask_sync(self, prompt: str, synthesize: bool = True) -> CouncilResult:
"""
return self._run_sync(lambda: self.ask(prompt, synthesize=synthesize), "ask_sync")

def debate_sync(self, prompt: str, rounds: int = 2) -> CouncilResult:
def debate_sync(
self, prompt: str, rounds: int = 2, converge_threshold: float | None = None
) -> CouncilResult:
"""Synchronous wrapper around :meth:`debate`."""
return self._run_sync(lambda: self.debate(prompt, rounds=rounds), "debate_sync")
return self._run_sync(
lambda: self.debate(prompt, rounds=rounds, converge_threshold=converge_threshold),
"debate_sync",
)

def adversarial_sync(self, prompt: str, proposer: str | None = None) -> CouncilResult:
"""Synchronous wrapper around :meth:`adversarial`."""
Expand Down
Loading
Loading