mirror of
https://github.com/NCBM/plyngent.git
synced 2026-11-07 07:17:54 +08:00
Compare commits
7 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 2054423ebb | |||
| 3c8ca5fcde | |||
| 3cd95504c6 | |||
| 3c555278d5 | |||
| f39ec5fb1a | |||
| 06a5bafab7 | |||
| 9156df2b02 |
@@ -78,13 +78,13 @@ Async SQLAlchemy + aiosqlite. `MemoryStore`: schema init (+ lightweight SQLite `
|
||||
|
||||
- **Kind-based dispatch**: clients expose `kind` (`chat_completions` | `responses` | `messages`). Agent history stays chat-completions-shaped; transport adapters return synthetic `ChatCompletionResponse` / stream chunks.
|
||||
- **`responses_bridge` / `responses_dispatch`**: chat ↔ OpenAI Responses; stream text/reasoning deltas (`response.reasoning_text.done` repeats what its fragments spelled out, so the loop streams each part once and still keeps the whole text in the round's message); tool calls + usage on `response.completed`; a request carrying `tools` returns every round's reasoning as a `reasoning` item of `reasoning_text` parts — "必须完整回传 `reasoning_content` … 即使该轮模型未实际进行工具调用" (a missing item and an empty part are both a 400, so a round that kept none sends a blank line) — and a round's `function_call` items stay contiguous ahead of their outputs (`call, output, call, output` was a 400: the second call reads as a round of its own without reasoning); reasoning is read back from the item's `content` parts.
|
||||
- **`messages_bridge` / `messages_dispatch`**: chat ↔ Anthropic Messages; system fold; tool_use/tool_result; stream text + tool argument fragments; `thinking` from the request's `reasoning_effort` (level → token budget, floor 1024, `max_tokens` grown when the budget would leave no room) plus the client's configured `thinking_budget_tokens`; a server tool (`server_tool_use` + its `web_search_tool_result`) is the surface's own business — it streams `input_json_delta` fragments like a call but nothing local asked for it, so only a `tool_use` block becomes a tool call and nothing is executed; `thinking` and the two search blocks are kept as `AssistantChatMessage.provider_blocks` (the thinking text also lands in `reasoning_content`, so it streams as reasoning) and go back with the round, which is also why a round that kept only those blocks is not an empty completion; a chat-completions request drops the field (`compat.coerce_chat_messages`).
|
||||
- **`messages_bridge` / `messages_dispatch`**: chat ↔ Anthropic Messages; system fold; tool_use/tool_result; stream text + tool argument fragments; `thinking` from the request's `reasoning_effort` (level → token budget, floor 1024, `max_tokens` grown when the budget would leave no room) plus the client's configured `thinking_budget_tokens`; a thinking block the model refuses (from Claude 4.7 on, `thinking.type.enabled` is a 400 asking for adaptive thinking + `output_config.effort`; a model whose thinking is always on refuses `thinking.type.disabled`) gets one retry with the shape it takes — adaptive thinking with the level as the effort, or no block at all — and the refusal is remembered per model on the client, so only its first request pays the round trip (only a refusal that arrives before the stream yields anything is retried; a later failure has already been seen); a server tool (`server_tool_use` + its `web_search_tool_result`) is the surface's own business — it streams `input_json_delta` fragments like a call but nothing local asked for it, so only a `tool_use` block becomes a tool call and nothing is executed; `thinking` and the two search blocks are kept as `AssistantChatMessage.provider_blocks` (the thinking text also lands in `reasoning_content`, so it streams as reasoning) and go back with the round, which is also why a round that kept only those blocks is not an empty completion; a chat-completions request drops the field (`compat.coerce_chat_messages`).
|
||||
- **Provider-side tools**: `provider_tools` on the OpenAI **and** DeepSeek presets (list of dicts; `[{type="web_search"}]` when omitted; `[]` disables) threaded via `ChatAgent`/`run_chat_loop` into Responses and Messages; never executed by `ToolRegistry`. The Messages surface declares its own server tool instead — a bare `web_search` becomes `{type="web_search_20250305", name="web_search"}` (the search then runs inside the response, and its `encrypted_content` results go back verbatim with the round — `AssistantChatMessage.provider_blocks`) — and a DeepSeek provider on a surface that ignores hosted tools (`config.routing.local_web_search`) gets the local `web_search` tool from `tools/net/search.py` instead.
|
||||
- **`@tool` / `ToolRegistry`**: decorator infers JSON Schema from type hints; execute tools by name. Before running, `execute` answers a call that omits a required argument with the accepted names (`_missing_required_error`) — the handler's bare `TypeError` never reaches the model, and no confirm prompt fires for a call that cannot run. Shape mistakes inside a *present* argument are the tool's own job (see the `chat` bullet). `@tool(writes=("path",))` names a handler's file arguments: one batch applies calls that name the same file one at a time, in call order — the lock is taken before the confirm prompt, so a pending prompt cannot be overtaken — while different files keep running in parallel.
|
||||
- **`run_chat_loop`**: multi-round tool loop; default **streaming** text deltas + stream tool-call merge; parallel tools; tool-result char budget; soft context compact on request (**API-calibrated** after first usage when available); cooperative cancel points; optional `on_limit`; resolved `reasoning_effort` on every request (see Config).
|
||||
- **`ChatAgent`**: optional `MemoryStore` (user message persisted immediately; **completed tool batches checkpointed** mid-turn; unfinished assistant suffix rolled back on failure); `stream`; system prompt; `provider_tools`; `retry()` continues incomplete turns (user-only **or** after committed tools — does not re-run those tools).
|
||||
- **`run_chat_loop`**: multi-round tool loop; default **streaming** text deltas + stream tool-call merge; parallel tools; tool-result char budget; soft context compact on request (**API-calibrated** after first usage when available); cooperative cancel points; optional `on_limit`; resolved `reasoning_effort` on every request (see Config). A round the provider cut off at the output limit is kept, not failed: the answer and its thinking blocks stay in `messages`, the round's tool calls are dropped (a half-written call must not run — the model re-issues it), and up to `max_truncation_continues` (2) further rounds carry on from a truncation notice; 0 spends the turn as unfinished instead, for `/retry`.
|
||||
- **`ChatAgent`**: optional `MemoryStore` (user message persisted immediately; **completed tool batches checkpointed** mid-turn; unfinished assistant suffix rolled back on failure); `stream`; system prompt; `provider_tools`; `retry()` continues incomplete turns (user-only, after committed tools — does not re-run those tools — or after a kept truncated answer) and `pending_retry_text` reads a truncation notice as unfinished work.
|
||||
- **`/compact`**: soft-compact tool dumps → model summary (no tools) via the same kind dispatch → **new** session seeded with summary message. Soft-compact keeps the last **12** tool results full-size by default (`DEFAULT_RECENT_TOOL_RESULTS`); older tool payloads shrink first.
|
||||
- Events: text_delta, **reasoning_delta**, assistant_message, tool_call/result, max_rounds, **error** (`retryable`/`source`), **cancelled** (`reason`), **usage** (`TokenUsage`).
|
||||
- Events: text_delta, **reasoning_delta**, assistant_message, tool_call/result, max_rounds, **truncated** (`kept_chars`/`continued`), **error** (`retryable`/`source`), **cancelled** (`reason`), **usage** (`TokenUsage`).
|
||||
- Usage: API `usage` from completions (stream with `include_usage`); **char≈token fallback** (~4 chars/token) when omitted; **context size** = last request ``prompt_tokens`` (API preferred); `last_turn_usage` / `session_usage` are **billed sums** (tool rounds re-send history); CLI end-of-turn + `/status`.
|
||||
- Config ``[agent]``: `system_prompt` (persona; default `DEFAULT_SYSTEM_PROMPT`; `""` omits persona), `tool_directives` (tool playbook; default `DEFAULT_TOOL_DIRECTIVES`; `""` omits playbook; both empty → no system), `mcp_instructions` (fold connected MCP servers' initialize ``instructions`` into the system prompt when tools on; default true), `allow_paths` (static out-of-workspace pre-allow `path → read|write|exec`; never prompts), `auto_continue_limits` (auto-raise tool/PTY limits without prompting; default false), `reasoning_effort` / `thinking_budget_tokens` (thinking strength; see Config), `max_tool_result_chars`, `parallel_tools`, `confirm_destructive`, `path_denylist`, `max_context_tokens` (default 200k est. tokens). Truncated tool results carry a model hint (`[Truncated (N chars max; M omitted). You may ask for user to increase the limit via configuration file.]`) so the model can ask the user to raise `max_tool_result_chars`; request-time compact shrinks (`hint=False`) keep the terse `...[truncated N characters]` marker.
|
||||
|
||||
@@ -121,7 +121,7 @@ Click app + readline REPL. Entry: `plyngent` / `python -m plyngent`.
|
||||
- `/btw` side questions: forked session; main transcript/DB untouched. `--tools=read` (default) exposes only `READ_ONLY` tools under `read_only_context()` (fetch=GET) with the main session view; `--tools=no` off; `--tools=full` whole-registry clone with fresh session. Tools off: default falls back to no tools; explicit `read`/`full` errors. The model gets an `aside` notice as its last message before the question (unsaved exchange + tool scope), shown to the user as `[notice] aside: …`.
|
||||
- Explicit `/resume` or `--session` from another workspace prompts: **keep** / **update** / **abort**. A session resumed by a **freshly started process** (`--session` / default latest) also gets a `resume` notice about what a restart drops (PTY sessions, running commands, temp workspaces) — resuming inside a live process pushes nothing.
|
||||
- Host notices (`agent/notices.py`): system-level messages appended as trailing `developer` rows, never system-prompt edits — `Notice.to_message()` for the model, `.summary` for the user via `cli.display.echo_notice` (`--quiet` hides the user line only). Durable hosts persist them (`ChatAgent.push_notice`); a side agent without memory keeps its notice local. `has_notice()` dedupes a session-scoped notice: the `skills` one is pushed once per session (feature enabled + tools on), lists nothing, and makes the model ask with `skill_list` / `skill_search`. Trailing developer messages never end a turn, so `/retry` still sees an interrupted turn behind a notice or checkpoint.
|
||||
- Failed/cancelled turns: user kept; **committed tool rounds kept**, and a batch cut short keeps the calls that already produced a result (`rollback_tail`; written to the store, so a resume continues from them instead of re-running them) — side effects are not re-run on `/retry`, and only unfinished assistant text is dropped; Ctrl+C cancels the in-flight turn (a session-scoped asyncio SIGINT handler routes it, `cli/interrupt.py` — a stray Ctrl+C while the loop waits never tears the REPL down); TTY confirms/ask prompts run off-loop and register their own SIGINT target, so Ctrl+C there cancels just the poll-based read (`wait` reports `cancelled by user`) or is ignored with a hint for readline prompts (`ask_user_*`, confirms) — never the turn, never the loop; auto-retry waits 5s/10s/15s/20s then +10s (10 attempts), its countdown is Ctrl+C-cancellable, and the budget **resets when an attempt completes a model round** (connection recovered); then `/retry`.
|
||||
- Failed/cancelled turns: user kept; **committed tool rounds kept**, and a batch cut short keeps the calls that already produced a result (`rollback_tail`; written to the store, so a resume continues from them instead of re-running them) — side effects are not re-run on `/retry`, and only unfinished assistant text is dropped; a round cut off at the output limit is not a failure at all: it is kept under a `truncated` notice (see the `run_chat_loop` bullet), which also keeps the turn unfinished for `/retry`; Ctrl+C cancels the in-flight turn (a session-scoped asyncio SIGINT handler routes it, `cli/interrupt.py` — a stray Ctrl+C while the loop waits never tears the REPL down); TTY confirms/ask prompts run off-loop and register their own SIGINT target, so Ctrl+C there cancels just the poll-based read (`wait` reports `cancelled by user`) or is ignored with a hint for readline prompts (`ask_user_*`, confirms) — never the turn, never the loop; auto-retry waits 5s/10s/15s/20s then +10s (10 attempts), its countdown is Ctrl+C-cancellable, and the budget **resets when an attempt completes a model round** (connection recovered); then `/retry`.
|
||||
- Pretty tool lines: every builtin tool prints its `* Verb 'target' ` prefix as the call starts and appends the outcome (`L1-4 (done)`, `(exit code 0)`, `(session 3)`) when the result lands, so a blocking tool stays visible while it runs; namespaced MCP tools share one generic line (`* MCP <server>:<tool> (done)`). Plugin tools and custom tool calls keep the `[tool] …` / `[tool ok] …` whole-line style. Interactive TTY only (`interactive_terminal()`): piped/plain output and `--verbose` keep whole lines. A second call of the same round holds its prefix back until the open row takes its own detail (results arrive in call order), so each call still prints one finished row; a prompt box (or other output) printed while the tool runs *ends* the open prefix as its own line and falls back to whole lines for the rest of the batch — the cursor is never moved, so a prefix wider than the terminal cannot be half-erased. A batch (one round's calls) that only touches todo tools prints one `* Todo x<N>: Push, Update` line plus its last result, whose text already carries the final stack, instead of one header and a full list render per call. `run_argv_batch` lists its commands above its row: one `$ command` line per step, a `pipe_out` chain joined into one line with ` | `, each line elided past 80 characters.
|
||||
- **`plyngent providers`**, **`config path|edit`**. Config open: `$VISUAL`/`$EDITOR` (wait), else system default (`xdg-open` / `open` / `startfile`, non-blocking). `/edit` stays blocking-only. No providers → optional edit then reload when waited.
|
||||
- Tools default on; workspace defaults to cwd; `--max-rounds` default 32. Readline history under platformdirs (`repl_history`). PTY: `close_all` on chat exit.
|
||||
|
||||
@@ -175,7 +175,7 @@ max_context_tokens = 200000
|
||||
|
||||
Per-provider **`timeout`** is passed to the HTTP session for chat/completions, Responses, and `GET /models`. A single number sets one timeout; `{ connect, read }` splits TCP/TLS setup vs idle wait between response bytes (SSE can run longer than `read` while chunks keep arriving). Tool/process timeouts (`run_argv`, PTY, policy confirm) are separate.
|
||||
|
||||
**Thinking strength** (`reasoning_effort`) can be set on `[agent]`, on a provider, or on a single model: the most specific layer wins, per field, so a model may override only the effort or only the budget. `""` sends nothing and lets the provider apply its own default. Each API surface maps the value onto its own wire field — chat completions send `reasoning_effort`, Responses send `reasoning.effort`, and Anthropic has no effort level at all, so it sends `thinking` with the token budget that level maps onto (`minimal` 1024 … `xhigh`/`max` 32768; `none` disables thinking). `max` is DeepSeek's own top level on chat completions and clamps to `xhigh` everywhere else. `thinking_budget_tokens` overrides that mapping; either way the request stays valid: the budget never drops below 1024, and when it would swallow the output limit `max_tokens` grows so the answer still has room. DeepSeek chat keeps `thinking.type = "enabled"` unless the effort is `none`.
|
||||
**Thinking strength** (`reasoning_effort`) can be set on `[agent]`, on a provider, or on a single model: the most specific layer wins, per field, so a model may override only the effort or only the budget. `""` sends nothing and lets the provider apply its own default. Each API surface maps the value onto its own wire field — chat completions send `reasoning_effort`, Responses send `reasoning.effort`, and Anthropic has no effort level at all, so it sends `thinking` with the token budget that level maps onto (`minimal` 1024 … `xhigh`/`max` 32768; `none` disables thinking); a thinking block the model refuses is retried once with the shape it takes — adaptive thinking with the level itself as `output_config.effort` (from Claude 4.7 on), or no block at all where thinking cannot be turned off — and the refusal is remembered for the session, so `thinking_budget_tokens` is ignored on such a model. `max` is DeepSeek's own top level on chat completions and clamps to `xhigh` everywhere else. `thinking_budget_tokens` overrides that mapping; either way the request stays valid: the budget never drops below 1024, and when it would swallow the output limit `max_tokens` grows so the answer still has room. DeepSeek chat keeps `thinking.type = "enabled"` unless the effort is `none`.
|
||||
|
||||
```toml
|
||||
[providers.deepseek]
|
||||
@@ -294,7 +294,7 @@ Type `/help` in the REPL for the live list. Common ones:
|
||||
| `/stream` `/verbose` `/markdown` `/tools` `/rounds` | Toggles and limits |
|
||||
| `/yolo [on\|off\|once]` | Soft destructive confirms: sticky skip, off, or next turn only |
|
||||
|
||||
| `/retry` | Re-run incomplete last user turn (after error/cancel) |
|
||||
| `/retry` | Retry/continue the incomplete last user turn (after error, cancel, or output limit) |
|
||||
| `/btw [--tools read\|no\|full] [--fresh]` | Side question without changing the main session (read-only tools by default; the model is told the exchange is unsaved and how its tools are scoped) |
|
||||
| `/provider` `/model` | Switch without restarting |
|
||||
| `/model --persist` | Save current model id into `plyngent.toml` catalog |
|
||||
@@ -306,7 +306,7 @@ Type `/help` in the REPL for the live list. Common ones:
|
||||
| `/config` | Edit `plyngent.toml` ($VISUAL/$EDITOR or system open); reload after blocking editor |
|
||||
| `/quit` | Leave the REPL |
|
||||
|
||||
User messages are saved immediately. On API error or Ctrl+C, unfinished assistant text is discarded — tool calls that already produced a result stay, so `/retry` does not run them again — but the user message stays so `/retry` works after resume. Interactive auto-retry waits 5s, 10s, 15s, 20s, then +10s each step (10 attempts); an attempt that gets a full model round back **resets that budget**, so a connection that recovers and drops again starts over from 5s. Tool calls of one round run in parallel, but writes to the same file apply one at a time, in call order — a later edit sees the earlier one's result instead of stale text.
|
||||
User messages are saved immediately. On API error or Ctrl+C, unfinished assistant text is discarded — tool calls that already produced a result stay, so `/retry` does not run them again — but the user message stays so `/retry` works after resume. Interactive auto-retry waits 5s, 10s, 15s, 20s, then +10s each step (10 attempts); an attempt that gets a full model round back **resets that budget**, so a connection that recovers and drops again starts over from 5s. A round cut off at the output limit (`max_tokens`) is not an error: what the model had written — thinking blocks included — stays in the transcript, its tool calls are dropped (they are re-issued, not replayed), and a notice tells it where it stopped, so up to 2 further rounds carry on from there; when that budget is spent the turn ends with the partial answer kept and `/retry` continues it. Tool calls of one round run in parallel, but writes to the same file apply one at a time, in call order — a later edit sees the earlier one's result instead of stale text.
|
||||
|
||||
Ctrl+C cancels the in-flight turn, and during the auto-retry countdown it cancels the retry. Model-initiated prompts run off the event loop: a `wait` prompt is cancelled by Ctrl+C (the tool reports `cancelled by user` and the turn continues), while `ask_user_*` and confirm prompts ignore it with a one-line hint — answer them to continue. Ctrl+C never exits the REPL (use Ctrl+D or `/quit`), and MCP servers run in their own process group, so it never kills them either.
|
||||
|
||||
|
||||
@@ -105,7 +105,8 @@ max_context_tokens = 200000
|
||||
# reasoning_effort = "medium"
|
||||
# Explicit Anthropic thinking budget in tokens, overriding that mapping; the
|
||||
# request stays valid either way (floor 1024, and ``max_tokens`` grows when the
|
||||
# budget would leave no room for the answer).
|
||||
# budget would leave no room for the answer). A model that only takes adaptive
|
||||
# thinking ignores it (see the anthropic provider below).
|
||||
# thinking_budget_tokens = 8000
|
||||
|
||||
# Optional: custom compact/summarisation prompts (empty = built-in defaults).
|
||||
@@ -160,7 +161,10 @@ access_key_or_token = "sk-replace-me"
|
||||
# # url defaults to https://api.anthropic.com/v1 when empty
|
||||
# timeout = { connect = 10, read = 600 }
|
||||
# # Anthropic has no effort level: an effort becomes a thinking token budget
|
||||
# # (see the [agent] section above). Set the budget explicitly to pin it:
|
||||
# # (see the [agent] section above). A model that no longer takes one (from
|
||||
# # Claude 4.7 on) is retried with adaptive thinking, whose depth is the effort
|
||||
# # level itself, and a model that cannot stop thinking gets no thinking block
|
||||
# # at all. Set the budget explicitly to pin it:
|
||||
# reasoning_effort = "high"
|
||||
# thinking_budget_tokens = 16000
|
||||
#
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "plyngent"
|
||||
version = "0.9.0"
|
||||
version = "0.9.1"
|
||||
description = "LLM chat and agent toolkit"
|
||||
authors = [
|
||||
{ name = "HivertMoZara", email = "worldmozara@163.com" },
|
||||
|
||||
@@ -17,6 +17,7 @@ from .events import ReasoningDeltaEvent as ReasoningDeltaEvent
|
||||
from .events import TextDeltaEvent as TextDeltaEvent
|
||||
from .events import ToolCallEvent as ToolCallEvent
|
||||
from .events import ToolResultEvent as ToolResultEvent
|
||||
from .events import TruncatedEvent as TruncatedEvent
|
||||
from .events import UsageEvent as UsageEvent
|
||||
from .loop import DEFAULT_MAX_ROUNDS as DEFAULT_MAX_ROUNDS
|
||||
from .loop import run_chat_loop as run_chat_loop
|
||||
@@ -26,6 +27,7 @@ from .notices import format_moment as format_moment
|
||||
from .notices import has_notice as has_notice
|
||||
from .notices import resume_notice as resume_notice
|
||||
from .notices import skill_tools_notice as skill_tools_notice
|
||||
from .notices import truncated_turn_notice as truncated_turn_notice
|
||||
from .tools import DangerClassifier as DangerClassifier
|
||||
from .tools import ToolConfirmHook as ToolConfirmHook
|
||||
from .tools import ToolDefinition as ToolDefinition
|
||||
|
||||
@@ -27,7 +27,8 @@ from .directive_checkpoint import (
|
||||
parse_checkpoint_bands,
|
||||
)
|
||||
from .events import UsageEvent
|
||||
from .loop import DEFAULT_MAX_ROUNDS, run_chat_loop
|
||||
from .loop import DEFAULT_MAX_ROUNDS, DEFAULT_MAX_TRUNCATION_CONTINUES, run_chat_loop
|
||||
from .notices import TRUNCATED_NOTICE_KIND, notice_of_kind
|
||||
from .todo_nag import (
|
||||
DEFAULT_TODO_NAG_STRATEGY,
|
||||
parse_todo_nag_strategy,
|
||||
@@ -140,8 +141,10 @@ def incomplete_turn_user_text(messages: Sequence[AnyChatMessage]) -> str | None:
|
||||
"""User text for an incomplete turn, if history can be retried.
|
||||
|
||||
Incomplete when the last non-system message is a user message (failed before
|
||||
any commit), or tool results after a complete tool batch (failed on a later
|
||||
model round — keep tools so retry does not re-execute side effects).
|
||||
any commit), tool results after a complete tool batch (failed on a later
|
||||
model round — keep tools so retry does not re-execute side effects), or a
|
||||
partial answer under a truncation notice (the round hit the output limit; its
|
||||
answer was kept, so retry continues it instead of regenerating).
|
||||
Trailing developer messages (host notices, directive checkpoints) belong to
|
||||
the turn but never end it, so they are skipped when reading the tail.
|
||||
"""
|
||||
@@ -150,14 +153,17 @@ def incomplete_turn_user_text(messages: Sequence[AnyChatMessage]) -> str | None:
|
||||
if messages and isinstance(messages[0], SystemChatMessage):
|
||||
start = 1
|
||||
body = list(messages[start:])
|
||||
truncated = False
|
||||
while body and isinstance(body[-1], DeveloperChatMessage):
|
||||
if notice_of_kind(body[-1], TRUNCATED_NOTICE_KIND):
|
||||
truncated = True
|
||||
_ = body.pop()
|
||||
if not body:
|
||||
return None
|
||||
last = body[-1]
|
||||
if isinstance(last, UserChatMessage):
|
||||
return last.content
|
||||
if isinstance(last, ToolChatMessage):
|
||||
if isinstance(last, ToolChatMessage) or (truncated and isinstance(last, AssistantChatMessage)):
|
||||
# Walk back to the user that opened this turn.
|
||||
for msg in reversed(body):
|
||||
if isinstance(msg, UserChatMessage):
|
||||
@@ -289,6 +295,8 @@ class ChatAgent:
|
||||
memory: MemoryStore | None
|
||||
session_id: int | None
|
||||
max_rounds: int
|
||||
# Rounds a turn may carry on after hitting the output limit (0 = none).
|
||||
max_truncation_continues: int
|
||||
temperature: float | None
|
||||
# Resolved thinking strength ("" = provider default); see config.reasoning.
|
||||
reasoning_effort: str
|
||||
@@ -322,6 +330,7 @@ class ChatAgent:
|
||||
memory: MemoryStore | None = None,
|
||||
session_id: int | None = None,
|
||||
max_rounds: int = DEFAULT_MAX_ROUNDS,
|
||||
max_truncation_continues: int = DEFAULT_MAX_TRUNCATION_CONTINUES,
|
||||
temperature: float | None = None,
|
||||
reasoning_effort: str = "",
|
||||
messages: Sequence[AnyChatMessage] | None = None,
|
||||
@@ -345,6 +354,7 @@ class ChatAgent:
|
||||
self.memory = memory
|
||||
self.session_id = session_id
|
||||
self.max_rounds = max_rounds
|
||||
self.max_truncation_continues = max(0, int(max_truncation_continues))
|
||||
self.temperature = temperature
|
||||
self.reasoning_effort = reasoning_effort
|
||||
self.on_limit = on_limit
|
||||
@@ -603,6 +613,7 @@ class ChatAgent:
|
||||
model=self.model,
|
||||
tools=self.tools,
|
||||
max_rounds=self.max_rounds,
|
||||
max_truncation_continues=self.max_truncation_continues,
|
||||
temperature=self.temperature,
|
||||
reasoning_effort=self.reasoning_effort,
|
||||
on_limit=self.on_limit,
|
||||
@@ -747,6 +758,7 @@ class ChatAgent:
|
||||
memory=None,
|
||||
session_id=None,
|
||||
max_rounds=rounds,
|
||||
max_truncation_continues=self.max_truncation_continues,
|
||||
temperature=self.temperature,
|
||||
reasoning_effort=self.reasoning_effort,
|
||||
on_limit=self.on_limit,
|
||||
|
||||
@@ -34,6 +34,19 @@ class MaxRoundsEvent(Struct, tag_field="type", tag="max_rounds"):
|
||||
continued: bool = False
|
||||
|
||||
|
||||
class TruncatedEvent(Struct, tag_field="type", tag="truncated"):
|
||||
"""A round the provider cut off at the output limit (``finish_reason=length``).
|
||||
|
||||
The partial answer stays in history rather than being discarded, and the
|
||||
model is told where it stopped (see ``notices.truncated_turn_notice``);
|
||||
``continued`` says whether the turn went on with another round or is waiting
|
||||
for ``/retry``.
|
||||
"""
|
||||
|
||||
kept_chars: int = 0
|
||||
continued: bool = False
|
||||
|
||||
|
||||
class ErrorEvent(Struct, tag_field="type", tag="error"):
|
||||
message: str
|
||||
retryable: bool = True
|
||||
@@ -60,6 +73,7 @@ type AgentEvent = (
|
||||
| ToolCallEvent
|
||||
| ToolResultEvent
|
||||
| MaxRoundsEvent
|
||||
| TruncatedEvent
|
||||
| ErrorEvent
|
||||
| CancelledEvent
|
||||
| UsageEvent
|
||||
|
||||
+81
-14
@@ -43,8 +43,11 @@ from .events import (
|
||||
TextDeltaEvent,
|
||||
ToolCallEvent,
|
||||
ToolResultEvent,
|
||||
TruncatedEvent,
|
||||
UsageEvent,
|
||||
)
|
||||
from .finish_reason import chat_finish_reason
|
||||
from .notices import truncated_turn_notice
|
||||
from .todo_nag import (
|
||||
DEFAULT_TODO_NAG_STRATEGY,
|
||||
inject_todo_nag_for_stack_with_events,
|
||||
@@ -68,6 +71,10 @@ if TYPE_CHECKING:
|
||||
|
||||
|
||||
DEFAULT_MAX_ROUNDS = 32
|
||||
# Rounds cut off at the output limit are kept and the model carries on in another
|
||||
# round, this many times per turn; 0 stops after the truncated round (`/retry`
|
||||
# continues it from the kept partial answer).
|
||||
DEFAULT_MAX_TRUNCATION_CONTINUES = 2
|
||||
|
||||
|
||||
async def _call_on_limit(on_limit: LimitContinueHook, reason: str) -> bool:
|
||||
@@ -150,6 +157,11 @@ def _assistant_has_payload(assistant: AssistantChatMessage) -> bool:
|
||||
return bool(isinstance(reasoning, str) and reasoning.strip())
|
||||
|
||||
|
||||
def _kept_chars(assistant: AssistantChatMessage) -> int:
|
||||
"""How much answer a cut-off round had written (0 if only reasoning/blocks)."""
|
||||
return len(assistant.content) if isinstance(assistant.content, str) else 0
|
||||
|
||||
|
||||
def _finish_reason_value(finish: object) -> str | None:
|
||||
if finish is UNSET or finish is None:
|
||||
return None
|
||||
@@ -163,20 +175,26 @@ def _validate_assistant_terminal(
|
||||
*,
|
||||
finish_reason: str | None,
|
||||
stream_terminal: bool,
|
||||
) -> None:
|
||||
"""Raise when the round is not a usable agent stop.
|
||||
) -> bool:
|
||||
"""Raise when the round is not a usable agent stop; True if it hit the output limit.
|
||||
|
||||
Distinguishes empty generation, truncated/filtered stops, and missing
|
||||
stream terminals (network/client glitch) from a normal stop/tool_calls end.
|
||||
Distinguishes empty generation, filtered stops, and missing stream terminals
|
||||
(network/client glitch) from a normal stop/tool_calls end. A round cut off at
|
||||
the output limit is not an error of its own: what it wrote is usable, so the
|
||||
caller keeps it and tells the model where it stopped. A truncated round that
|
||||
wrote nothing stays a failure — there is nothing to continue from.
|
||||
"""
|
||||
reason = (finish_reason or "").lower() or None
|
||||
canonical = chat_finish_reason(reason, has_tool_calls=False)
|
||||
|
||||
if reason in {"length", "content_filter"}:
|
||||
label = "truncated (max tokens)" if reason == "length" else "content filter"
|
||||
detail = ""
|
||||
if isinstance(assistant.content, str) and assistant.content.strip():
|
||||
detail = f"; partial text kept ({len(assistant.content)} chars)"
|
||||
msg = f"model stopped early: {label}{detail}"
|
||||
if canonical == "length":
|
||||
if _assistant_has_payload(assistant):
|
||||
return True
|
||||
msg = "model stopped early: output limit reached with nothing written"
|
||||
raise RuntimeError(msg)
|
||||
|
||||
if canonical == "content_filter":
|
||||
msg = "model stopped early: content filter"
|
||||
raise RuntimeError(msg)
|
||||
|
||||
if reason in {"failed", "incomplete", "cancelled"}:
|
||||
@@ -184,7 +202,7 @@ def _validate_assistant_terminal(
|
||||
raise RuntimeError(msg)
|
||||
|
||||
if _assistant_has_payload(assistant):
|
||||
return
|
||||
return False
|
||||
|
||||
if not stream_terminal and reason is None:
|
||||
msg = (
|
||||
@@ -249,7 +267,7 @@ async def _non_stream_round(
|
||||
choice = response.choices[0]
|
||||
assistant = choice.message
|
||||
finish = _finish_reason_value(choice.finish_reason)
|
||||
_validate_assistant_terminal(
|
||||
truncated = _validate_assistant_terminal(
|
||||
assistant,
|
||||
finish_reason=finish,
|
||||
stream_terminal=True,
|
||||
@@ -260,6 +278,8 @@ async def _non_stream_round(
|
||||
if isinstance(assistant.content, str) and assistant.content:
|
||||
yield TextDeltaEvent(content=assistant.content)
|
||||
yield AssistantMessageEvent(message=assistant)
|
||||
if truncated:
|
||||
yield TruncatedEvent(kept_chars=_kept_chars(assistant))
|
||||
yield UsageEvent(
|
||||
usage=resolve_round_usage(response.usage, param.messages, assistant),
|
||||
)
|
||||
@@ -376,12 +396,14 @@ async def _stream_round(
|
||||
# If we got payload but no finish_reason, treat as terminal (some providers
|
||||
# omit it); if empty and no finish_reason, flag as missing terminal.
|
||||
stream_terminal = saw_terminal or _assistant_has_payload(assistant)
|
||||
_validate_assistant_terminal(
|
||||
truncated = _validate_assistant_terminal(
|
||||
assistant,
|
||||
finish_reason=finish_reason,
|
||||
stream_terminal=stream_terminal,
|
||||
)
|
||||
yield AssistantMessageEvent(message=assistant)
|
||||
if truncated:
|
||||
yield TruncatedEvent(kept_chars=_kept_chars(assistant))
|
||||
yield UsageEvent(
|
||||
usage=resolve_round_usage(last_api_usage, param.messages, assistant),
|
||||
)
|
||||
@@ -408,6 +430,27 @@ async def _assistant_round(
|
||||
yield event
|
||||
|
||||
|
||||
def _keep_truncated_round(
|
||||
messages: list[AnyChatMessage],
|
||||
truncated: TruncatedEvent,
|
||||
*,
|
||||
continued: bool,
|
||||
) -> TruncatedEvent:
|
||||
"""Keep a round cut off at the output limit and tell the model where it stopped.
|
||||
|
||||
The answer and its thinking blocks stay in ``messages`` (the provider verifies
|
||||
those blocks, so dropping them would make the round uncontinuable), but the
|
||||
round's tool calls go: a half-written call must not run, its arguments may not
|
||||
even parse, and the model re-issues it after reading the notice. Returns the
|
||||
event to yield, with the continuation outcome filled in.
|
||||
"""
|
||||
last = messages[-1]
|
||||
if isinstance(last, AssistantChatMessage) and last.tool_calls is not UNSET and last.tool_calls:
|
||||
messages[-1] = msgspec.structs.replace(last, tool_calls=UNSET)
|
||||
messages.append(truncated_turn_notice().to_message())
|
||||
return msgspec.structs.replace(truncated, continued=continued)
|
||||
|
||||
|
||||
def _last_assistant(messages: list[AnyChatMessage], pre_len: int) -> AssistantChatMessage:
|
||||
if len(messages) <= pre_len:
|
||||
msg = "model round produced no assistant message"
|
||||
@@ -448,13 +491,14 @@ async def _maybe_inject_directive_checkpoints(
|
||||
return new_band
|
||||
|
||||
|
||||
async def run_chat_loop( # noqa: C901 — multi-phase tool loop
|
||||
async def run_chat_loop( # noqa: C901, PLR0915 — multi-phase tool loop
|
||||
client: AnyLLMClient,
|
||||
messages: list[AnyChatMessage],
|
||||
*,
|
||||
model: str,
|
||||
tools: ToolRegistry | None = None,
|
||||
max_rounds: int = DEFAULT_MAX_ROUNDS,
|
||||
max_truncation_continues: int = DEFAULT_MAX_TRUNCATION_CONTINUES,
|
||||
temperature: float | None = None,
|
||||
reasoning_effort: str = "",
|
||||
on_limit: LimitContinueHook | None = None,
|
||||
@@ -481,6 +525,12 @@ async def run_chat_loop( # noqa: C901 — multi-phase tool loop
|
||||
the provider's own default); each API surface maps it onto its own wire field
|
||||
(see ``config.reasoning``).
|
||||
|
||||
A round the provider cuts off at the output limit (``finish_reason=length``)
|
||||
is kept instead of failing the turn: its answer and thinking blocks stay in
|
||||
``messages``, its tool calls are dropped, and a truncation notice tells the
|
||||
model where it stopped. Up to *max_truncation_continues* rounds per turn then
|
||||
carry on from there (0 = stop, leaving the turn for ``/retry``).
|
||||
|
||||
*provider_tools* are hosted/provider-side tools (OpenAI Responses only, e.g.
|
||||
``web_search``) merged into the request as opaque dicts; never executed by the
|
||||
:class:`ToolRegistry`. Ignored by non-Responses clients.
|
||||
@@ -500,6 +550,7 @@ async def run_chat_loop( # noqa: C901 — multi-phase tool loop
|
||||
|
||||
rounds_used = 0
|
||||
allowance = max_rounds
|
||||
truncation_continues = 0
|
||||
# Calibrate soft-compact from last model call's prompt_tokens (API preferred).
|
||||
prompt_tokens_hint: int | None = None
|
||||
sent_estimate_tokens: int | None = None
|
||||
@@ -532,6 +583,7 @@ async def run_chat_loop( # noqa: C901 — multi-phase tool loop
|
||||
|
||||
pre_len = len(messages)
|
||||
last_usage_event: UsageEvent | None = None
|
||||
truncated: TruncatedEvent | None = None
|
||||
async for event in _assistant_round(
|
||||
client,
|
||||
param,
|
||||
@@ -544,9 +596,24 @@ async def run_chat_loop( # noqa: C901 — multi-phase tool loop
|
||||
prompt_tokens_hint = event.usage.prompt_tokens
|
||||
sent_estimate_tokens = sent_est
|
||||
last_usage_event = event
|
||||
if isinstance(event, TruncatedEvent):
|
||||
# Yielded below, once the continuation decision is known.
|
||||
truncated = event
|
||||
continue
|
||||
yield event
|
||||
|
||||
assistant = _last_assistant(messages, pre_len)
|
||||
if truncated is not None:
|
||||
# The provider cut the round off at the output limit: keep what it
|
||||
# wrote and tell the model where it stopped (see the helper).
|
||||
continued = truncation_continues < max_truncation_continues
|
||||
if continued:
|
||||
truncation_continues += 1
|
||||
yield _keep_truncated_round(messages, truncated, continued=continued)
|
||||
if continued:
|
||||
continue
|
||||
return
|
||||
|
||||
tool_calls = assistant.tool_calls
|
||||
has_tools = tool_calls is not UNSET and bool(tool_calls) and tools is not None
|
||||
if has_tools:
|
||||
|
||||
@@ -17,6 +17,7 @@ from plyngent.lmproto.anthropic.model import (
|
||||
AnthropicAssistantMessage,
|
||||
AnthropicMessageResponse,
|
||||
AnthropicMessagesParam,
|
||||
AnthropicOutputConfig,
|
||||
AnthropicResponseText,
|
||||
AnthropicServerToolUseContent,
|
||||
AnthropicTextContent,
|
||||
@@ -384,7 +385,7 @@ def _thinking_from_param(
|
||||
*,
|
||||
thinking_budget_tokens: int,
|
||||
) -> AnthropicThinkingConfig | Unset:
|
||||
"""Build the ``thinking`` block for *param* (UNSET = send nothing).
|
||||
"""Build the extended-thinking block for *param* (UNSET = send nothing).
|
||||
|
||||
``reasoning_effort = "none"`` disables thinking; any other level enables it
|
||||
with the explicit *thinking_budget_tokens*, else with the budget that level
|
||||
@@ -423,11 +424,77 @@ def _thinking_and_max_tokens(
|
||||
return max_tokens, thinking
|
||||
|
||||
|
||||
# Effort level → Anthropic ``output_config.effort`` on an adaptive-thinking
|
||||
# model. ``minimal`` has no API level of its own and lands on the lowest one;
|
||||
# unknown levels fall back to "medium" (the budgets above do the same).
|
||||
_ANTHROPIC_ADAPTIVE_EFFORTS: dict[str, str] = {
|
||||
"minimal": "low",
|
||||
"low": "low",
|
||||
"medium": "medium",
|
||||
"high": "high",
|
||||
"xhigh": "xhigh",
|
||||
"max": "max",
|
||||
}
|
||||
_ANTHROPIC_FALLBACK_ADAPTIVE_EFFORT = "medium"
|
||||
|
||||
|
||||
def _adaptive_thinking_from_param(
|
||||
param: ChatCompletionsParam,
|
||||
) -> tuple[AnthropicThinkingConfig | Unset, AnthropicOutputConfig | Unset]:
|
||||
"""Build the adaptive-thinking blocks for *param* (UNSET = send nothing).
|
||||
|
||||
The shape for a model that rejects ``thinking.type.enabled``: adaptive
|
||||
thinking, steered by ``output_config.effort`` rather than a token budget.
|
||||
``reasoning_effort = "none"`` still disables thinking; a configured
|
||||
``thinking_budget_tokens`` has no adaptive equivalent and is left out.
|
||||
"""
|
||||
effort = param.reasoning_effort
|
||||
if effort is UNSET:
|
||||
return UNSET, UNSET
|
||||
if effort == "none":
|
||||
return AnthropicThinkingConfig(type="disabled"), UNSET
|
||||
level = _ANTHROPIC_ADAPTIVE_EFFORTS.get(str(effort), _ANTHROPIC_FALLBACK_ADAPTIVE_EFFORT)
|
||||
return AnthropicThinkingConfig(type="adaptive"), AnthropicOutputConfig(effort=level)
|
||||
|
||||
|
||||
def thinking_config_refused(message: str) -> bool:
|
||||
"""Whether a ``/messages`` failure is the API refusing a request's thinking block.
|
||||
|
||||
The documented wordings are ``"thinking.type.enabled" is not supported for
|
||||
this model. Use "thinking.type.adaptive" and "output_config.effort" …`` and
|
||||
the same sentence for ``"thinking.type.disabled"``; gateways paraphrase them
|
||||
(``claude-opus-5-5 requires adaptive thinking; omit thinking or use
|
||||
thinking.type=adaptive …``), so match the one word they all carry — callers
|
||||
only get here after a request that sent a thinking block.
|
||||
"""
|
||||
return "adaptive" in message.lower()
|
||||
|
||||
|
||||
def _thinking_for_param(
|
||||
param: ChatCompletionsParam,
|
||||
*,
|
||||
thinking_budget_tokens: int,
|
||||
adaptive: bool,
|
||||
) -> tuple[int, AnthropicThinkingConfig | Unset, AnthropicOutputConfig | Unset]:
|
||||
"""Resolve *param*'s ``max_tokens`` and thinking blocks for one shape.
|
||||
|
||||
Extended thinking needs ``max_tokens`` raised above its budget (see
|
||||
:func:`_thinking_and_max_tokens`); adaptive thinking carries no budget, so
|
||||
the caller's ``max_tokens`` stands.
|
||||
"""
|
||||
if adaptive:
|
||||
thinking, output_config = _adaptive_thinking_from_param(param)
|
||||
return _max_tokens_from_param(param), thinking, output_config
|
||||
max_tokens, thinking = _thinking_and_max_tokens(param, thinking_budget_tokens=thinking_budget_tokens)
|
||||
return max_tokens, thinking, UNSET
|
||||
|
||||
|
||||
def chat_param_to_anthropic_param(
|
||||
param: ChatCompletionsParam,
|
||||
*,
|
||||
thinking_budget_tokens: int = 0,
|
||||
provider_tools: Sequence[dict[str, Any]] | None = None,
|
||||
adaptive_thinking: bool = False,
|
||||
) -> AnthropicMessagesParam:
|
||||
"""Build :class:`AnthropicMessagesParam` from a chat completions param.
|
||||
|
||||
@@ -435,13 +502,22 @@ def chat_param_to_anthropic_param(
|
||||
(0 = derive it from the param's ``reasoning_effort``); *provider_tools* are
|
||||
hosted tools (web_search, …) as opaque dicts, merged after local function
|
||||
tools and never executed by the registry.
|
||||
|
||||
*adaptive_thinking* asks for the thinking shape of a model that rejects
|
||||
extended thinking: ``thinking.type = "adaptive"`` with the effort in
|
||||
``output_config`` instead of a ``budget_tokens`` budget (see
|
||||
:func:`_adaptive_thinking_from_param`).
|
||||
"""
|
||||
system, messages = chat_messages_to_anthropic(param.messages)
|
||||
tools: list[AnthropicToolDefinition | dict[str, Any]] = list(
|
||||
tool_items_to_anthropic_tools(param.tools if param.tools is not UNSET else None)
|
||||
)
|
||||
tools.extend(provider_tools_to_anthropic_tools(provider_tools))
|
||||
max_tokens, thinking = _thinking_and_max_tokens(param, thinking_budget_tokens=thinking_budget_tokens)
|
||||
max_tokens, thinking, output_config = _thinking_for_param(
|
||||
param,
|
||||
thinking_budget_tokens=thinking_budget_tokens,
|
||||
adaptive=adaptive_thinking,
|
||||
)
|
||||
|
||||
kwargs: dict[str, Any] = {
|
||||
"model": param.model,
|
||||
@@ -450,6 +526,8 @@ def chat_param_to_anthropic_param(
|
||||
}
|
||||
if thinking is not UNSET:
|
||||
kwargs["thinking"] = thinking
|
||||
if output_config is not UNSET:
|
||||
kwargs["output_config"] = output_config
|
||||
if system:
|
||||
kwargs["system"] = system
|
||||
if tools:
|
||||
|
||||
@@ -11,11 +11,15 @@ from __future__ import annotations
|
||||
import json
|
||||
from typing import TYPE_CHECKING, Any, Final, cast
|
||||
|
||||
import msgspec
|
||||
from msgspec import UNSET
|
||||
|
||||
from plyngent.agent.messages_bridge import (
|
||||
anthropic_response_to_chat_completion,
|
||||
anthropic_stop_to_finish_reason,
|
||||
anthropic_usage_to_dict,
|
||||
chat_param_to_anthropic_param,
|
||||
thinking_config_refused,
|
||||
)
|
||||
from plyngent.agent.stream_chunks import (
|
||||
finish_reason_chunk,
|
||||
@@ -40,7 +44,7 @@ if TYPE_CHECKING:
|
||||
from collections.abc import AsyncIterator, Sequence
|
||||
|
||||
from plyngent.lmproto.anthropic.client import AnthropicClient
|
||||
from plyngent.lmproto.anthropic.model import AnthropicStreamEvent
|
||||
from plyngent.lmproto.anthropic.model import AnthropicMessagesParam, AnthropicStreamEvent
|
||||
from plyngent.lmproto.openai_compatible.model import (
|
||||
ChatCompletionChunk,
|
||||
ChatCompletionResponse,
|
||||
@@ -53,6 +57,61 @@ def _thinking_budget(client: AnthropicClient) -> int:
|
||||
return max(0, int(getattr(client, "thinking_budget_tokens", 0)))
|
||||
|
||||
|
||||
# The thinking types a refusal has a fallback for: extended thinking becomes
|
||||
# adaptive, and a block disabling thinking is dropped (such a model thinks
|
||||
# whatever the request says).
|
||||
_RETRYABLE_THINKING_TYPES: Final = frozenset({"enabled", "disabled"})
|
||||
|
||||
|
||||
def _create_param(
|
||||
client: AnthropicClient,
|
||||
param: ChatCompletionsParam,
|
||||
provider_tools: Sequence[dict[str, Any]] | None,
|
||||
) -> AnthropicMessagesParam:
|
||||
"""Build the ``/messages`` request for *param*, the way its model now takes it.
|
||||
|
||||
A model this session found refusing extended thinking gets adaptive thinking;
|
||||
one that refuses being told to stop thinking is sent no block at all (its
|
||||
thinking is always on, so the API answers a block that disables it with a
|
||||
400 — see :func:`_retry_with_thinking_fallback`).
|
||||
"""
|
||||
model = param.model
|
||||
create = chat_param_to_anthropic_param(
|
||||
param,
|
||||
thinking_budget_tokens=_thinking_budget(client),
|
||||
provider_tools=provider_tools,
|
||||
adaptive_thinking=client.refuses_thinking_type(model, "enabled"),
|
||||
)
|
||||
if (
|
||||
create.thinking is not UNSET
|
||||
and create.thinking.type == "disabled"
|
||||
and client.refuses_thinking_type(model, "disabled")
|
||||
):
|
||||
return msgspec.structs.replace(create, thinking=UNSET)
|
||||
return create
|
||||
|
||||
|
||||
def _retry_with_thinking_fallback(
|
||||
client: AnthropicClient,
|
||||
param: ChatCompletionsParam,
|
||||
provider_tools: Sequence[dict[str, Any]] | None,
|
||||
error: Exception,
|
||||
) -> AnthropicMessagesParam | None:
|
||||
"""The request to retry with when the model refused the thinking block.
|
||||
|
||||
``None`` when the failure is not that refusal, or when there is nothing left
|
||||
to try. The refusal is remembered on the client, so only the model's first
|
||||
request pays the extra round trip.
|
||||
"""
|
||||
if not thinking_config_refused(str(error)):
|
||||
return None
|
||||
create = _create_param(client, param, provider_tools)
|
||||
if create.thinking is UNSET or create.thinking.type not in _RETRYABLE_THINKING_TYPES:
|
||||
return None
|
||||
client.note_refused_thinking_type(param.model, create.thinking.type)
|
||||
return _create_param(client, param, provider_tools)
|
||||
|
||||
|
||||
class _ToolBlockState:
|
||||
"""Tracks an in-flight tool_use content block during SSE streaming."""
|
||||
|
||||
@@ -365,6 +424,33 @@ def _chunks_for_content_delta(
|
||||
]
|
||||
|
||||
|
||||
async def _stream_events(
|
||||
client: AnthropicClient,
|
||||
param: ChatCompletionsParam,
|
||||
*,
|
||||
provider_tools: Sequence[dict[str, Any]] | None = None,
|
||||
) -> AsyncIterator[AnthropicStreamEvent]:
|
||||
"""Open the Messages stream, retrying once with the shape the model takes.
|
||||
|
||||
A refusal arrives before the stream yields anything, so the retry cannot
|
||||
duplicate events the caller has already seen.
|
||||
"""
|
||||
create = _create_param(client, param, provider_tools)
|
||||
saw_event = False
|
||||
try:
|
||||
async for event in await client.messages(create, stream=True):
|
||||
saw_event = True
|
||||
yield event
|
||||
except RuntimeError as exc:
|
||||
if saw_event:
|
||||
raise
|
||||
retry = _retry_with_thinking_fallback(client, param, provider_tools, exc)
|
||||
if retry is None:
|
||||
raise
|
||||
async for event in await client.messages(retry, stream=True):
|
||||
yield event
|
||||
|
||||
|
||||
async def _stream_as_chat_chunks(
|
||||
client: AnthropicClient,
|
||||
param: ChatCompletionsParam,
|
||||
@@ -379,15 +465,9 @@ async def _stream_as_chat_chunks(
|
||||
``message_stop``. Errors raise ``RuntimeError`` so the agent loop can
|
||||
surface them as retryable failures.
|
||||
"""
|
||||
create = chat_param_to_anthropic_param(
|
||||
param,
|
||||
thinking_budget_tokens=_thinking_budget(client),
|
||||
provider_tools=provider_tools,
|
||||
)
|
||||
stream = await client.messages(create, stream=True)
|
||||
state = _StreamState(param.model)
|
||||
|
||||
async for event in stream:
|
||||
async for event in _stream_events(client, param, provider_tools=provider_tools):
|
||||
for chunk in _chunks_for_event(event, state):
|
||||
yield chunk
|
||||
if state.saw_message_stop:
|
||||
@@ -419,14 +499,16 @@ async def dispatch_messages(
|
||||
surface's own server-tool definition is built from them and the results
|
||||
arrive inside the same response, so nothing is executed locally.
|
||||
"""
|
||||
create = chat_param_to_anthropic_param(
|
||||
param,
|
||||
thinking_budget_tokens=_thinking_budget(client),
|
||||
provider_tools=provider_tools,
|
||||
)
|
||||
if stream:
|
||||
return _stream_as_chat_chunks(client, param, provider_tools=provider_tools)
|
||||
response = await client.messages(create, stream=False)
|
||||
create = _create_param(client, param, provider_tools)
|
||||
try:
|
||||
response = await client.messages(create, stream=False)
|
||||
except RuntimeError as exc:
|
||||
retry = _retry_with_thinking_fallback(client, param, provider_tools, exc)
|
||||
if retry is None:
|
||||
raise
|
||||
response = await client.messages(retry, stream=False)
|
||||
return anthropic_response_to_chat_completion(response)
|
||||
|
||||
|
||||
|
||||
@@ -27,6 +27,11 @@ if TYPE_CHECKING:
|
||||
# developer messages (durable directive checkpoints share the role).
|
||||
NOTICE_MARKER = "[plyngent notice"
|
||||
|
||||
# Notice kind for a turn the provider cut off at the output limit. The turn's
|
||||
# partial answer is kept, so the agent reads such a turn as unfinished and
|
||||
# ``/retry`` continues it (see ``ChatAgent.pending_retry_text``).
|
||||
TRUNCATED_NOTICE_KIND = "truncated"
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Notice:
|
||||
@@ -41,14 +46,18 @@ class Notice:
|
||||
return DeveloperChatMessage(content=f"{NOTICE_MARKER}: {self.kind}]\n{self.body.strip()}")
|
||||
|
||||
|
||||
def notice_of_kind(message: AnyChatMessage, kind: str) -> bool:
|
||||
"""Whether *message* is a notice of *kind*."""
|
||||
return isinstance(message, DeveloperChatMessage) and message.content.startswith(f"{NOTICE_MARKER}: {kind}]")
|
||||
|
||||
|
||||
def has_notice(messages: Sequence[AnyChatMessage], kind: str) -> bool:
|
||||
"""Whether *messages* already carries a notice of *kind*.
|
||||
|
||||
Notices are persisted with the transcript, so a host asks this before
|
||||
pushing one that belongs to the session rather than to an event.
|
||||
"""
|
||||
marker = f"{NOTICE_MARKER}: {kind}]"
|
||||
return any(isinstance(message, DeveloperChatMessage) and message.content.startswith(marker) for message in messages)
|
||||
return any(notice_of_kind(message, kind) for message in messages)
|
||||
|
||||
|
||||
def format_moment(moment: datetime | None) -> str:
|
||||
@@ -82,6 +91,27 @@ def resume_notice(*, last_active: datetime | None) -> Notice:
|
||||
return Notice(kind="resume", body=body, summary=summary)
|
||||
|
||||
|
||||
def truncated_turn_notice() -> Notice:
|
||||
"""Notice for a round the provider cut off at the output limit.
|
||||
|
||||
The partial answer is kept in the transcript (thinking blocks included, so
|
||||
the provider still verifies them), which is why the model has to be told the
|
||||
round was not finished: it continues from what it already wrote, re-issues
|
||||
the tool call that was dropped, and keeps the remainder short enough to fit.
|
||||
"""
|
||||
body = (
|
||||
"Your previous round was cut off at the output limit (max_tokens). The partial answer it wrote "
|
||||
"was kept in the transcript, and any tool call in that round was discarded — re-issue it if you "
|
||||
"still need it. Continue from where the answer stopped instead of writing it again from the "
|
||||
"start, and keep the rest short enough to finish: split the work, or hand off what is left."
|
||||
)
|
||||
return Notice(
|
||||
kind=TRUNCATED_NOTICE_KIND,
|
||||
body=body,
|
||||
summary="output truncated (max_tokens); partial answer kept",
|
||||
)
|
||||
|
||||
|
||||
_ASIDE_BODIES: dict[str, str] = {
|
||||
"read": (
|
||||
"Side question: this exchange is not saved to the session transcript and does not "
|
||||
|
||||
@@ -21,6 +21,7 @@ from plyngent.agent import (
|
||||
TextDeltaEvent,
|
||||
ToolCallEvent,
|
||||
ToolResultEvent,
|
||||
TruncatedEvent,
|
||||
UsageEvent,
|
||||
)
|
||||
from plyngent.lmproto.openai_compatible.model import AssistantFunctionToolCall
|
||||
@@ -1289,6 +1290,14 @@ async def render_events( # noqa: C901, PLR0915 — one branch per event type
|
||||
)
|
||||
else:
|
||||
click.secho(f"\n[max rounds reached: {event.rounds}]", fg="red")
|
||||
elif isinstance(event, TruncatedEvent):
|
||||
flush_assistant()
|
||||
tool_lines.close_line()
|
||||
tail = "continuing" if event.continued else "/retry continues from here"
|
||||
click.secho(
|
||||
f"\n[limit] output truncated ({event.kept_chars} chars kept); {tail}",
|
||||
fg="yellow",
|
||||
)
|
||||
elif isinstance(event, UsageEvent):
|
||||
_ = event
|
||||
else:
|
||||
|
||||
@@ -36,6 +36,10 @@ class AnthropicClient:
|
||||
models_decoder: msgspec.json.Decoder[AnthropicModelsResponse]
|
||||
# Explicit ``thinking`` budget for this client (0 = derive from the effort).
|
||||
thinking_budget_tokens: int = 0
|
||||
# Per model id, the ``thinking.type`` values the API refused this session; the
|
||||
# dispatch retries with the shape the model takes instead (see
|
||||
# ``agent.messages_dispatch``).
|
||||
refused_thinking_types: dict[str, set[str]]
|
||||
_api_key: str
|
||||
_api_version: str
|
||||
|
||||
@@ -46,9 +50,18 @@ class AnthropicClient:
|
||||
self.stream_decoder = msgspec.json.Decoder(AnthropicStreamEvent)
|
||||
self.models_decoder = msgspec.json.Decoder(AnthropicModelsResponse)
|
||||
self.thinking_budget_tokens = max(0, config.thinking_budget_tokens)
|
||||
self.refused_thinking_types = {}
|
||||
self._api_key = config.api_key
|
||||
self._api_version = config.anthropic_version
|
||||
|
||||
def refuses_thinking_type(self, model: str, thinking_type: str) -> bool:
|
||||
"""Whether the API already refused *thinking_type* for *model* this session."""
|
||||
return thinking_type in self.refused_thinking_types.get(model, ())
|
||||
|
||||
def note_refused_thinking_type(self, model: str, thinking_type: str) -> None:
|
||||
"""Remember that the API refuses *thinking_type* for *model*, saving the retry."""
|
||||
self.refused_thinking_types.setdefault(model, set()).add(thinking_type)
|
||||
|
||||
def _headers(self) -> dict[str, str]:
|
||||
return {
|
||||
"Content-Type": "application/json",
|
||||
|
||||
@@ -66,21 +66,34 @@ class AnthropicMetadata(Struct, omit_defaults=True):
|
||||
|
||||
|
||||
class AnthropicThinkingConfig(Struct, omit_defaults=True):
|
||||
"""``thinking`` request block (extended thinking).
|
||||
"""``thinking`` request block (extended or adaptive thinking).
|
||||
|
||||
``budget_tokens`` is the reasoning budget the model may spend before it
|
||||
answers; the API requires it to be at least 1024 and smaller than
|
||||
``max_tokens`` (see ``agent.messages_bridge``, which keeps both valid).
|
||||
``enabled`` is extended thinking: ``budget_tokens`` is the reasoning budget
|
||||
the model may spend before it answers; the API requires it to be at least
|
||||
1024 and smaller than ``max_tokens`` (see ``agent.messages_bridge``, which
|
||||
keeps both valid). Models from Claude 4.7 on reject ``enabled`` and take
|
||||
``adaptive`` instead, where the depth comes from ``output_config.effort``;
|
||||
``disabled`` turns thinking off where the model allows it.
|
||||
|
||||
``type`` carries no default: ``omit_defaults`` would drop a defaulted one,
|
||||
and the server reads the block as malformed without it — DeepSeek answers
|
||||
``thinking: missing field ``type``.
|
||||
"""
|
||||
|
||||
type: Literal["enabled", "disabled"]
|
||||
type: Literal["enabled", "disabled", "adaptive"]
|
||||
budget_tokens: int | Unset = UNSET
|
||||
|
||||
|
||||
class AnthropicOutputConfig(Struct, omit_defaults=True):
|
||||
"""``output_config`` request block: how much the model spends on a turn.
|
||||
|
||||
``effort`` is the adaptive-thinking knob (``low`` … ``max``); it replaces a
|
||||
``thinking.budget_tokens`` budget on the models that no longer take one.
|
||||
"""
|
||||
|
||||
effort: str | Unset = UNSET
|
||||
|
||||
|
||||
class AnthropicMessagesParam(Struct, omit_defaults=True):
|
||||
model: str
|
||||
max_tokens: int = 8192
|
||||
@@ -97,6 +110,7 @@ class AnthropicMessagesParam(Struct, omit_defaults=True):
|
||||
top_p: float | Unset = UNSET
|
||||
top_k: int | Unset = UNSET
|
||||
thinking: AnthropicThinkingConfig | Unset = UNSET
|
||||
output_config: AnthropicOutputConfig | Unset = UNSET
|
||||
|
||||
|
||||
class AnthropicUsage(Struct, omit_defaults=True):
|
||||
|
||||
@@ -14,10 +14,12 @@ from plyngent.agent import (
|
||||
ToolCallEvent,
|
||||
ToolRegistry,
|
||||
ToolResultEvent,
|
||||
TruncatedEvent,
|
||||
UsageEvent,
|
||||
run_chat_loop,
|
||||
tool,
|
||||
)
|
||||
from plyngent.agent.notices import TRUNCATED_NOTICE_KIND, notice_of_kind
|
||||
from plyngent.config.models import DatabaseConfig
|
||||
from plyngent.lmproto.openai_compatible.model import (
|
||||
AnyChatMessage,
|
||||
@@ -987,20 +989,115 @@ async def test_empty_completion_raises() -> None:
|
||||
_ = [e async for e in run_chat_loop(client, messages, model="m", stream=False)]
|
||||
|
||||
|
||||
async def test_length_finish_raises() -> None:
|
||||
async def test_length_finish_keeps_the_partial_answer_and_continues() -> None:
|
||||
client = ScriptedClient(
|
||||
[
|
||||
_response(
|
||||
AssistantChatMessage(content="partial"),
|
||||
finish_reason="length",
|
||||
)
|
||||
_response(AssistantChatMessage(content="partial"), finish_reason="length"),
|
||||
_response(AssistantChatMessage(content="rest")),
|
||||
]
|
||||
)
|
||||
messages: list[AnyChatMessage] = [UserChatMessage(content="hi")]
|
||||
with pytest.raises(RuntimeError, match="truncated"):
|
||||
events = [e async for e in run_chat_loop(client, messages, model="m", stream=False)]
|
||||
|
||||
truncated = [e for e in events if isinstance(e, TruncatedEvent)]
|
||||
assert len(truncated) == 1
|
||||
assert truncated[0].kept_chars == len("partial")
|
||||
assert truncated[0].continued
|
||||
# The partial answer stays in history under a truncation notice, and the
|
||||
# continuation round is sent that history — the model continues its answer
|
||||
# rather than writing it again from the start.
|
||||
assert isinstance(messages[1], AssistantChatMessage)
|
||||
assert messages[1].content == "partial"
|
||||
assert notice_of_kind(messages[2], TRUNCATED_NOTICE_KIND)
|
||||
assert isinstance(messages[3], AssistantChatMessage)
|
||||
assert messages[3].content == "rest"
|
||||
assert any(notice_of_kind(m, TRUNCATED_NOTICE_KIND) for m in client.calls[1].messages)
|
||||
|
||||
|
||||
async def test_length_finish_stops_once_the_continuation_budget_is_spent() -> None:
|
||||
client = ScriptedClient([_response(AssistantChatMessage(content="partial"), finish_reason="length")] * 2)
|
||||
messages: list[AnyChatMessage] = [UserChatMessage(content="hi")]
|
||||
events = [
|
||||
e
|
||||
async for e in run_chat_loop(
|
||||
client,
|
||||
messages,
|
||||
model="m",
|
||||
stream=False,
|
||||
max_truncation_continues=1,
|
||||
)
|
||||
]
|
||||
|
||||
assert [e.continued for e in events if isinstance(e, TruncatedEvent)] == [True, False]
|
||||
assert len(client.calls) == 2
|
||||
# Nothing raised: the turn ends with the partial answer kept, so `/retry`
|
||||
# continues it instead of regenerating.
|
||||
assert isinstance(messages[-2], AssistantChatMessage)
|
||||
assert messages[-2].content == "partial"
|
||||
assert notice_of_kind(messages[-1], TRUNCATED_NOTICE_KIND)
|
||||
|
||||
|
||||
async def test_length_finish_drops_the_rounds_tool_calls() -> None:
|
||||
ran: list[str] = []
|
||||
|
||||
@tool(name="boom", register=False)
|
||||
async def _boom(argument: str) -> str:
|
||||
ran.append(argument)
|
||||
return "ran"
|
||||
|
||||
call = AssistantFunctionToolCall(
|
||||
id="c1",
|
||||
function=AssistantFunctionTool(name="boom", arguments='{"argument": "x"'),
|
||||
)
|
||||
client = ScriptedClient(
|
||||
[
|
||||
_response(
|
||||
AssistantChatMessage(content="partial", tool_calls=[call]),
|
||||
finish_reason="length",
|
||||
),
|
||||
_response(AssistantChatMessage(content="done")),
|
||||
]
|
||||
)
|
||||
messages: list[AnyChatMessage] = [UserChatMessage(content="hi")]
|
||||
events = [e async for e in run_chat_loop(client, messages, model="m", tools=ToolRegistry([_boom]), stream=False)]
|
||||
|
||||
# A call cut off mid-write is not replayed: the model re-issues it after
|
||||
# reading the notice.
|
||||
assert ran == []
|
||||
assert not [e for e in events if isinstance(e, (ToolCallEvent, ToolResultEvent))]
|
||||
kept = messages[1]
|
||||
assert isinstance(kept, AssistantChatMessage)
|
||||
assert kept.content == "partial"
|
||||
assert kept.tool_calls is UNSET
|
||||
|
||||
|
||||
async def test_length_finish_without_content_still_raises() -> None:
|
||||
client = ScriptedClient([_response(AssistantChatMessage(content=None), finish_reason="length")])
|
||||
messages: list[AnyChatMessage] = [UserChatMessage(content="hi")]
|
||||
with pytest.raises(RuntimeError, match="output limit reached with nothing written"):
|
||||
_ = [e async for e in run_chat_loop(client, messages, model="m", stream=False)]
|
||||
|
||||
|
||||
async def test_truncated_turn_is_retryable() -> None:
|
||||
store = await MemoryStore.open(DatabaseConfig())
|
||||
session = await store.create_session(name="t")
|
||||
client = ScriptedClient(
|
||||
[
|
||||
_response(AssistantChatMessage(content="partial"), finish_reason="length"),
|
||||
_response(AssistantChatMessage(content="rest")),
|
||||
]
|
||||
)
|
||||
agent = ChatAgent(client, model="m", memory=store, session_id=session.sid, max_truncation_continues=0, stream=False)
|
||||
_ = [e async for e in agent.run("hi")]
|
||||
|
||||
# Kept, not rolled back: the turn is unfinished, so retry continues it.
|
||||
assert agent.pending_retry_text == "hi"
|
||||
_ = [e async for e in agent.retry()]
|
||||
assert any(notice_of_kind(m, TRUNCATED_NOTICE_KIND) for m in client.calls[1].messages)
|
||||
assert agent.pending_retry_text is None
|
||||
await store.close()
|
||||
|
||||
|
||||
async def test_content_filter_finish_raises() -> None:
|
||||
client = ScriptedClient(
|
||||
[
|
||||
|
||||
@@ -15,6 +15,7 @@ from plyngent.agent.messages_bridge import (
|
||||
chat_messages_to_anthropic,
|
||||
chat_param_to_anthropic_param,
|
||||
provider_tools_to_anthropic_tools,
|
||||
thinking_config_refused,
|
||||
tool_items_to_anthropic_tools,
|
||||
)
|
||||
from plyngent.lmproto.anthropic.model import (
|
||||
@@ -387,3 +388,64 @@ def test_chat_param_to_anthropic_param_thinking_disabled() -> None:
|
||||
assert body.thinking is not UNSET
|
||||
assert body.thinking.type == "disabled"
|
||||
assert body.thinking.budget_tokens is UNSET
|
||||
|
||||
|
||||
def test_chat_param_to_anthropic_param_adaptive_thinking_from_effort() -> None:
|
||||
body = chat_param_to_anthropic_param(_effort_param("high"), adaptive_thinking=True)
|
||||
assert body.thinking is not UNSET
|
||||
assert body.thinking.type == "adaptive"
|
||||
assert body.thinking.budget_tokens is UNSET
|
||||
assert body.output_config is not UNSET
|
||||
assert body.output_config.effort == "high"
|
||||
# Adaptive thinking spends no fixed budget, so max_tokens stays as asked.
|
||||
assert body.max_tokens == 8192
|
||||
|
||||
|
||||
def test_chat_param_to_anthropic_param_adaptive_effort_levels() -> None:
|
||||
def effort_for(level: ReasoningEffort) -> str:
|
||||
body = chat_param_to_anthropic_param(_effort_param(level), adaptive_thinking=True)
|
||||
assert body.output_config is not UNSET
|
||||
return str(body.output_config.effort)
|
||||
|
||||
assert effort_for("minimal") == "low" # no such API level; the lowest one
|
||||
assert effort_for("xhigh") == "xhigh"
|
||||
|
||||
|
||||
def test_chat_param_to_anthropic_param_adaptive_ignores_a_configured_budget() -> None:
|
||||
body = chat_param_to_anthropic_param(_effort_param("medium"), thinking_budget_tokens=20000, adaptive_thinking=True)
|
||||
assert body.thinking is not UNSET
|
||||
assert body.thinking.type == "adaptive"
|
||||
assert body.thinking.budget_tokens is UNSET
|
||||
assert body.max_tokens == 8192
|
||||
|
||||
|
||||
def test_chat_param_to_anthropic_param_adaptive_disabled_carries_no_effort() -> None:
|
||||
body = chat_param_to_anthropic_param(_effort_param("none"), adaptive_thinking=True)
|
||||
assert body.thinking is not UNSET
|
||||
assert body.thinking.type == "disabled"
|
||||
assert body.output_config is UNSET
|
||||
|
||||
|
||||
def test_chat_param_to_anthropic_param_adaptive_without_effort_sends_nothing() -> None:
|
||||
body = chat_param_to_anthropic_param(_effort_param(), adaptive_thinking=True)
|
||||
assert body.thinking is UNSET
|
||||
assert body.output_config is UNSET
|
||||
|
||||
|
||||
def test_thinking_config_refused_matches_the_api_and_gateway_wording() -> None:
|
||||
documented_enabled = (
|
||||
'messages HTTP 400: {"error":{"message":"\\"thinking.type.enabled\\" is not supported for this '
|
||||
'model. Use \\"thinking.type.adaptive\\" and \\"output_config.effort\\" to control thinking behavior."}}'
|
||||
)
|
||||
documented_disabled = (
|
||||
'messages HTTP 400: {"error":{"message":"\\"thinking.type.disabled\\" is not supported for this '
|
||||
'model. Use \\"thinking.type.adaptive\\" and \\"output_config.effort\\" to control thinking behavior."}}'
|
||||
)
|
||||
gateway = (
|
||||
'messages HTTP 400: {"error":{"message":"claude-opus-5-5 requires adaptive thinking; omit thinking '
|
||||
'or use thinking.type=adaptive and output_config.effort"}}'
|
||||
)
|
||||
assert thinking_config_refused(documented_enabled)
|
||||
assert thinking_config_refused(documented_disabled)
|
||||
assert thinking_config_refused(gateway)
|
||||
assert not thinking_config_refused('messages HTTP 400: {"error":{"message":"max_tokens: required"}}')
|
||||
|
||||
@@ -4,6 +4,7 @@ from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any, Literal, cast, overload
|
||||
|
||||
import pytest
|
||||
from msgspec import UNSET
|
||||
|
||||
from plyngent.agent import (
|
||||
@@ -45,6 +46,21 @@ if TYPE_CHECKING:
|
||||
from collections.abc import AsyncIterator, Sequence
|
||||
|
||||
|
||||
# What the API answers a request whose thinking block it does not take (a gateway
|
||||
# paraphrase of the documented wording, which names adaptive thinking either way).
|
||||
_THINKING_REFUSED_400: dict[str, str] = {
|
||||
"enabled": (
|
||||
'messages HTTP 400: {"error":{"type":"error","message":"claude-opus-5-5 requires adaptive '
|
||||
'thinking; omit thinking or use thinking.type=adaptive and output_config.effort"}}'
|
||||
),
|
||||
"disabled": (
|
||||
'messages HTTP 400: {"error":{"type":"error","message":"\\"thinking.type.disabled\\" is not '
|
||||
'supported for this model. Use \\"thinking.type.adaptive\\" and \\"output_config.effort\\" to '
|
||||
'control thinking behavior."}}'
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _message_response(
|
||||
*,
|
||||
text: str = "hello",
|
||||
@@ -73,21 +89,43 @@ class ScriptedMessagesClient:
|
||||
|
||||
kind: str = "messages"
|
||||
calls: list[AnthropicMessagesParam]
|
||||
# Mirrors AnthropicClient: explicit thinking budget (0 = derive from effort).
|
||||
# Mirrors AnthropicClient: explicit thinking budget (0 = derive from effort),
|
||||
# and the thinking types the dispatch found the API refusing.
|
||||
thinking_budget_tokens: int
|
||||
refused_thinking_types: dict[str, set[str]]
|
||||
_non_stream: list[AnthropicMessageResponse]
|
||||
_stream_events: list[list[AnthropicStreamEvent]]
|
||||
_refusals: frozenset[str]
|
||||
_error: str | None
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
non_stream: Sequence[AnthropicMessageResponse] | None = None,
|
||||
stream_events: Sequence[Sequence[AnthropicStreamEvent]] | None = None,
|
||||
refusals: Sequence[str] = (),
|
||||
error: str | None = None,
|
||||
) -> None:
|
||||
self.calls = []
|
||||
self.thinking_budget_tokens = 0
|
||||
self.refused_thinking_types = {}
|
||||
self._non_stream = list(non_stream or ())
|
||||
self._stream_events = [list(events) for events in (stream_events or ())]
|
||||
self._refusals = frozenset(refusals)
|
||||
self._error = error
|
||||
|
||||
def refuses_thinking_type(self, model: str, thinking_type: str) -> bool:
|
||||
return thinking_type in self.refused_thinking_types.get(model, ())
|
||||
|
||||
def note_refused_thinking_type(self, model: str, thinking_type: str) -> None:
|
||||
self.refused_thinking_types.setdefault(model, set()).add(thinking_type)
|
||||
|
||||
def _reject(self, param: AnthropicMessagesParam) -> None:
|
||||
"""Raise what the API raises for a request it refuses to run."""
|
||||
if self._error is not None:
|
||||
raise RuntimeError(self._error)
|
||||
if param.thinking is not UNSET and param.thinking.type in self._refusals:
|
||||
raise RuntimeError(_THINKING_REFUSED_400[param.thinking.type])
|
||||
|
||||
@overload
|
||||
async def messages(
|
||||
@@ -107,13 +145,16 @@ class ScriptedMessagesClient:
|
||||
if not self._stream_events:
|
||||
msg = "no more scripted stream events"
|
||||
raise RuntimeError(msg)
|
||||
events = self._stream_events.pop(0)
|
||||
|
||||
async def _gen() -> AsyncIterator[AnthropicStreamEvent]:
|
||||
for event in events:
|
||||
# The real client reads the error body on the first iteration,
|
||||
# so a refused request consumes no scripted events.
|
||||
self._reject(param)
|
||||
for event in self._stream_events.pop(0):
|
||||
yield event
|
||||
|
||||
return _gen()
|
||||
self._reject(param)
|
||||
if not self._non_stream:
|
||||
msg = "no more scripted messages"
|
||||
raise RuntimeError(msg)
|
||||
@@ -602,3 +643,85 @@ async def test_search_turn_reports_the_prompt_the_delta_counts() -> None:
|
||||
assert isinstance(usage, dict)
|
||||
assert usage["prompt_tokens"] == 12468
|
||||
assert usage["completion_tokens"] == 198
|
||||
|
||||
|
||||
def _thinking_param(model: str = "claude-opus-5-5", effort: str = "high") -> ChatCompletionsParam:
|
||||
return ChatCompletionsParam(
|
||||
model=model,
|
||||
messages=[UserChatMessage(content="hi")],
|
||||
reasoning_effort=cast("Any", effort),
|
||||
)
|
||||
|
||||
|
||||
async def test_dispatch_messages_retries_with_adaptive_thinking() -> None:
|
||||
client = ScriptedMessagesClient(non_stream=[_message_response(text="done")], refusals=("enabled",))
|
||||
result = await dispatch_messages(cast("Any", client), _thinking_param(), stream=False)
|
||||
assert result.choices[0].message.content == "done"
|
||||
assert len(client.calls) == 2
|
||||
refused, retried = client.calls
|
||||
assert refused.thinking is not UNSET
|
||||
assert refused.thinking.type == "enabled"
|
||||
assert retried.thinking is not UNSET
|
||||
assert retried.thinking.type == "adaptive"
|
||||
assert retried.output_config is not UNSET
|
||||
assert retried.output_config.effort == "high"
|
||||
|
||||
|
||||
async def test_dispatch_messages_remembers_the_refusing_model() -> None:
|
||||
client = ScriptedMessagesClient(
|
||||
non_stream=[_message_response(text="one"), _message_response(text="two")],
|
||||
refusals=("enabled",),
|
||||
)
|
||||
param = _thinking_param()
|
||||
_ = await dispatch_messages(cast("Any", client), param, stream=False)
|
||||
_ = await dispatch_messages(cast("Any", client), param, stream=False)
|
||||
# Only the model's first request pays the refused attempt.
|
||||
assert [call.thinking.type for call in client.calls] == ["enabled", "adaptive", "adaptive"]
|
||||
|
||||
|
||||
async def test_dispatch_messages_drops_a_refused_disabled_block() -> None:
|
||||
"""A model whose thinking is always on answers ``disabled`` with a 400."""
|
||||
client = ScriptedMessagesClient(
|
||||
non_stream=[_message_response(text="one"), _message_response(text="two")],
|
||||
refusals=("enabled", "disabled"),
|
||||
)
|
||||
param = _thinking_param(effort="none")
|
||||
result = await dispatch_messages(cast("Any", client), param, stream=False)
|
||||
assert result.choices[0].message.content == "one"
|
||||
assert len(client.calls) == 2
|
||||
assert client.calls[0].thinking.type == "disabled"
|
||||
# No block at all: there is nothing left to ask such a model for.
|
||||
assert client.calls[1].thinking is UNSET
|
||||
_ = await dispatch_messages(cast("Any", client), param, stream=False)
|
||||
assert client.calls[2].thinking is UNSET
|
||||
|
||||
|
||||
async def test_dispatch_messages_stream_retries_with_adaptive_thinking() -> None:
|
||||
events: list[AnthropicStreamEvent] = [
|
||||
AnthropicContentBlockStart(index=0, content_block=AnthropicRawContentBlock(type="text", text="")),
|
||||
AnthropicContentBlockDelta(index=0, delta=AnthropicRawContentBlock(type="text_delta", text="streamed")),
|
||||
AnthropicMessageDelta(delta={"stop_reason": "end_turn"}, usage=AnthropicUsage(output_tokens=3)),
|
||||
AnthropicMessageStop(),
|
||||
]
|
||||
client = ScriptedMessagesClient(stream_events=[events], refusals=("enabled",))
|
||||
stream = await dispatch_messages(cast("Any", client), _thinking_param(), stream=True)
|
||||
chunks = [chunk async for chunk in cast("Any", stream)]
|
||||
texts = [c.choices[0].delta.content for c in chunks if c.choices and isinstance(c.choices[0].delta.content, str)]
|
||||
assert texts == ["streamed"]
|
||||
assert len(client.calls) == 2
|
||||
assert client.calls[1].thinking.type == "adaptive"
|
||||
|
||||
|
||||
async def test_dispatch_messages_does_not_retry_other_failures() -> None:
|
||||
client = ScriptedMessagesClient(error='messages HTTP 400: {"error":{"message":"max_tokens: required"}}')
|
||||
with pytest.raises(RuntimeError, match="max_tokens"):
|
||||
_ = await dispatch_messages(cast("Any", client), _thinking_param(), stream=False)
|
||||
assert len(client.calls) == 1
|
||||
|
||||
|
||||
async def test_dispatch_messages_does_not_retry_a_request_without_thinking() -> None:
|
||||
client = ScriptedMessagesClient(error=_THINKING_REFUSED_400["enabled"])
|
||||
param = ChatCompletionsParam(model="claude-opus-5-5", messages=[UserChatMessage(content="hi")])
|
||||
with pytest.raises(RuntimeError, match="adaptive thinking"):
|
||||
_ = await dispatch_messages(cast("Any", client), param, stream=False)
|
||||
assert len(client.calls) == 1
|
||||
|
||||
@@ -2,7 +2,16 @@ from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta, timezone
|
||||
|
||||
from plyngent.agent import ChatAgent, Notice, aside_notice, format_moment, has_notice, resume_notice, skill_tools_notice
|
||||
from plyngent.agent import (
|
||||
ChatAgent,
|
||||
Notice,
|
||||
aside_notice,
|
||||
format_moment,
|
||||
has_notice,
|
||||
resume_notice,
|
||||
skill_tools_notice,
|
||||
truncated_turn_notice,
|
||||
)
|
||||
from plyngent.agent.chat import incomplete_turn_user_text
|
||||
from plyngent.config.models import DatabaseConfig
|
||||
from plyngent.lmproto.openai_compatible.model import (
|
||||
@@ -124,3 +133,26 @@ def test_trailing_developer_message_keeps_incomplete_turn_detectable() -> None:
|
||||
# A finished turn (trailing assistant) stays non-retryable past a notice.
|
||||
done = [UserChatMessage(content="do it"), AssistantChatMessage(content="ok"), notice]
|
||||
assert incomplete_turn_user_text(done) is None
|
||||
|
||||
|
||||
def test_truncated_turn_notice_keeps_the_partial_answer_retryable() -> None:
|
||||
"""A kept partial answer is unfinished work, so retry continues it."""
|
||||
tail = [
|
||||
UserChatMessage(content="do it"),
|
||||
AssistantChatMessage(content="partial answer"),
|
||||
truncated_turn_notice().to_message(),
|
||||
]
|
||||
assert incomplete_turn_user_text(tail) == "do it"
|
||||
# The same tail without the truncation notice is a finished turn.
|
||||
assert incomplete_turn_user_text(tail[:2]) is None
|
||||
|
||||
|
||||
def test_truncated_turn_notice_tells_the_model_where_it_stopped() -> None:
|
||||
notice = truncated_turn_notice()
|
||||
assert notice.kind == "truncated"
|
||||
assert notice.to_message().content.startswith("[plyngent notice: truncated]")
|
||||
assert "max_tokens" in notice.body
|
||||
# The dropped call and the kept answer are both spelled out.
|
||||
assert "discarded" in notice.body
|
||||
assert "kept in the transcript" in notice.body
|
||||
assert "max_tokens" in notice.summary
|
||||
|
||||
@@ -2,7 +2,14 @@ from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from plyngent.agent import ErrorEvent, ReasoningDeltaEvent, TextDeltaEvent, ToolCallEvent, ToolResultEvent
|
||||
from plyngent.agent import (
|
||||
ErrorEvent,
|
||||
ReasoningDeltaEvent,
|
||||
TextDeltaEvent,
|
||||
ToolCallEvent,
|
||||
ToolResultEvent,
|
||||
TruncatedEvent,
|
||||
)
|
||||
from plyngent.cli.display import (
|
||||
_RUN_BATCH_CMD_LIMIT,
|
||||
_clear_streamed_lines,
|
||||
@@ -1257,3 +1264,12 @@ async def test_pretty_prefix_closed_before_error_line(
|
||||
assert "* Read 'a.txt' \n" in out
|
||||
assert "[error] source=tool boom\n" in out
|
||||
assert "* Read 'a.txt' (error)\n" in out
|
||||
|
||||
|
||||
async def test_render_truncated_round(capsys: pytest.CaptureFixture[str]) -> None:
|
||||
"""A round cut off at the output limit says so, and how the turn goes on."""
|
||||
await render_events(_aiter([TruncatedEvent(kept_chars=42, continued=True)]))
|
||||
assert "[limit] output truncated (42 chars kept); continuing\n" in capsys.readouterr().out
|
||||
|
||||
await render_events(_aiter([TruncatedEvent(kept_chars=42, continued=False)]))
|
||||
assert "[limit] output truncated (42 chars kept); /retry continues from here\n" in capsys.readouterr().out
|
||||
|
||||
Reference in New Issue
Block a user