From 8926980a023dda0c5f4b0ca4464c09113c4dae6b Mon Sep 17 00:00:00 2001 From: worldmozara Date: Wed, 21 Oct 2026 21:39:41 +0800 Subject: [PATCH] docs: describe the image gate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `image_in` is not just the mention switch any more: the table names where each image-carrying path is gated, and says why the gate exists — a text-only model answers `[Unsupported Image]`, and a gateway that cannot read image blocks feeds the model raw base64, which is easy to mistake for a delivered picture. --- AGENTS.md | 2 +- doc/images.md | 24 +++++++++++++++++++++--- 2 files changed, 22 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 7b8d79d..d9aeb55 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -83,7 +83,7 @@ Async SQLAlchemy + aiosqlite. `MemoryStore`: schema init (+ lightweight SQLite ` - **Kind-based dispatch**: clients expose `kind` (`chat_completions` | `responses` | `messages`). Agent history stays chat-completions-shaped; transport adapters return synthetic `ChatCompletionResponse` / stream chunks. - **`responses_bridge` / `responses_dispatch`**: chat ↔ OpenAI Responses; stream text/reasoning deltas (`response.reasoning_text.done` repeats what its fragments spelled out, so the loop streams each part once and still keeps the whole text in the round's message); tool calls + usage on `response.completed`; a request carrying `tools` returns every round's reasoning as a `reasoning` item of `reasoning_text` parts — "必须完整回传 `reasoning_content` … 即使该轮模型未实际进行工具调用" (a missing item and an empty part are both a 400, so a round that kept none sends a blank line) — and a round's `function_call` items stay contiguous ahead of their outputs (`call, output, call, output` was a 400: the second call reads as a round of its own without reasoning); reasoning is read back from the item's `content` parts. - **`messages_bridge` / `messages_dispatch`**: chat ↔ Anthropic Messages; system fold; tool_use/tool_result; stream text + tool argument fragments; `thinking` from the request's `reasoning_effort` (level → token budget, floor 1024, `max_tokens` grown when the budget would leave no room) plus the client's configured `thinking_budget_tokens`; a thinking block the model refuses (from Claude 4.7 on, `thinking.type.enabled` is a 400 asking for adaptive thinking + `output_config.effort`; a model whose thinking is always on refuses `thinking.type.disabled`) gets one retry with the shape it takes — adaptive thinking with the level as the effort, or no block at all — and the refusal is remembered per model on the client, so only its first request pays the round trip (only a refusal that arrives before the stream yields anything is retried; a later failure has already been seen); a server tool (`server_tool_use` + its `web_search_tool_result`) is the surface's own business — it streams `input_json_delta` fragments like a call but nothing local asked for it, so only a `tool_use` block becomes a tool call and nothing is executed; `thinking` and the two search blocks are kept as `AssistantChatMessage.provider_blocks` (the thinking text also lands in `reasoning_content`, so it streams as reasoning) and go back with the round, which is also why a round that kept only those blocks is not an empty completion; a chat-completions request drops the field (`compat.coerce_chat_messages`). -- **Images** (see `doc/images.md`): a user turn's `content` may be a part array (`ChatTextPart` + `ChatImagePart`), and a tool result carries its pictures in `ToolChatMessage.images`; each surface lifts them into what it can express — chat-completions content blocks (tool-result images move into the user turn after the tool run, since a `tool` message holds text only), Anthropic `image` blocks (inside `tool_result` for a tool result), Responses `input_image` parts (in a `function_call_output` item). Pictures belong to user turns: an image in a system or assistant message is a provider 400. `agent/images.py` sniffs the format from the bytes and caps an inline picture at 8 MiB, because every round re-sends it; `[providers.*.models.*].image_in` is the opt-in. +- **Images** (see `doc/images.md`): a user turn's `content` may be a part array (`ChatTextPart` + `ChatImagePart`), and a tool result carries its pictures in `ToolChatMessage.images`; each surface lifts them into what it can express — chat-completions content blocks (tool-result images move into the user turn after the tool run, since a `tool` message holds text only), Anthropic `image` blocks (inside `tool_result` for a tool result), Responses `input_image` parts (in a `function_call_output` item). Pictures belong to user turns: an image in a system or assistant message is a provider 400. `agent/images.py` sniffs the format from the bytes and caps an inline picture at 8 MiB, because every round re-sends it; `[providers.*.models.*].image_in` is the opt-in, and every image-carrying path checks it — an `@path` mention, `view_image`, `fetch_local`'s image body, and (as a safety net in the registry) an MCP server's or plugin's `ToolOutput`. - **Provider-side tools**: `provider_tools` on the OpenAI **and** DeepSeek presets (list of dicts; `[{type="web_search"}]` when omitted; `[]` disables) threaded via `ChatAgent`/`run_chat_loop` into Responses and Messages; never executed by `ToolRegistry`. The Messages surface declares its own server tool instead — a bare `web_search` becomes `{type="web_search_20250305", name="web_search"}` (the search then runs inside the response, and its `encrypted_content` results go back verbatim with the round — `AssistantChatMessage.provider_blocks`) — and a DeepSeek provider on a surface that ignores hosted tools (`config.routing.local_web_search`) gets the local `web_search` tool from `tools/net/search.py` instead. - **`@tool` / `ToolRegistry`**: decorator infers JSON Schema from type hints; execute tools by name. A handler returns its text, or a `ToolOutput(text, images)` when it also produced a picture: `execute` keeps the text-only contract and `execute_output` returns the pair, which `run_chat_loop` stores as the tool message's text plus its `images` (`view_image`, an image body from `fetch_local`, and an MCP `image` content block all take that path). Before running, `execute` answers a call that omits a required argument with the accepted names (`_missing_required_error`) — the handler's bare `TypeError` never reaches the model, and no confirm prompt fires for a call that cannot run. Shape mistakes inside a *present* argument are the tool's own job (see the `chat` bullet). `@tool(writes=("path",))` names a handler's file arguments: one batch applies calls that name the same file one at a time, in call order — the lock is taken before the confirm prompt, so a pending prompt cannot be overtaken — while different files keep running in parallel. - **`run_chat_loop`**: multi-round tool loop; default **streaming** text deltas + stream tool-call merge; parallel tools; tool-result char budget; soft context compact on request (**API-calibrated** after first usage when available); cooperative cancel points; optional `on_limit`; resolved `reasoning_effort` on every request (see Config). A round the provider cut off at the output limit is kept, not failed: the answer and its thinking blocks stay in `messages`, the round's tool calls are dropped (a half-written call must not run — the model re-issues it), and up to `max_truncation_continues` (2) further rounds carry on from a truncation notice; 0 spends the turn as unfinished instead, for `/retry`. diff --git a/doc/images.md b/doc/images.md index 934aabd..cb8a1dc 100644 --- a/doc/images.md +++ b/doc/images.md @@ -93,9 +93,27 @@ completed in its quoted form). ## Opting in -`[providers..models.""].image_in` is the gate — an `@image` mention -is skipped, with a note naming the key, on a model that does not set it, and the -default catalogs mark what takes pictures (`deepseek-flash` yes, +`[providers..models.""].image_in` is the gate, and it covers every +way a picture can enter a turn — not only an `@image` mention: + +| Path | Where it is gated | +|---|---| +| an `@image` mention (`cli/mentions.py`) | skipped, with a note naming the key | +| the `view_image` tool | `error: model '…' does not accept image input (set image_in = true under [providers.x.models."y"])` | +| an image body from `fetch_local` | the metadata stays, the picture is left behind with the same reason | +| an MCP server's image content, a plugin's `ToolOutput` | dropped by the registry's safety net (`agent/tools.py`), with the reason appended to the text | + +`ReplState.sync_image_capability` publishes the selected model's verdict on +`InstanceState.workspace` and runs on every provider/model/client rebuild, so a +`/model` switch flips the gate at once. A host that binds no instance declares +nothing, and the picture is sent as usual. + +The point is to fail loudly: without the gate a text-only model answers +`[Unsupported Image]` one round later, and a gateway that cannot read image +blocks hands the model raw base64 — which reads like the picture arrived when it +did not. + +The default catalogs mark what takes pictures (`deepseek-flash` yes, `deepseek-v4-pro` no, OpenAI's defaults yes). A model missing from the catalog has to be declared: the flag is an explicit opt-in.