docs: describe image input across the three surfaces

`doc/images.md` holds the message shape, how `@path` attaches a file, the
per-surface mapping, the provider limits, what a live probe against DeepSeek
confirmed (`tool_result` and `function_call_output` images work, `detail: low`
is 3.4x cheaper, an assistant-message image is a 400), and why the Files API is
not used.

AGENTS.md gets the invariants and the mention/config keys; `doc/architecture.md`
and the example TOML point at it.
This commit is contained in:
2026-10-21 18:52:14 +08:00
parent 154509b929
commit 029ac47350
4 changed files with 121 additions and 6 deletions
+2
View File
@@ -39,4 +39,6 @@
Plugins (third-party entry points, allowlisted under ``[plugins]``): [plugins.md](./plugins.md).
Conversation units (a *round* = one request→response plus its tool batch, a
*turn* = a user message to the explicit stop): [turns-and-rounds.md](./turns-and-rounds.md).
Images (a turn's pictures, a tool result's pictures, what each wire surface
takes): [images.md](./images.md).
+108
View File
@@ -0,0 +1,108 @@
# Images
A model that takes pictures can look at a screenshot, a chart, or a diagram
instead of reading a description of it. plyngent carries images as content
parts on messages; each wire surface lifts them into its own shape.
## Shape
`lmproto/openai_compatible/model.py`:
| Type | Meaning |
|---|---|
| `ChatImageUrl(url, detail)` | one picture: an `http(s)` link or an inline `data:` URL |
| `ChatImagePart(image_url)` | that payload as a `{"type": "image_url"}` block |
| `ChatTextPart(text)` | the text block beside images |
| `UserChatMessage.content` | `str`, or `list[ChatTextPart \| ChatImagePart]` when the turn carries pictures |
| `ToolChatMessage.images` | the pictures a tool returned, beside its text `content` |
Prose-only readers go through `content_to_text`, which renders each image as
`[image]` (summaries, transcripts, exports, the retry preview).
`agent/images.py` owns the bytes: it sniffs the format from the file's content
(JPEG `ff d8 ff`, PNG, GIF, WebP `RIFF….WEBP`), builds the data URL, and caps an
inline picture at `MAX_INLINE_IMAGE_BYTES` (8 MiB) — far below what a provider
accepts, because the message is re-sent every round and lives in the session's
JSON message column.
## Per surface
| Surface | A user turn's images | A tool result's images |
|---|---|---|
| chat completions (`compat.chat_messages_with_images`) | already content blocks | a `tool` message may only carry text, so the images move into the user turn emitted right after the tool run |
| Anthropic `/messages` (`messages_bridge`) | `image` blocks beside the text block | inside the `tool_result` block's content, beside a text block |
| Responses (`responses_bridge`) | `input_text` + `input_image` parts | in the `function_call_output` item's `output` array |
A chat-completions request drops the parts of a message it cannot express; no
history is rewritten (the stored message keeps its own images).
## Limits (provider side)
| Limit | Value |
|---|---|
| Formats | JPEG, PNG, GIF, WebP (sniffed from bytes, never the file name) |
| Request body | 48 MiB |
| One image, inline or URL | 32 MiB |
| One image, Files API `file_id` | 64 MiB |
| Images per request | 600 |
| Image side | 8192 px, or 4096 px when a request holds 15+ images |
| External URL | 8192 characters, downloaded within 60 s |
| Tokens per image | at most 1024 (a picture is resized to ~1300x1300 px, and one below ~544x544 is scaled up) |
Images belong to **user** messages only: a picture in a system or assistant
message is a 400 (`Image in assistant message is unsupported`). A tool result
lives inside a user turn, which is why it can carry one.
## Verified against the DeepSeek API
The 2026-10-17 docs plus a live probe (`deepseek-flash`):
- the same picture is accepted on all three surfaces, and the usage is identical;
- an image inside `tool_result` content works, and the model reads it — the
round that asked for a screenshot got the layout, the colours and the pixel
span back;
- an image inside `function_call_output` works too (the round needs its
`reasoning` item, which `responses_bridge` already sends);
- `detail: "low"` measured 3.4x cheaper than `original` on a 1200x900 picture
(240 vs 708 prompt tokens) with no loss on "which colour where"; the floor is
210 tokens even for a tiny picture, because small images are scaled up;
- re-sending a picture in a later round hits the provider's prefix cache
(`cache_read_input_tokens`), so the "history is paid for again" cost is not
the worst case;
- `/anthropic/messages` and `/anthropic/v1/messages` both answer, so the
client's path stays as it is.
## Attaching one
Type `@path` in the line — no command, no separate step:
```
what is wrong with this layout? @shots/broken.png
```
`cli/mentions.py` reads the token (whitespace-delimited, `@"path with spaces"`
for names that hold one; an e-mail address is not a mention), resolves it against
the working directory, checks the path denylist, and attaches it. An image
becomes an image part; any other file is appended as text under a
`--- @path ---` header, capped like a tool result (its `truncate_token` keeps the
remainder resumable). A problem — a missing file, a denylisted path, a picture a
text-only model cannot take — prints a note and leaves the turn going.
Tab completes the path after `@` (directories end in `/`, a name with a space is
completed in its quoted form).
## Opting in
`[providers.<name>.models."<model>"].image_in` is the gate — an `@image` mention
is skipped, with a note naming the key, on a model that does not set it, and the
default catalogs mark what takes pictures (`deepseek-flash` yes,
`deepseek-v4-pro` no, OpenAI's defaults yes). A model missing from the catalog
has to be declared: the flag is an explicit opt-in.
## Not implemented
- **Files API** (`POST /files` → `file_id`): it lifts the 32 MiB per-image cap
and avoids re-uploading the same bytes, but it does not change the token bill
(the model still processes the picture in every request that carries it), so a
local 8 MiB cap plus resizing covers the common case.
- **Image output**: the API documents none.
+6 -2
View File
@@ -124,7 +124,9 @@ access_key_or_token = "sk-replace-me"
# timeout = { connect = 10, read = 600 }
[providers.openai_compat.models]
"gpt-4o-mini" = { text = true, cost_factor = 1.0 }
# ``image_in`` is the opt-in for a model that reads pictures (JPEG, PNG, GIF,
# WebP); an ``@image`` mention is skipped without it. See doc/images.md.
"gpt-4o-mini" = { text = true, image_in = true, cost_factor = 1.0 }
# --- OpenAI platform (Responses API; tag is "openai") ---
# If ``preset`` is omitted, it defaults to "openai".
@@ -202,5 +204,7 @@ access_key_or_token = "sk-replace-me"
# search_sources = ["deepseek"]
#
# [providers.deepseek.models]
# "deepseek-flash" = { text = true, convention = "responses" }
# Defaults: "deepseek-flash" takes pictures (image_in = true), "deepseek-v4-pro"
# is text-only.
# "deepseek-flash" = { text = true, image_in = true, convention = "responses" }
# "deepseek-v4-pro" = { text = true, reasoning_effort = "max" }