mirror of
https://github.com/NCBM/plyngent.git
synced 2026-11-07 23:47:54 +08:00
docs: describe image input across the three surfaces
`doc/images.md` holds the message shape, how `@path` attaches a file, the per-surface mapping, the provider limits, what a live probe against DeepSeek confirmed (`tool_result` and `function_call_output` images work, `detail: low` is 3.4x cheaper, an assistant-message image is a 400), and why the Files API is not used. AGENTS.md gets the invariants and the mention/config keys; `doc/architecture.md` and the example TOML point at it.
This commit is contained in:
@@ -39,4 +39,6 @@
|
||||
Plugins (third-party entry points, allowlisted under ``[plugins]``): [plugins.md](./plugins.md).
|
||||
Conversation units (a *round* = one request→response plus its tool batch, a
|
||||
*turn* = a user message to the explicit stop): [turns-and-rounds.md](./turns-and-rounds.md).
|
||||
Images (a turn's pictures, a tool result's pictures, what each wire surface
|
||||
takes): [images.md](./images.md).
|
||||
|
||||
|
||||
+108
@@ -0,0 +1,108 @@
|
||||
# Images
|
||||
|
||||
A model that takes pictures can look at a screenshot, a chart, or a diagram
|
||||
instead of reading a description of it. plyngent carries images as content
|
||||
parts on messages; each wire surface lifts them into its own shape.
|
||||
|
||||
## Shape
|
||||
|
||||
`lmproto/openai_compatible/model.py`:
|
||||
|
||||
| Type | Meaning |
|
||||
|---|---|
|
||||
| `ChatImageUrl(url, detail)` | one picture: an `http(s)` link or an inline `data:` URL |
|
||||
| `ChatImagePart(image_url)` | that payload as a `{"type": "image_url"}` block |
|
||||
| `ChatTextPart(text)` | the text block beside images |
|
||||
| `UserChatMessage.content` | `str`, or `list[ChatTextPart \| ChatImagePart]` when the turn carries pictures |
|
||||
| `ToolChatMessage.images` | the pictures a tool returned, beside its text `content` |
|
||||
|
||||
Prose-only readers go through `content_to_text`, which renders each image as
|
||||
`[image]` (summaries, transcripts, exports, the retry preview).
|
||||
|
||||
`agent/images.py` owns the bytes: it sniffs the format from the file's content
|
||||
(JPEG `ff d8 ff`, PNG, GIF, WebP `RIFF….WEBP`), builds the data URL, and caps an
|
||||
inline picture at `MAX_INLINE_IMAGE_BYTES` (8 MiB) — far below what a provider
|
||||
accepts, because the message is re-sent every round and lives in the session's
|
||||
JSON message column.
|
||||
|
||||
## Per surface
|
||||
|
||||
| Surface | A user turn's images | A tool result's images |
|
||||
|---|---|---|
|
||||
| chat completions (`compat.chat_messages_with_images`) | already content blocks | a `tool` message may only carry text, so the images move into the user turn emitted right after the tool run |
|
||||
| Anthropic `/messages` (`messages_bridge`) | `image` blocks beside the text block | inside the `tool_result` block's content, beside a text block |
|
||||
| Responses (`responses_bridge`) | `input_text` + `input_image` parts | in the `function_call_output` item's `output` array |
|
||||
|
||||
A chat-completions request drops the parts of a message it cannot express; no
|
||||
history is rewritten (the stored message keeps its own images).
|
||||
|
||||
## Limits (provider side)
|
||||
|
||||
| Limit | Value |
|
||||
|---|---|
|
||||
| Formats | JPEG, PNG, GIF, WebP (sniffed from bytes, never the file name) |
|
||||
| Request body | 48 MiB |
|
||||
| One image, inline or URL | 32 MiB |
|
||||
| One image, Files API `file_id` | 64 MiB |
|
||||
| Images per request | 600 |
|
||||
| Image side | 8192 px, or 4096 px when a request holds 15+ images |
|
||||
| External URL | 8192 characters, downloaded within 60 s |
|
||||
| Tokens per image | at most 1024 (a picture is resized to ~1300x1300 px, and one below ~544x544 is scaled up) |
|
||||
|
||||
Images belong to **user** messages only: a picture in a system or assistant
|
||||
message is a 400 (`Image in assistant message is unsupported`). A tool result
|
||||
lives inside a user turn, which is why it can carry one.
|
||||
|
||||
## Verified against the DeepSeek API
|
||||
|
||||
The 2026-10-17 docs plus a live probe (`deepseek-flash`):
|
||||
|
||||
- the same picture is accepted on all three surfaces, and the usage is identical;
|
||||
- an image inside `tool_result` content works, and the model reads it — the
|
||||
round that asked for a screenshot got the layout, the colours and the pixel
|
||||
span back;
|
||||
- an image inside `function_call_output` works too (the round needs its
|
||||
`reasoning` item, which `responses_bridge` already sends);
|
||||
- `detail: "low"` measured 3.4x cheaper than `original` on a 1200x900 picture
|
||||
(240 vs 708 prompt tokens) with no loss on "which colour where"; the floor is
|
||||
210 tokens even for a tiny picture, because small images are scaled up;
|
||||
- re-sending a picture in a later round hits the provider's prefix cache
|
||||
(`cache_read_input_tokens`), so the "history is paid for again" cost is not
|
||||
the worst case;
|
||||
- `/anthropic/messages` and `/anthropic/v1/messages` both answer, so the
|
||||
client's path stays as it is.
|
||||
|
||||
## Attaching one
|
||||
|
||||
Type `@path` in the line — no command, no separate step:
|
||||
|
||||
```
|
||||
what is wrong with this layout? @shots/broken.png
|
||||
```
|
||||
|
||||
`cli/mentions.py` reads the token (whitespace-delimited, `@"path with spaces"`
|
||||
for names that hold one; an e-mail address is not a mention), resolves it against
|
||||
the working directory, checks the path denylist, and attaches it. An image
|
||||
becomes an image part; any other file is appended as text under a
|
||||
`--- @path ---` header, capped like a tool result (its `truncate_token` keeps the
|
||||
remainder resumable). A problem — a missing file, a denylisted path, a picture a
|
||||
text-only model cannot take — prints a note and leaves the turn going.
|
||||
|
||||
Tab completes the path after `@` (directories end in `/`, a name with a space is
|
||||
completed in its quoted form).
|
||||
|
||||
## Opting in
|
||||
|
||||
`[providers.<name>.models."<model>"].image_in` is the gate — an `@image` mention
|
||||
is skipped, with a note naming the key, on a model that does not set it, and the
|
||||
default catalogs mark what takes pictures (`deepseek-flash` yes,
|
||||
`deepseek-v4-pro` no, OpenAI's defaults yes). A model missing from the catalog
|
||||
has to be declared: the flag is an explicit opt-in.
|
||||
|
||||
## Not implemented
|
||||
|
||||
- **Files API** (`POST /files` → `file_id`): it lifts the 32 MiB per-image cap
|
||||
and avoids re-uploading the same bytes, but it does not change the token bill
|
||||
(the model still processes the picture in every request that carries it), so a
|
||||
local 8 MiB cap plus resizing covers the common case.
|
||||
- **Image output**: the API documents none.
|
||||
@@ -124,7 +124,9 @@ access_key_or_token = "sk-replace-me"
|
||||
# timeout = { connect = 10, read = 600 }
|
||||
|
||||
[providers.openai_compat.models]
|
||||
"gpt-4o-mini" = { text = true, cost_factor = 1.0 }
|
||||
# ``image_in`` is the opt-in for a model that reads pictures (JPEG, PNG, GIF,
|
||||
# WebP); an ``@image`` mention is skipped without it. See doc/images.md.
|
||||
"gpt-4o-mini" = { text = true, image_in = true, cost_factor = 1.0 }
|
||||
|
||||
# --- OpenAI platform (Responses API; tag is "openai") ---
|
||||
# If ``preset`` is omitted, it defaults to "openai".
|
||||
@@ -202,5 +204,7 @@ access_key_or_token = "sk-replace-me"
|
||||
# search_sources = ["deepseek"]
|
||||
#
|
||||
# [providers.deepseek.models]
|
||||
# "deepseek-flash" = { text = true, convention = "responses" }
|
||||
# Defaults: "deepseek-flash" takes pictures (image_in = true), "deepseek-v4-pro"
|
||||
# is text-only.
|
||||
# "deepseek-flash" = { text = true, image_in = true, convention = "responses" }
|
||||
# "deepseek-v4-pro" = { text = true, reasoning_effort = "max" }
|
||||
|
||||
Reference in New Issue
Block a user