mirror of
https://github.com/NCBM/plyngent.git
synced 2026-11-07 07:17:54 +08:00
NCBM
b2d6acae26
Uploading a picture and referencing it by id is the one image path the API offers that we do not use, and the vendor shapes do not line up: the upload is the same multipart `POST …/files` with a `file` part everywhere, but the object that comes back is `object`/`bytes`/unix time on OpenAI, `type`/`size_bytes`/ RFC 3339 on Anthropic, and DeepSeek serves both dialects under two base URLs; the reference block differs a third time per surface, and OpenAI's chat completions cannot name an image by id at all. Recorded next to the inline-path limits, with the three constraints that decide any future wiring: an id belongs to the provider that minted it (like `provider_blocks`), the OpenAI path implies the Responses convention, and the token bill is unchanged — uploading only buys the inline size cap, a smaller request body, and not re-sending the same bytes every round. The Anthropic per-file limit is marked unverified: its API reference is unreachable from this host's region.
159 lines
8.8 KiB
Markdown
159 lines
8.8 KiB
Markdown
# Images
|
||
|
||
A model that takes pictures can look at a screenshot, a chart, or a diagram
|
||
instead of reading a description of it. plyngent carries images as content
|
||
parts on messages; each wire surface lifts them into its own shape.
|
||
|
||
## Shape
|
||
|
||
`lmproto/openai_compatible/model.py`:
|
||
|
||
| Type | Meaning |
|
||
|---|---|
|
||
| `ChatImageUrl(url, detail)` | one picture: an `http(s)` link or an inline `data:` URL |
|
||
| `ChatImagePart(image_url)` | that payload as a `{"type": "image_url"}` block |
|
||
| `ChatTextPart(text)` | the text block beside images |
|
||
| `UserChatMessage.content` | `str`, or `list[ChatTextPart \| ChatImagePart]` when the turn carries pictures |
|
||
| `ToolChatMessage.images` | the pictures a tool returned, beside its text `content` |
|
||
|
||
Prose-only readers go through `content_to_text`, which renders each image as
|
||
`[image]` (summaries, transcripts, exports, the retry preview).
|
||
|
||
`agent/images.py` owns the bytes: it sniffs the format from the file's content
|
||
(JPEG `ff d8 ff`, PNG, GIF, WebP `RIFF….WEBP`), builds the data URL, and caps an
|
||
inline picture at `MAX_INLINE_IMAGE_BYTES` (8 MiB) — far below what a provider
|
||
accepts, because the message is re-sent every round and lives in the session's
|
||
JSON message column.
|
||
|
||
## Per surface
|
||
|
||
| Surface | A user turn's images | A tool result's images |
|
||
|---|---|---|
|
||
| chat completions (`compat.chat_messages_with_images`) | already content blocks | a `tool` message may only carry text, so the images move into the user turn emitted right after the tool run |
|
||
| Anthropic `/messages` (`messages_bridge`) | `image` blocks beside the text block | inside the `tool_result` block's content, beside a text block |
|
||
| Responses (`responses_bridge`) | `input_text` + `input_image` parts | in the `function_call_output` item's `output` array |
|
||
|
||
A chat-completions request drops the parts of a message it cannot express; no
|
||
history is rewritten (the stored message keeps its own images).
|
||
|
||
## Limits (provider side)
|
||
|
||
| Limit | Value |
|
||
|---|---|
|
||
| Formats | JPEG, PNG, GIF, WebP (sniffed from bytes, never the file name) |
|
||
| Request body | 48 MiB |
|
||
| One image, inline or URL | 32 MiB |
|
||
| One image, Files API `file_id` | 64 MiB |
|
||
| Images per request | 600 |
|
||
| Image side | 8192 px, or 4096 px when a request holds 15+ images |
|
||
| External URL | 8192 characters, downloaded within 60 s |
|
||
| Tokens per image | at most 1024 (a picture is resized to ~1300x1300 px, and one below ~544x544 is scaled up) |
|
||
|
||
Images belong to **user** messages only: a picture in a system or assistant
|
||
message is a 400 (`Image in assistant message is unsupported`). A tool result
|
||
lives inside a user turn, which is why it can carry one.
|
||
|
||
## Files API (upload once, reference by id)
|
||
|
||
Every surface we speak to can take a picture as an uploaded file instead of
|
||
inline bytes. The upload half of that is de-facto one shape; the object it
|
||
returns and the way a message refers to it are not standardized at all. Field
|
||
names below come from each vendor's own reference (DeepSeek `guides/files_api`,
|
||
OpenAI `resources/files` + `guides/file-inputs`, Anthropic `api/beta/files`).
|
||
|
||
| | OpenAI | Anthropic | DeepSeek |
|
||
|---|---|---|---|
|
||
| Upload | `POST /v1/files`, multipart: `file` + `purpose` (`user_data`, `vision`, …), optional `expires_after[anchor]` / `[seconds]` | `POST /v1/files`, multipart: `file` only; needs `anthropic-version` and `anthropic-beta: files-api-2025-04-14` | `POST /files`, OpenAI shape (`file` + `purpose="user_data"`, optional `expires_after[…]`); the same operations are also served under `/anthropic/v1/files` in the Anthropic shape |
|
||
| Id | `file-abc123` | `file_01HX…` | `file-api-…` |
|
||
| File object | `object:"file"`, `bytes`, `created_at` (unix seconds), `filename`, `purpose`, `expires_at` when it was asked for | `type:"file"`, `size_bytes`, `created_at` (RFC 3339), `filename`, `mime_type`, `downloadable` | both dialects: OpenAI shape on `/files`, Anthropic shape on `/anthropic/v1/files` |
|
||
| Reference in a message | Responses `{"type":"input_image","file_id":…}` (mutually exclusive with `image_url`); chat completions cannot reference an image by id at all — its `image_url` part has no `file_id`, and its `file` part is documents only | `{"type":"image","source":{"type":"file","file_id":…}}`, plus the beta header | chat: `{"type":"file","file_id":…}`, or inline `file_data` + `filename`; Anthropic: `source.type="file"` plus beta; Responses: `input_image.file_id` |
|
||
| Limits | 512 MB per file (what `purpose` allows) | not verified here (the API reference is unreachable from this host's region) | 64 MiB per file, 25 GiB and 10 000 files per account, expiry 1 h – 30 d or permanent |
|
||
|
||
What is shared: `POST …/files` as `multipart/form-data` with a `file` part,
|
||
and a string id back. What is not: the field names (`bytes` vs `size_bytes`),
|
||
the type tag (`object` vs `type`), the timestamp format, the cursor and paging
|
||
fields, the beta header, and every reference block above. A client therefore
|
||
needs one uploader and three reference shapes — the split this module already
|
||
has.
|
||
|
||
Three rules if this is ever wired in:
|
||
|
||
- a `file_id` is scoped to the provider (and account) that minted it, so a
|
||
stored id belongs to the surface that produced it, exactly like
|
||
`AssistantChatMessage.provider_blocks`; `/provider` cannot carry it across;
|
||
- OpenAI's chat-completions surface cannot reference an image id, so on that
|
||
provider the feature implies the Responses convention;
|
||
- the token bill does not change: the model still processes the picture in
|
||
every request that carries it. Uploading buys the 32 MiB inline cap, a
|
||
smaller request body, and not re-sending the same screenshot each round.
|
||
|
||
## Verified against the DeepSeek API
|
||
|
||
The 2026-10-17 docs plus a live probe (`deepseek-flash`):
|
||
|
||
- the same picture is accepted on all three surfaces, and the usage is identical;
|
||
- an image inside `tool_result` content works, and the model reads it — the
|
||
round that asked for a screenshot got the layout, the colours and the pixel
|
||
span back;
|
||
- an image inside `function_call_output` works too (the round needs its
|
||
`reasoning` item, which `responses_bridge` already sends);
|
||
- `detail: "low"` measured 3.4x cheaper than `original` on a 1200x900 picture
|
||
(240 vs 708 prompt tokens) with no loss on "which colour where"; the floor is
|
||
210 tokens even for a tiny picture, because small images are scaled up;
|
||
- re-sending a picture in a later round hits the provider's prefix cache
|
||
(`cache_read_input_tokens`), so the "history is paid for again" cost is not
|
||
the worst case;
|
||
- `/anthropic/messages` and `/anthropic/v1/messages` both answer, so the
|
||
client's path stays as it is.
|
||
|
||
## Attaching one
|
||
|
||
Type `@path` in the line — no command, no separate step:
|
||
|
||
```
|
||
what is wrong with this layout? @shots/broken.png
|
||
```
|
||
|
||
`cli/mentions.py` reads the token (whitespace-delimited, `@"path with spaces"`
|
||
for names that hold one; an e-mail address is not a mention), resolves it against
|
||
the working directory, checks the path denylist, and attaches it. An image
|
||
becomes an image part; any other file is appended as text under a
|
||
`--- @path ---` header, capped like a tool result (its `truncate_token` keeps the
|
||
remainder resumable). A problem — a missing file, a denylisted path, a picture a
|
||
text-only model cannot take — prints a note and leaves the turn going.
|
||
|
||
Tab completes the path after `@` (directories end in `/`, a name with a space is
|
||
completed in its quoted form).
|
||
|
||
## Opting in
|
||
|
||
`[providers.<name>.models."<model>"].image_in` is the gate, and it covers every
|
||
way a picture can enter a turn — not only an `@image` mention:
|
||
|
||
| Path | Where it is gated |
|
||
|---|---|
|
||
| an `@image` mention (`cli/mentions.py`) | skipped, with a note naming the key |
|
||
| the `view_image` tool | `error: model '…' does not accept image input (set image_in = true under [providers.x.models."y"])` |
|
||
| an image body from `fetch_local` | the metadata stays, the picture is left behind with the same reason |
|
||
| an MCP server's image content, a plugin's `ToolOutput` | dropped by the registry's safety net (`agent/tools.py`), with the reason appended to the text |
|
||
|
||
`ReplState.sync_image_capability` publishes the selected model's verdict on
|
||
`InstanceState.workspace` and runs on every provider/model/client rebuild, so a
|
||
`/model` switch flips the gate at once. A host that binds no instance declares
|
||
nothing, and the picture is sent as usual.
|
||
|
||
The point is to fail loudly: without the gate a text-only model answers
|
||
`[Unsupported Image]` one round later, and a gateway that cannot read image
|
||
blocks hands the model raw base64 — which reads like the picture arrived when it
|
||
did not.
|
||
|
||
The default catalogs mark what takes pictures (`deepseek-flash` yes,
|
||
`deepseek-v4-pro` no, OpenAI's defaults yes). A model missing from the catalog
|
||
has to be declared: the flag is an explicit opt-in.
|
||
|
||
## Not implemented
|
||
|
||
- **Files API**: see above — no image path uses it today; a local 8 MiB cap plus
|
||
resizing covers the common case, and the upload changes no token cost.
|
||
- **Image output**: the API documents none.
|