plyngent/doc/images.md

# Images

A model that takes pictures can look at a screenshot, a chart, or a diagram
instead of reading a description of it. plyngent carries images as content
parts on messages; each wire surface lifts them into its own shape.

## Shape

`lmproto/openai_compatible/model.py`:

| Type | Meaning |
|---|---|
| `ChatImageUrl(url, detail)` | one picture: an `http(s)` link or an inline `data:` URL |
| `ChatImagePart(image_url)` | that payload as a `{"type": "image_url"}` block |
| `ChatTextPart(text)` | the text block beside images |
| `UserChatMessage.content` | `str`, or `list[ChatTextPart \| ChatImagePart]` when the turn carries pictures |
| `ToolChatMessage.images` | the pictures a tool returned, beside its text `content` |

Prose-only readers go through `content_to_text`, which renders each image as
`[image]` (summaries, transcripts, exports, the retry preview).

`agent/images.py` owns the bytes: it sniffs the format from the file's content
(JPEG `ff d8 ff`, PNG, GIF, WebP `RIFF….WEBP`), builds the data URL, and caps an
inline picture at `MAX_INLINE_IMAGE_BYTES` (8 MiB) — far below what a provider
accepts, because the message is re-sent every round and lives in the session's
JSON message column.

## Per surface

| Surface | A user turn's images | A tool result's images |
|---|---|---|
| chat completions (`compat.chat_messages_with_images`) | already content blocks | a `tool` message may only carry text, so the images move into the user turn emitted right after the tool run |
| Anthropic `/messages` (`messages_bridge`) | `image` blocks beside the text block | inside the `tool_result` block's content, beside a text block |
| Responses (`responses_bridge`) | `input_text` + `input_image` parts | in the `function_call_output` item's `output` array |

A chat-completions request drops the parts of a message it cannot express; no
history is rewritten (the stored message keeps its own images).

## Limits (provider side)

| Limit | Value |
|---|---|
| Formats | JPEG, PNG, GIF, WebP (sniffed from bytes, never the file name) |
| Request body | 48 MiB |
| One image, inline or URL | 32 MiB |
| One image, Files API `file_id` | 64 MiB |
| Images per request | 600 |
| Image side | 8192 px, or 4096 px when a request holds 15+ images |
| External URL | 8192 characters, downloaded within 60 s |
| Tokens per image | at most 1024 (a picture is resized to ~1300x1300 px, and one below ~544x544 is scaled up) |

Images belong to **user** messages only: a picture in a system or assistant
message is a 400 (`Image in assistant message is unsupported`). A tool result
lives inside a user turn, which is why it can carry one.

## Files API (upload once, reference by id)

Every surface we speak to can take a picture as an uploaded file instead of
inline bytes. The upload half of that is de-facto one shape; the object it
returns and the way a message refers to it are not standardized at all. Field
names below come from each vendor's own reference (DeepSeek `guides/files_api`,
OpenAI `resources/files` + `guides/file-inputs`, Anthropic `api/beta/files`).

| | OpenAI | Anthropic | DeepSeek |
|---|---|---|---|
| Upload | `POST /v1/files`, multipart: `file` + `purpose` (`user_data`, `vision`, …), optional `expires_after[anchor]` / `[seconds]` | `POST /v1/files`, multipart: `file` only; needs `anthropic-version` and `anthropic-beta: files-api-2025-04-14` | `POST /files`, OpenAI shape (`file` + `purpose="user_data"`, optional `expires_after[…]`); the same operations are also served under `/anthropic/v1/files` in the Anthropic shape |
| Id | `file-abc123` | `file_01HX…` | `file-api-…` |
| File object | `object:"file"`, `bytes`, `created_at` (unix seconds), `filename`, `purpose`, `expires_at` when it was asked for | `type:"file"`, `size_bytes`, `created_at` (RFC 3339), `filename`, `mime_type`, `downloadable` | both dialects: OpenAI shape on `/files`, Anthropic shape on `/anthropic/v1/files` |
| Reference in a message | Responses `{"type":"input_image","file_id":…}` (mutually exclusive with `image_url`); chat completions cannot reference an image by id at all — its `image_url` part has no `file_id`, and its `file` part is documents only | `{"type":"image","source":{"type":"file","file_id":…}}`, plus the beta header | chat: `{"type":"file","file_id":…}`, or inline `file_data` + `filename`; Anthropic: `source.type="file"` plus beta; Responses: `input_image.file_id` |
| Limits | 512 MB per file (what `purpose` allows) | not verified here (the API reference is unreachable from this host's region) | 64 MiB per file, 25 GiB and 10 000 files per account, expiry 1 h – 30 d or permanent |

What is shared: `POST …/files` as `multipart/form-data` with a `file` part,
and a string id back. What is not: the field names (`bytes` vs `size_bytes`),
the type tag (`object` vs `type`), the timestamp format, the cursor and paging
fields, the beta header, and every reference block above. A client therefore
needs one uploader and three reference shapes — the split this module already
has.

Three rules if this is ever wired in:

- a `file_id` is scoped to the provider (and account) that minted it, so a
  stored id belongs to the surface that produced it, exactly like
  `AssistantChatMessage.provider_blocks`; `/provider` cannot carry it across;
- OpenAI's chat-completions surface cannot reference an image id, so on that
  provider the feature implies the Responses convention;
- the token bill does not change: the model still processes the picture in
  every request that carries it. Uploading buys the 32 MiB inline cap, a
  smaller request body, and not re-sending the same screenshot each round.

## Verified against the DeepSeek API

The 2026-10-17 docs plus a live probe (`deepseek-flash`):

- the same picture is accepted on all three surfaces, and the usage is identical;
- an image inside `tool_result` content works, and the model reads it — the
  round that asked for a screenshot got the layout, the colours and the pixel
  span back;
- an image inside `function_call_output` works too (the round needs its
  `reasoning` item, which `responses_bridge` already sends);
- `detail: "low"` measured 3.4x cheaper than `original` on a 1200x900 picture
  (240 vs 708 prompt tokens) with no loss on "which colour where"; the floor is
  210 tokens even for a tiny picture, because small images are scaled up;
- re-sending a picture in a later round hits the provider's prefix cache
  (`cache_read_input_tokens`), so the "history is paid for again" cost is not
  the worst case;
- `/anthropic/messages` and `/anthropic/v1/messages` both answer, so the
  client's path stays as it is.

## Attaching one

Type `@path` in the line — no command, no separate step:

```
what is wrong with this layout? @shots/broken.png
```

`cli/mentions.py` reads the token (whitespace-delimited, `@"path with spaces"`
for names that hold one; an e-mail address is not a mention), resolves it against
the working directory, checks the path denylist, and attaches it. An image
becomes an image part; any other file is appended as text under a
`--- @path ---` header, capped like a tool result (its `truncate_token` keeps the
remainder resumable). A problem — a missing file, a denylisted path, a picture a
text-only model cannot take — prints a note and leaves the turn going.

Tab completes the path after `@` (directories end in `/`, a name with a space is
completed in its quoted form).

## Opting in

`[providers.<name>.models."<model>"].image_in` is the gate, and it covers every
way a picture can enter a turn — not only an `@image` mention:

| Path | Where it is gated |
|---|---|
| an `@image` mention (`cli/mentions.py`) | skipped, with a note naming the key |
| the `view_image` tool | `error: model '…' does not accept image input (set image_in = true under [providers.x.models."y"])` |
| an image body from `fetch_local` | the metadata stays, the picture is left behind with the same reason |
| an MCP server's image content, a plugin's `ToolOutput` | dropped by the registry's safety net (`agent/tools.py`), with the reason appended to the text |

`ReplState.sync_image_capability` publishes the selected model's verdict on
`InstanceState.workspace` and runs on every provider/model/client rebuild, so a
`/model` switch flips the gate at once. A host that binds no instance declares
nothing, and the picture is sent as usual.

The point is to fail loudly: without the gate a text-only model answers
`[Unsupported Image]` one round later, and a gateway that cannot read image
blocks hands the model raw base64 — which reads like the picture arrived when it
did not.

The default catalogs mark what takes pictures (`deepseek-flash` yes,
`deepseek-v4-pro` no, OpenAI's defaults yes). A model missing from the catalog
has to be declared: the flag is an explicit opt-in.

## Not implemented

- **Files API**: see above — no image path uses it today; a local 8 MiB cap plus
  resizing covers the common case, and the upload changes no token cost.
- **Image output**: the API documents none.