plyngent/doc/images.md

Images

A model that takes pictures can look at a screenshot, a chart, or a diagram instead of reading a description of it. plyngent carries images as content parts on messages; each wire surface lifts them into its own shape.

Shape

lmproto/openai_compatible/model.py:

Type	Meaning
ChatImageUrl(url, detail)	one picture: an http(s) link or an inline data: URL
ChatImagePart(image_url)	that payload as a {"type": "image_url"} block
ChatTextPart(text)	the text block beside images
UserChatMessage.content	str, or list[ChatTextPart | ChatImagePart] when the turn carries pictures
ToolChatMessage.images	the pictures a tool returned, beside its text content

Prose-only readers go through content_to_text, which renders each image as [image] (summaries, transcripts, exports, the retry preview).

agent/images.py owns the bytes: it sniffs the format from the file's content (JPEG ff d8 ff, PNG, GIF, WebP RIFF….WEBP), builds the data URL, and caps an inline picture at MAX_INLINE_IMAGE_BYTES (8 MiB) — far below what a provider accepts, because the message is re-sent every round and lives in the session's JSON message column.

Per surface

Surface	A user turn's images	A tool result's images
chat completions (compat.chat_messages_with_images)	already content blocks	a tool message may only carry text, so the images move into the user turn emitted right after the tool run
Anthropic /messages (messages_bridge)	image blocks beside the text block	inside the tool_result block's content, beside a text block
Responses (responses_bridge)	input_text + input_image parts	in the function_call_output item's output array

A chat-completions request drops the parts of a message it cannot express; no history is rewritten (the stored message keeps its own images).

Limits (provider side)

Limit	Value
Formats	JPEG, PNG, GIF, WebP (sniffed from bytes, never the file name)
Request body	48 MiB
One image, inline or URL	32 MiB
One image, Files API file_id	64 MiB
Images per request	600
Image side	8192 px, or 4096 px when a request holds 15+ images
External URL	8192 characters, downloaded within 60 s
Tokens per image	at most 1024 (a picture is resized to ~1300x1300 px, and one below ~544x544 is scaled up)

Images belong to user messages only: a picture in a system or assistant message is a 400 (Image in assistant message is unsupported). A tool result lives inside a user turn, which is why it can carry one.

Verified against the DeepSeek API

The 2026-10-17 docs plus a live probe (deepseek-flash):

• the same picture is accepted on all three surfaces, and the usage is identical;
• an image inside tool_result content works, and the model reads it — the round that asked for a screenshot got the layout, the colours and the pixel span back;
• an image inside function_call_output works too (the round needs its reasoning item, which responses_bridge already sends);
• detail: "low" measured 3.4x cheaper than original on a 1200x900 picture (240 vs 708 prompt tokens) with no loss on "which colour where"; the floor is 210 tokens even for a tiny picture, because small images are scaled up;
• re-sending a picture in a later round hits the provider's prefix cache (cache_read_input_tokens), so the "history is paid for again" cost is not the worst case;
• /anthropic/messages and /anthropic/v1/messages both answer, so the client's path stays as it is.

Attaching one

Type @path in the line — no command, no separate step:

what is wrong with this layout? @shots/broken.png

cli/mentions.py reads the token (whitespace-delimited, @"path with spaces" for names that hold one; an e-mail address is not a mention), resolves it against the working directory, checks the path denylist, and attaches it. An image becomes an image part; any other file is appended as text under a --- @path --- header, capped like a tool result (its truncate_token keeps the remainder resumable). A problem — a missing file, a denylisted path, a picture a text-only model cannot take — prints a note and leaves the turn going.

Tab completes the path after @ (directories end in /, a name with a space is completed in its quoted form).

Opting in

[providers.<name>.models."<model>"].image_in is the gate, and it covers every way a picture can enter a turn — not only an @image mention:

Path	Where it is gated
an @image mention (cli/mentions.py)	skipped, with a note naming the key
the view_image tool	error: model '…' does not accept image input (set image_in = true under [providers.x.models."y"])
an image body from fetch_local	the metadata stays, the picture is left behind with the same reason
an MCP server's image content, a plugin's ToolOutput	dropped by the registry's safety net (agent/tools.py), with the reason appended to the text

ReplState.sync_image_capability publishes the selected model's verdict on InstanceState.workspace and runs on every provider/model/client rebuild, so a /model switch flips the gate at once. A host that binds no instance declares nothing, and the picture is sent as usual.

The point is to fail loudly: without the gate a text-only model answers [Unsupported Image] one round later, and a gateway that cannot read image blocks hands the model raw base64 — which reads like the picture arrived when it did not.

The default catalogs mark what takes pictures (deepseek-flash yes, deepseek-v4-pro no, OpenAI's defaults yes). A model missing from the catalog has to be declared: the flag is an explicit opt-in.

Not implemented

• Files API (POST /files → file_id): it lifts the 32 MiB per-image cap and avoids re-uploading the same bytes, but it does not change the token bill (the model still processes the picture in every request that carries it), so a local 8 MiB cap plus resizing covers the common case.
• Image output: the API documents none.