core/tools: show images instead of describing them

New `view_image` reads a local picture (JPEG, PNG, GIF, WebP, sniffed from its
bytes) and hands it to the model as an image part, with an optional `detail`
level — `low` measured 3.4x cheaper on a 1200x900 picture. `read_file` now
points at it instead of returning decode noise.

`fetch_local` returns an image response as the picture itself, and MCP image
content blocks become image parts instead of `[image: ...]` placeholders, so a
screenshot server finally reaches the model.
This commit is contained in:
2026-10-21 18:32:50 +08:00
parent c6a94e0b66
commit e9ee71e853
14 changed files with 266 additions and 26 deletions
+18
View File
@@ -11,6 +11,8 @@ import os
import sys
_FAKE_INSTRUCTIONS = os.environ.get("PLYNGENT_MCP_FAKE_INSTRUCTIONS", "")
# A real 1x1 PNG (base64), what an image-carrying tool result looks like.
_PNG_BASE64 = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg=="
def respond(request_id: int, result: object) -> None:
@@ -61,6 +63,11 @@ def main() -> None:
"description": "Sleeps past any client timeout.",
"inputSchema": {"type": "object", "properties": {}},
},
{
"name": "shot",
"description": "Returns a picture beside its text.",
"inputSchema": {"type": "object", "properties": {}},
},
]
},
)
@@ -75,6 +82,17 @@ def main() -> None:
"isError": False,
},
)
elif name == "shot":
respond(
int(request_id),
{
"content": [
{"type": "text", "text": "shot.png"},
{"type": "image", "data": _PNG_BASE64, "mimeType": "image/png"},
],
"isError": False,
},
)
elif name == "fail":
respond(
int(request_id),