ih,dZddlmZddlZddlZddlZddlmZddlm Z m Z m Z m Z m Z ejeZehdZd$d Zd%dZd&dZd'dZd(dZd)d*dZd+dZd,d!Zd"d#gZdS)-uRouting helpers for inbound user-attached images. Two modes: native — attach images as OpenAI-style ``image_url`` content parts on the user turn. Provider adapters (Anthropic, Gemini, Bedrock, Codex, OpenAI chat.completions) already translate these into their vendor-specific multimodal formats. text — run ``vision_analyze`` on each image up-front and prepend the description to the user's text. The model never sees the pixels; it only sees a lossy text summary. This is the pre-existing behaviour and still the right choice for non-vision models. The decision is made once per message turn by :func:`decide_image_input_mode`. It reads ``agent.image_input_mode`` from config.yaml (``auto`` | ``native`` | ``text``, default ``auto``) and the active model's capability metadata. In ``auto`` mode: - If the user has explicitly configured ``auxiliary.vision.provider`` (i.e. not ``auto`` and not empty), we assume they want the text pipeline regardless of the main model — they've opted in to a specific vision backend for a reason (cost, quality, local-only, etc.). - Otherwise, if the active model reports ``supports_vision=True`` in its models.dev metadata, we attach natively. - Otherwise (non-vision model, no explicit override), we fall back to text. This keeps ``vision_analyze`` surfaced as a tool in every session — skills and agent flows that chain it (browser screenshots, deeper inspection of URL-referenced images, style-gating loops) keep working. The routing only affects *how user-attached images on the current turn* are presented to the main model. ) annotationsN)Path)AnyDictListOptionalTuple>autotextnativerawrreturnstrct|tsdS|}|tvr|SdS)z5Normalize a config value into one of the valid modes.r ) isinstancerstriplower _VALID_MODES)r vals 8/home/piyush/.hermes/hermes-agent/agent/image_routing.py _coerce_moder1sG c3  v ))++    C l 6cfgOptional[Dict[str, Any]]boolcht|tsdS|dpi}t|tsdS|dpi}t|tsdSt|dpd}t|dpd}t|dpd}|dvr|s|sdSd S) zTrue when the user configured a specific auxiliary vision backend. An explicit override means the user *wants* the text pipeline (they're paying for a dedicated vision model), so we don't silently bypass it. F auxiliaryvisionprovidermodelbase_url)r r T)rdictgetrrr)rauxrrr!r"s r_explicit_aux_vision_overrider&;s  c4 u ''+   $"C c4 u WWX   $"F fd # #u6::j))/R006688>>@@H  7##)r * * 0 0 2 2E6::j))/R006688H<hu 4rrr!Optional[bool]c|r|sdS ddlm}|||}n5#t$r(}td|||Yd}~dSd}~wwxYw|dSt |jS)z:Return True/False if we can resolve caps, None if unknown.Nr)get_model_capabilitiesu2image_routing: caps lookup failed for %s:%s — %s)agent.models_devr) Exceptionloggerdebugrsupports_vision)rr!r)capsexcs r_lookup_supports_visionr1Ts 5t;;;;;;%%h66  I8UZ\_```ttttt |t $ % %%s A AA c>d}t|trN|dpi}t|tr"t|d}|dkrdS|dkrdSt |rdSt ||}|durdSdS)a1Return ``"native"`` or ``"text"`` for the given turn. Args: provider: active inference provider ID (e.g. ``"anthropic"``, ``"openrouter"``). model: active model slug as it would be sent to the provider. cfg: loaded config.yaml dict, or None. When None, behaves as auto. r agentimage_input_moder r T)rr#r$rr&r1)rr!rmode_cfg agent_cfgsupportss rdecide_image_input_moder8csH#tGGGG$$* i & & G#IMM2D$E$EFFH8x6v%S))v&x77H4x 6rbytes Optional[str]ct|sdS|drdS|drdS|dddvrdSt|d kr|dd d kr|d d d krdS|drdSt|d kr|d d dkr|d d dvrdSdS)aDetect image MIME from magic bytes. Returns None if unrecognised. Filename-based detection (``mimetypes.guess_type``) is unreliable when upstream platforms lie about content-type. Discord, for example, can serve a PNG with ``content_type=image/webp`` for proxied/animated stickers, custom emoji previews, or images uploaded via certain bots. Anthropic strictly validates that declared media_type matches the actual bytes and returns HTTP 400 on mismatch, so we sniff to be safe. NsPNG  image/pngs image/jpeg)sGIF87asGIF89a image/gif sRIFFsWEBP image/webpsBM image/bmpsftyp)sheicsheixshevcshevxsmif1smsf1sheimsheisz image/heic) startswithlenr s r_sniff_mime_from_bytesrHs t ~~*++{ ~~o&&| 2A2w((({ 3xx2~~#bqb'W,,QrTg1E1E| ~~e{ 3xx2~~#ac(g--#ad)@33| 4rpathrOptional[bytes]c|t|}|r|Stjt|\}}|r|dr|S|j}ddddddd|dS) zReturn image MIME type for *path*. If *raw* bytes are provided, magic-byte sniffing wins (authoritative). Otherwise we fall back to ``mimetypes`` then suffix-based defaults. Nzimage/r=r<r?rCrD)z.jpgz.jpegz.pngz.gifz.webpz.bmp)rH mimetypes guess_typerrEsuffixrr$)rIr sniffedmime_rNs r _guess_mimerRs  (--  N"3t99--GD! )) [   F    c&, rc |}n4#t$r'}td||Yd}~dSd}~wwxYwt ||}t j|d}d|d|S)u!Encode a local image as a base64 data URL at its native size. Size limits are NOT enforced here — the agent retry loop (``run_agent._try_shrink_image_parts_in_messages``) shrinks on the provider's first rejection. Keeping this simple means providers that accept large images (OpenAI 49 MB+, Gemini 100 MB) don't pay a silent quality tax just because one other provider is stricter. Returns None only if the file can't be read (missing, permission denied, etc.); the caller reports those paths in ``skipped``. u'image_routing: failed to read %s — %sNrGasciizdata:z;base64,) read_bytesr+r,warningrRbase64 b64encodedecode)rIr r0rPb64s r_file_to_data_urlr[soo @$LLLttttt t % % %D  3   & &w / /C &4 & & & &&s AAA user_text image_paths List[str]&Tuple[List[Dict[str, Any]], List[str]]cg}g}g}|D]}t|}|r|s#|t |\t |}|s#|t ||dd|id|t ||pd}|rI|pd} dd|D} | d| } d | d g} | || |fSg} |r| d |d | |fS) uBuild an OpenAI-style ``content`` list for a user turn. Shape: [{"type": "text", "text": "...\n\n[Image attached at: /local/path]"}, {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}, ...] The local path of each successfully attached image is appended to the text part as ``[Image attached at: ]``. The model still sees the pixels via the ``image_url`` part (full native vision); the path note just gives it a string handle so MCP/skill tools that take an image path or URL argument can be invoked on the same image without an extra round-trip. This parallels the text-mode hint produced by ``Runner._enrich_message_with_vision`` (``vision_analyze using image_url: ``) so behaviour is consistent across both image input modes. Images are attached at their native size. If a provider rejects the request because an image is too large (e.g. Anthropic's 5 MB per-image ceiling), the agent's retry loop transparently shrinks and retries once — see ``run_agent._try_shrink_image_parts_in_messages``. Returns (content_parts, skipped_paths). Skipped paths are files that couldn't be read from disk and are NOT advertised in the path hints. image_urlurl)typerar zWhat do you see in this image? c3"K|] }d|dV dS)z[Image attached at: ]N).0ps r z-build_native_content_parts..s?  ,- '1 ' ' '      rz r )rcr ) rexistsis_fileappendrr[rjoinextend) r\r]skipped image_partsattached_pathsraw_pathridata_urlr base_text path_hints combined_textpartss rbuild_native_content_partsrys8G(*K "N - - NNxxzz   NN3x== ) ) ) $Q''  NN3x== ) ) ) *      c(mm,,,, O " " $ $D<< YY  1?     %66*66 06 'N'N&O [!!!g~ E 5 fd33444 '>rr8ry)r rrr)rrrr)rrr!rrr')rrr!rrrrr)r r9rr:)N)rIrr rJrr)rIrrr:)r\rr]r^rr_)__doc__ __future__rrWloggingrLpathlibrtypingrrrrr getLogger__name__r, frozensetrrr&r1r8rHrRr[ry__all__rgrrrsg  D#""""" 33333333333333  8 $ $y33344 2 & & & &`    F     4'''',AAAAJ  r