Skip to main content
Send multiple media parts in a conversation to compare scenes, locate the same object in different inputs, or use a reference image to guide a search. The optional asset_idx attribute identifies which input an annotation describes. The model may omit it, including in multi-asset answers and when the prompt asks for it. If no explicit or inherited selector is present, the annotation refers to the last asset available to the model at that point in the conversation.

​
How assets are numbered

Assets are numbered from zero in their order of appearance across the messages supplied in the request, then within each message’s content array. An annotation refers to the media available up to its position in that conversation. Text does not consume an index. Images, videos, and standalone audio share one sequence. Each media occurrence has a position. Repeating the same URL or uploaded file_id in a separate content part introduces another position; file reuse does not make the selector a persistent file identifier. Keep these distinctions in mind:
  • One video_url, video_file_id, or video_frames group is one asset. Individual frames inside that group are addressed by time.
  • One input_audio, audio_url, or audio_file_id part is one asset. For an image followed by a standalone recording, the image is asset 0 and the audio is asset 1. Enabling a video’s soundtrack does not add another asset; its frames and audio belong to the same video.
  • Frames supplied as separate image_url parts are separate image assets, even if they came from the same video.
  • Images returned by functions count in the conversation’s asset order. The original image and an image crop returned by a tool are separate assets with their own coordinate systems.
  • Numbering does not reset at a new user turn. Resend the relevant conversation history when continuing a conversation.
  • Removing or reordering earlier media changes the mapping. Recompute selectors for the actual request instead of reusing an index from a different history.
The request limit is 256 media units. This is a separate count from asset_idx: every frame inside a video_frames group consumes one media unit, although the group has only one asset index.

​
Compare a reference image with a target

Install the Perceptron Python SDK and set PERCEPTRON_API_KEY. This example uses two public sample images and requests a box in the second image.
The response is text containing annotation markup, not a separate JSON array of boxes. An illustrative annotation is:
The coordinates above demonstrate the format; they are not measured output for these sample images.

​
Resolve selectors before drawing

Select the asset first, then use that asset’s dimensions to convert normalized coordinates into pixels. For example, (500,500) is (960,540) on a 1920×1080 image and (320,240) on a 640×480 image. An annotation can inherit its selector from a collection. An explicit child selector overrides the collection’s selector:
Here the first box belongs to asset 0 and the second to asset 1. This override is useful for collections; each <track> must stay within one asset. For tracking the same object in two videos, request two tracks with explicit selectors. Their presence is not guaranteed. Resolve an annotation’s asset in this order:
  1. Use its explicit asset_idx, if present.
  2. Otherwise, use an inherited selector from its collection or track.
  3. If neither is present, use the index of the last asset available when that annotation was produced: number_of_available_assets - 1.
In the reference-plus-target example above, a box without asset_idx refers to target image 1. With only one media asset, the default is 0. Count media from earlier supplied messages and tool results up to that annotation’s position, across all modalities. For example, suppose the user supplies images A (0) and B (1), and the assistant returns a box without a selector. That box refers to B. If a later tool result supplies image C (2), an omitted selector in the next assistant answer refers to C. The earlier box still refers to B; adding C does not change its binding. Store each answer with the asset mapping available when it was produced. Validate explicit and inherited selectors against the asset range available at that annotation’s position. An out-of-range or malformed selector is invalid; do not treat it as omitted or replace it with the default. Preserve explicit 0 by testing for a missing value rather than using a truthiness check. In the SDK, response.annotations(strict=True) applies inherited selectors to parsed annotations. Use response.resolve_asset_idx(annotation) to resolve the remaining omissions using the asset count captured for that answer. Keep that response with its original asset mapping; do not resolve an older annotation using a newer response. Check the selected asset’s modality before drawing geometry. If an image is followed by standalone audio, an omitted selector resolves to the audio, not automatically to the last image. Request an explicit selector for image annotations and handle an incompatible or missing visual target instead of drawing on a different asset.

​
Multilook uses a separate sequence for each prompt

Each Multilook prompt sees the shared context followed by its own content. Asset numbering follows that individual sequence, not the entire batch. Both query images are asset 1 in their respective responses. An omitted selector defaults to the last asset in that prompt’s context-plus-prompt sequence, so it also selects asset 1 in both examples. Keep the response’s prompt_index alongside the asset selector when associating results with inputs. The request-wide media-unit limit still includes the shared context and all prompts. Multilook is for independent prompts. Function calling and tool-result histories belong on /v1/chat/completions; see tool calling.

​
Next steps