Skip to main content
Perceptron Mk1.5 can ground an answer with points, boxes, polygons, clips, and tracks. These annotations appear as markup inside choices[0].message.content, alongside any prose. Streaming responses deliver the same text through choices[0].delta.content. Use vision_config.annotation_format to request "point", "box", "polygon", or "clip" annotations. For video tracking with bounding boxes, choose "box" and ask for tracks in the prompt. There is no "track" annotation-format value.

​
Spatial annotations

Coordinates use a normalized 0–1000 grid. The origin is the top-left corner; x increases to the right and y increases downward. (500,500) is the center, independently of image dimensions. Convert to pixels using the dimensions of the selected image or video frame:
A coordinate at 1000 denotes the image boundary. If your drawing library requires a pixel index, clamp the final index to width - 1 or height - 1. Keep the normalized values for storage and later resizing. The coordinate system guide covers per-asset dimensions and crop transforms. See Rendering annotations for a complete flat-box drawing example.

​
Attributes

Quote attribute values. asset_idx answers which asset; t answers when within that asset. An image selector is never a timestamp. asset_idx is optional on annotations, collections, and tracks. The model may omit it even when several assets are present or the prompt requests it. Its absence does not make an annotation malformed. For example, a box can appear without a selector:
Resolve inherited selectors first. If asset_idx is neither explicit nor inherited, the intended asset is the last asset available when that annotation was produced. With N assets available at that point, the default index is N - 1: 0 for one asset, 1 for two assets, and so on. Count supplied media in earlier messages and tool results; later media does not change an earlier annotation’s binding. Requesting explicit selectors helps distinguish multiple inputs but does not guarantee their presence. See multiple assets for ordering and resolution rules.

​
Collections

A collection groups related annotations. Its mention describes the group. Children inherit its asset_idx unless they specify their own selector; each child can also carry its own mention.
Collections can contain different geometry types, clips, and tracks, but cannot contain other collections. They can also group annotations from different assets: put selectors on the children, or override an inherited selector explicitly.

​
Clips

A clip identifies a moment or a time interval in a video or audio recording. It is self-closing and has no coordinate body. See video clipping or audio clipping for complete request examples.
For an interval, the first timestamp is the start and the second is the end. These examples illustrate the format; the values are not measured predictions.

​
Tracks

A track groups observations of one object within one video asset. The track can carry an object description and optional asset selector; each child provides a timestamp and geometry. Children inherit the track’s selector when they do not specify one, and a track can inherit a selector from its enclosing collection.
Use separate tracks for separate objects or videos, even when their descriptions match. A collection can group these tracks. A track must not switch assets between observations. Every track child must have a timestamp and use the same geometry type: all point, all point_box, or all polygon. Tracks cannot contain clips, collections, or other tracks. Track waypoints can be sparse. A track does not promise a box for every source frame, continuous visibility, or a stable identity across separate requests. See video tracking for a request example and High Fidelity Object Tracking for estimating positions between model waypoints with optical flow and an appearance tracker.

​
Parse and validate annotations

Treat annotations as a model-output format rather than ordinary HTML. In HTML, track is a void element: a browser HTML parser may move its children outside the track and lose their grouping. The response can also contain prose, so the entire answer is not an XML document. A dedicated parser should recognize the supported annotation tags, retain their nesting, and resolve inherited asset selectors before producing application objects. Keep any IDs, pixel coordinates, and interpolation metadata you add separate from the returned markup. Validate parsed results before using them:
  • Resolve explicit and inherited asset_idx values against the assets available at that annotation’s position; otherwise use the last available asset’s index. Preserve 0; test for a missing value instead of using a truthiness check. Keep omitted values distinct from invalid or out-of-range selectors.
  • Check finite coordinates, box ordering, and the expected 0–1000 range. Choose an explicit policy for invalid geometry rather than silently placing it on another asset.
  • Check that track timestamps belong to the selected video and increase in the order you use for playback.
  • Accept explicit second units and bare numeric t values in generated output. When authoring annotations in prompts or history, use explicit seconds to avoid ambiguity with frame indices.
  • Preserve completed observations separately from a partial tag or unfinished track. A connection ending does not make partial markup complete.
Annotation markup is different from a JSON Schema response and from function-call arguments. See structured outputs to choose the appropriate format.