> ## Documentation Index > Fetch the complete documentation index at: https://docs.perceptron.inc/llms.txt > Use this file to discover all available pages before exploring further. # Annotation format > Read spatial and temporal grounding, including tracks and asset selectors. Perceptron Mk1.5 can ground an answer with points, boxes, polygons, clips, and tracks. These annotations appear as markup inside `choices[0].message.content`, alongside any prose. Streaming responses deliver the same text through `choices[0].delta.content`. Use `vision_config.annotation_format` to request `"point"`, `"box"`, `"polygon"`, or `"clip"` annotations. For video tracking with bounding boxes, choose `"box"` and ask for tracks in the prompt. There is no `"track"` annotation-format value. ## Spatial annotations ```html theme={null} (120,200) (100,150) (300,350) (20,40) (60,40) (60,80) (20,80) ``` | Tag | Body | | ----------- | ---------------------------------------------------- | | `point` | One `(x,y)` coordinate. | | `point_box` | Two coordinates: top-left, then bottom-right. | | `polygon` | An ordered sequence of vertices around the boundary. | Coordinates use a normalized 0–1000 grid. The origin is the top-left corner; x increases to the right and y increases downward. `(500,500)` is the center, independently of image dimensions. Convert to pixels using the dimensions of the selected image or video frame: ```python theme={null} def to_pixels(x, y, width, height): return x * width / 1000, y * height / 1000 ``` A coordinate at `1000` denotes the image boundary. If your drawing library requires a pixel index, clamp the final index to `width - 1` or `height - 1`. Keep the normalized values for storage and later resizing. The [coordinate system guide](/perceptron-mk1.5/concepts/coordinates) covers per-asset dimensions and crop transforms. See [Rendering annotations](/perceptron-mk1.5/guides/rendering-annotations) for a complete flat-box drawing example. ## Attributes | Attribute | Meaning | | ----------- | ------------------------------------------------------------------------------------------------------------------ | | `mention` | A human-readable description of the object or event. It is not a persistent object ID. | | `asset_idx` | Optional zero-based position of a media occurrence among the assets supplied up to that point in the conversation. | | `t` | Time within a temporal asset. Author timestamps with explicit units, such as `"1.5 seconds"`. | Quote attribute values. `asset_idx` answers **which asset**; `t` answers **when within that asset**. An image selector is never a timestamp. `asset_idx` is optional on annotations, collections, and tracks. The model may omit it even when several assets are present or the prompt requests it. Its absence does not make an annotation malformed. For example, a box can appear without a selector: ```html theme={null} (100,150) (300,350) ``` Resolve inherited selectors first. If `asset_idx` is neither explicit nor inherited, the intended asset is **the last asset available when that annotation was produced**. With `N` assets available at that point, the default index is `N - 1`: `0` for one asset, `1` for two assets, and so on. Count supplied media in earlier messages and tool results; later media does not change an earlier annotation's binding. Requesting explicit selectors helps distinguish multiple inputs but does not guarantee their presence. See [multiple assets](/perceptron-mk1.5/guides/multiple-assets) for ordering and resolution rules. ## Collections A collection groups related annotations. Its `mention` describes the group. Children inherit its `asset_idx` unless they specify their own selector; each child can also carry its own `mention`. ```html theme={null} (100,150) (300,350) (500,150) (700,350) ``` Collections can contain different geometry types, clips, and tracks, but cannot contain other collections. They can also group annotations from different assets: put selectors on the children, or override an inherited selector explicitly. ```html theme={null} (100,150) (300,350) (400,300) (600,500) ``` ## Clips A clip identifies a moment or a time interval in a video or audio recording. It is self-closing and has no coordinate body. See [video clipping](/perceptron-mk1.5/capabilities/video-clipping) or [audio clipping](/perceptron-mk1.5/capabilities/audio-clipping) for complete request examples. ```html theme={null} ``` For an interval, the first timestamp is the start and the second is the end. These examples illustrate the format; the values are not measured predictions. ## Tracks A track groups observations of one object within one video asset. The track can carry an object description and optional asset selector; each child provides a timestamp and geometry. Children inherit the track's selector when they do not specify one, and a track can inherit a selector from its enclosing collection. ```html theme={null} (100,150) (180,230) (120,150) (200,230) ``` Use separate tracks for separate objects or videos, even when their descriptions match. A collection can group these tracks. A track must not switch assets between observations. Every track child must have a timestamp and use the same geometry type: all `point`, all `point_box`, or all `polygon`. Tracks cannot contain clips, collections, or other tracks. Track waypoints can be sparse. A track does not promise a box for every source frame, continuous visibility, or a stable identity across separate requests. See [video tracking](/perceptron-mk1.5/capabilities/video-tracking) for a request example and [High Fidelity Object Tracking](/perceptron-mk1.5/guides/high-fidelity-object-tracking) for estimating positions between model waypoints with optical flow and an appearance tracker. ## Parse and validate annotations Treat annotations as a model-output format rather than ordinary HTML. In HTML, `track` is a void element: a browser HTML parser may move its children outside the track and lose their grouping. The response can also contain prose, so the entire answer is not an XML document. A dedicated parser should recognize the supported annotation tags, retain their nesting, and resolve inherited asset selectors before producing application objects. Keep any IDs, pixel coordinates, and interpolation metadata you add separate from the returned markup. Validate parsed results before using them: * Resolve explicit and inherited `asset_idx` values against the assets available at that annotation's position; otherwise use the last available asset's index. Preserve `0`; test for a missing value instead of using a truthiness check. Keep omitted values distinct from invalid or out-of-range selectors. * Check finite coordinates, box ordering, and the expected 0–1000 range. Choose an explicit policy for invalid geometry rather than silently placing it on another asset. * Check that track timestamps belong to the selected video and increase in the order you use for playback. * Accept explicit second units and bare numeric `t` values in generated output. When authoring annotations in prompts or history, use explicit seconds to avoid ambiguity with frame indices. * Preserve completed observations separately from a partial tag or unfinished track. A connection ending does not make partial markup complete. Annotation markup is different from a JSON Schema response and from function-call arguments. See [structured outputs](/perceptron-mk1.5/capabilities/structured-outputs) to choose the appropriate format.