> ## Documentation Index > Fetch the complete documentation index at: https://docs.perceptron.inc/llms.txt > Use this file to discover all available pages before exploring further. # Video tracking > Follow objects through video with timestamped points, boxes, and track tags. Perceptron Mk1.5 can follow an object through a video and return its positions in a ``. Each observation combines a timestamp with spatial geometry. Use this for object trajectories, motion review, or video overlays. Tracking uses the chat completions endpoint. Request the object and track format in your prompt, and use `vision_config.annotation_format: "box"` for bounding boxes. The response contains markup in `message.content`. ## Track an object Install `perceptron>=0.4.0` and set `PERCEPTRON_API_KEY`. This example uses the public basketball video from the Perceptron cookbook. ```python theme={null} import os from perceptron import Client, video client = Client(api_key=os.environ["PERCEPTRON_API_KEY"]) video_url = ( "https://raw.githubusercontent.com/perceptron-ai-inc/perceptron/" "main/cookbook/_shared/assets/capabilities/video-clipping/mj_shot_short.mp4" ) request = { "model": "perceptron-mk1.5", "reasoning_effort": "high", "messages": [ { "role": "user", "content": [ video(video_url), { "type": "text", "text": ( "Track the basketball while it is visible. " "Return a track with mention=\"basketball\" and asset_idx=\"0\". " "Use point_box children with timestamps in seconds and " "coordinates normalized to 0–1000. " "Do not invent positions when the ball is occluded or off-screen." ), }, ], } ], "max_completion_tokens": 4096, "vision_config": {"annotation_format": "box"}, } response = client.chat.completions.create(**request) choice = response.choices[0] if choice.finish_reason != "stop": raise RuntimeError(f"Incomplete tracking answer: {choice.finish_reason}") print(choice.message.content or "") ``` An illustrative track looks like this. These coordinates and timestamps explain the format; they are not recorded output for the sample video. ```html theme={null} (410,520) (450,570) (470,300) (510,350) (580,180) (620,230) ``` When present, `asset_idx="0"` selects the first media asset. It is optional: the model can omit it from the track even though this prompt requests it. Without an explicit or inherited selector, the track refers to the last asset available when it was produced, which is video `0` in this request. For several videos, request explicit selectors and apply the same [last-asset default](/perceptron-mk1.5/guides/multiple-assets#resolve-selectors-before-drawing) when a selector is missing. Each `t` is a time within the selected video; each box gives top-left and bottom-right coordinates. For several objects, ask for separate tracks and describe how to distinguish them. See the [annotation reference](/perceptron-mk1.5/concepts/annotations) for the full grammar. Avoid passing the answer through an ordinary HTML parser: HTML treats `track` as a void element and can discard its grouping. ## Stream a tracking answer SSE chunks are transport boundaries. A chunk may end inside a tag, attribute, timestamp, or coordinate. The simplest reliable consumer accumulates text and validates the finish reason before parsing it. Run this after defining `client` and `request` above: ```python theme={null} with client.chat.completions.create( **request, stream=True, stream_options={"include_usage": True}, ) as stream: for chunk in stream: for choice in chunk.choices: if choice.delta is not None and choice.delta.content: print(choice.delta.content, end="", flush=True) final = stream.get_final_completion() if not final.complete or final.finish_reason != "stop" or final.tool_calls: raise RuntimeError(f"Tracking did not complete: {final.finish_reason}") text = final.text or "" print() print(final.usage) ``` The SDK accumulates the text and usage in `final`, including usage from a trailing chunk with an empty `choices` array. Network and API errors can interrupt the loop; allow them to fail the operation rather than treating the accumulated prefix as a complete answer. A `"length"` finish reason means the output budget was exhausted, even if some waypoints are usable. For live overlays, an incremental annotation parser can emit fully closed child observations while retaining the surrounding track context. Keep these provisional results separate from the final answer. Never append closing tags to make truncated output look complete; retain the completed observations and mark the track incomplete. ## Parse the tracks The SDK parses annotation markup without HTML's special treatment of ``. Run this after the non-streaming request above, or use `final` in place of `response` after streaming: ```python theme={null} annotations = response.annotations(strict=True) for track in annotations.tracks: if not track.complete: raise ValueError("The track markup is incomplete") print(track.mention, len(track.points)) # Flattened boxes carry any selector inherited from their track or collection. for waypoint in annotations.boxes: asset_idx = response.resolve_asset_idx(waypoint) if asset_idx != 0: raise ValueError("Expected a waypoint on the supplied video") print(asset_idx, waypoint.t, waypoint.top_left, waypoint.bottom_right) ``` `strict=True` raises reported parse errors, but it is not a complete schema validator. Retain the raw text and validate annotation syntax and selectors before using parsed results; see [rendering annotations](/perceptron-mk1.5/guides/rendering-annotations). `resolve_asset_idx()` preserves an explicit `0`, applies inherited selectors, and uses the last asset in this response's request when no selector is present. Keep the response associated with that request's media; later turns do not change its asset indices. Validate normalized coordinate ranges, box ordering, and timestamps against the source video before drawing. Parsing does not establish whether an observation is correct or within the video's duration. ## Turn waypoints into an overlay The model supplies observations, which may be sparse. To render them: 1. Resolve an explicit or inherited selector, or default to the last asset available when the track was produced. Confirm the selected asset is a video and use its displayed dimensions. 2. Validate the timestamps and normalized geometry, then sort observations by time. 3. Draw at observed timestamps. If needed, interpolate between nearby observations of the same object. 4. Stop or mark the overlay uncertain across long gaps, occlusion, or a scene cut. Interpolation is a client decision, not another model observation. Linear interpolation can work for short intervals with smooth motion, but it cannot reconstruct a bounce, a sudden turn, or an object passing behind an obstacle. Avoid extending a track before its first observation or after its last one without additional evidence. For denser overlays, follow [High Fidelity Object Tracking](/perceptron-mk1.5/guides/high-fidelity-object-tracking) to combine model waypoints with optical flow and an appearance tracker. Keep model observations and locally estimated positions distinguishable in downstream analysis. ## Improve tracking prompts * Identify the target with visible properties and an initial location, such as “the red car in the left lane at the start.” * State whether you want the whole object, a component, or its center. Choose `box` or `point` to match that task. * Ask for positions only while the object is visible. A missing observation is preferable to an invented trajectory. * Narrow the video interval or number of requested objects if the answer repeatedly reaches the output limit. * Keep timestamps in seconds when authoring annotations. Input `video_frames.timestamp_ms` uses milliseconds; convert explicitly when connecting the two. For event descriptions and temporal clips, see [video understanding](/perceptron-mk1.5/capabilities/video-understanding). For videos mixed with reference images, see [multiple assets](/perceptron-mk1.5/guides/multiple-assets).