Video tracking

Perceptron Mk1.5 can follow an object through a video and return its positions in a <track>. Each observation combines a timestamp with spatial geometry. Use this for object trajectories, motion review, or video overlays. Tracking uses the chat completions endpoint. Request the object and track format in your prompt, and use vision_config.annotation_format: "box" for bounding boxes. The response contains markup in message.content.

Track an object

Install perceptron>=0.4.0 and set PERCEPTRON_API_KEY. This example uses the public basketball video from the Perceptron cookbook.

import os

from perceptron import Client, video

client = Client(api_key=os.environ["PERCEPTRON_API_KEY"])
video_url = (
    "https://raw.githubusercontent.com/perceptron-ai-inc/perceptron/"
    "main/cookbook/_shared/assets/capabilities/video-clipping/mj_shot_short.mp4"
)
request = {
    "model": "perceptron-mk1.5",
    "reasoning_effort": "high",
    "messages": [
        {
            "role": "user",
            "content": [
                video(video_url),
                {
                    "type": "text",
                    "text": (
                        "Track the basketball while it is visible. "
                        "Return a track with mention=\"basketball\" and asset_idx=\"0\". "
                        "Use point_box children with timestamps in seconds and "
                        "coordinates normalized to 0–1000. "
                        "Do not invent positions when the ball is occluded or off-screen."
                    ),
                },
            ],
        }
    ],
    "max_completion_tokens": 4096,
    "vision_config": {"annotation_format": "box"},
}

response = client.chat.completions.create(**request)
choice = response.choices[0]
if choice.finish_reason != "stop":
    raise RuntimeError(f"Incomplete tracking answer: {choice.finish_reason}")
print(choice.message.content or "")

An illustrative track looks like this. These coordinates and timestamps explain the format; they are not recorded output for the sample video.

<track mention="basketball" asset_idx="0">
  <point_box t="0.5 seconds"> (410,520) (450,570) </point_box>
  <point_box t="1.0 seconds"> (470,300) (510,350) </point_box>
  <point_box t="1.5 seconds"> (580,180) (620,230) </point_box>
</track>

When present, asset_idx="0" selects the first media asset. It is optional: the model can omit it from the track even though this prompt requests it. Without an explicit or inherited selector, the track refers to the last asset available when it was produced, which is video 0 in this request. For several videos, request explicit selectors and apply the same last-asset default when a selector is missing. Each t is a time within the selected video; each box gives top-left and bottom-right coordinates. For several objects, ask for separate tracks and describe how to distinguish them. See the annotation reference for the full grammar. Avoid passing the answer through an ordinary HTML parser: HTML treats track as a void element and can discard its grouping.

Stream a tracking answer

SSE chunks are transport boundaries. A chunk may end inside a tag, attribute, timestamp, or coordinate. The simplest reliable consumer accumulates text and validates the finish reason before parsing it. Run this after defining client and request above:

with client.chat.completions.create(
    **request,
    stream=True,
    stream_options={"include_usage": True},
) as stream:
    for chunk in stream:
        for choice in chunk.choices:
            if choice.delta is not None and choice.delta.content:
                print(choice.delta.content, end="", flush=True)
    final = stream.get_final_completion()

if not final.complete or final.finish_reason != "stop" or final.tool_calls:
    raise RuntimeError(f"Tracking did not complete: {final.finish_reason}")

text = final.text or ""
print()
print(final.usage)

The SDK accumulates the text and usage in final, including usage from a trailing chunk with an empty choices array. Network and API errors can interrupt the loop; allow them to fail the operation rather than treating the accumulated prefix as a complete answer. A "length" finish reason means the output budget was exhausted, even if some waypoints are usable. For live overlays, an incremental annotation parser can emit fully closed child observations while retaining the surrounding track context. Keep these provisional results separate from the final answer. Never append closing tags to make truncated output look complete; retain the completed observations and mark the track incomplete.

Parse the tracks

The SDK parses annotation markup without HTML’s special treatment of <track>. Run this after the non-streaming request above, or use final in place of response after streaming:

annotations = response.annotations(strict=True)
for track in annotations.tracks:
    if not track.complete:
        raise ValueError("The track markup is incomplete")
    print(track.mention, len(track.points))

# Flattened boxes carry any selector inherited from their track or collection.
for waypoint in annotations.boxes:
    asset_idx = response.resolve_asset_idx(waypoint)
    if asset_idx != 0:
        raise ValueError("Expected a waypoint on the supplied video")
    print(asset_idx, waypoint.t, waypoint.top_left, waypoint.bottom_right)

strict=True raises reported parse errors, but it is not a complete schema validator. Retain the raw text and validate annotation syntax and selectors before using parsed results; see rendering annotations. resolve_asset_idx() preserves an explicit 0, applies inherited selectors, and uses the last asset in this response’s request when no selector is present. Keep the response associated with that request’s media; later turns do not change its asset indices. Validate normalized coordinate ranges, box ordering, and timestamps against the source video before drawing. Parsing does not establish whether an observation is correct or within the video’s duration.

Turn waypoints into an overlay

The model supplies observations, which may be sparse. To render them:

Resolve an explicit or inherited selector, or default to the last asset available when the track was produced. Confirm the selected asset is a video and use its displayed dimensions.
Validate the timestamps and normalized geometry, then sort observations by time.
Draw at observed timestamps. If needed, interpolate between nearby observations of the same object.
Stop or mark the overlay uncertain across long gaps, occlusion, or a scene cut.

Interpolation is a client decision, not another model observation. Linear interpolation can work for short intervals with smooth motion, but it cannot reconstruct a bounce, a sudden turn, or an object passing behind an obstacle. Avoid extending a track before its first observation or after its last one without additional evidence. For denser overlays, follow High Fidelity Object Tracking to combine model waypoints with optical flow and an appearance tracker. Keep model observations and locally estimated positions distinguishable in downstream analysis.

Improve tracking prompts

Identify the target with visible properties and an initial location, such as “the red car in the left lane at the start.”
State whether you want the whole object, a component, or its center. Choose box or point to match that task.
Ask for positions only while the object is visible. A missing observation is preferable to an invented trajectory.
Narrow the video interval or number of requested objects if the answer repeatedly reaches the output limit.
Keep timestamps in seconds when authoring annotations. Input video_frames.timestamp_ms uses milliseconds; convert explicitly when connecting the two.

For event descriptions and temporal clips, see video understanding. For videos mixed with reference images, see multiple assets.

Get Started

Capabilities

Developer Guides

Scaling & deployment

Best practices

Track an object

Stream a tracking answer

Parse the tracks

Turn waypoints into an overlay

Improve tracking prompts

​Track an object

​Stream a tracking answer

​Parse the tracks

​Turn waypoints into an overlay

​Improve tracking prompts

Track an object

Stream a tracking answer

Parse the tracks

Turn waypoints into an overlay

Improve tracking prompts