High Fidelity Object Tracking

Use Perceptron Mk1.5 to identify an object and return timestamped boxes, then follow the object through the frames between those boxes. This combines the model’s understanding of what to track with local tracking of how it moves. The workflow has two interpolation layers: optical flow first, followed by appearance tracking where flow is weak. Both use the video’s pixels to estimate intermediate positions. A straight line between model boxes cannot capture a bounce or sudden turn.

Stage	What it contributes
Perceptron video request	Sparse `<track>` waypoints that establish the object and anchor boxes.
Layer 1: optical flow	Motion estimates from visual features between nearby frames.
Layer 2: appearance tracking	Additional estimates from matching the object’s appearance when flow loses confidence.

The two local layers run in your application. They are not extra API parameters or built-in SDK interpolation methods. Start with one object and a short video; extend to multiple tracks once the result is reliable on your footage.

Get model waypoints

Install perceptron>=0.4.0, set PERCEPTRON_API_KEY, and save a short video containing a basketball as clip.mp4. Adapt the target description to your own footage:

from perceptron import Client, video

client = Client(provider="perceptron")
response = client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=[{
        "role": "user",
        "content": [
            video("clip.mp4"),
            {"type": "text", "text": (
                "Track the basketball while it is visible. Return one track with "
                "mention=\"basketball\" and asset_idx=\"0\". Use point_box children "
                "with timestamps in seconds and coordinates normalized to 0–1000. "
                "Include waypoints around changes in direction. Do not invent "
                "positions when the ball is occluded or off-screen."
            )},
        ],
    }],
    vision_config={"annotation_format": "box"},
    reasoning_effort="high",
    max_completion_tokens=4096,
)
if not response.complete or response.finish_reason != "stop" or response.tool_calls:
    raise RuntimeError(f"Incomplete tracking answer: {response.finish_reason}")
print(response.text)

video() encodes a local file inline. For larger or reused videos, upload once and pass the returned file to video(). See Video tracking for the annotation format and streaming. Treat the model’s boxes as anchors. For several objects, keep each <track> separate; identical labels do not establish that two tracks are the same object. asset_idx is optional: an omitted selector here resolves to the only supplied video, asset 0.

Extract the anchor boxes with the SDK

Run this after the request above. It expects one complete box track and produces anchors, sorted by time:

import math

from perceptron import BoundingBox

tracks = response.annotations(strict=True).tracks
if len(tracks) != 1 or not tracks[0].complete:
    raise ValueError("Expected one complete track; inspect the model answer")

anchors = []
for waypoint in tracks[0].points:
    if not isinstance(waypoint, BoundingBox) or response.resolve_asset_idx(waypoint) != 0:
        raise ValueError("Expected boxes on the supplied video")
    t = waypoint.t
    x1, y1 = waypoint.top_left.x, waypoint.top_left.y
    x2, y2 = waypoint.bottom_right.x, waypoint.bottom_right.y
    if t is None or not all(math.isfinite(v) for v in (t, x1, y1, x2, y2)):
        raise ValueError("Each anchor needs finite coordinates and a timestamp")
    if t < 0 or not (0 <= x1 < x2 <= 1000 and 0 <= y1 < y2 <= 1000):
        raise ValueError("Invalid timestamp or normalized box")
    anchors.append({"t": t, "box": [x1, y1, x2, y2]})

anchors.sort(key=lambda anchor: anchor["t"])
if len(anchors) < 2 or any(a["t"] == b["t"] for a, b in zip(anchors, anchors[1:])):
    raise ValueError("Interpolation needs at least two distinct anchor times")
print(anchors)

strict=True raises reported parse errors; it is not a complete markup validator. Retain the raw answer and apply the annotation validation guidance. Also check anchor times against the video’s duration and inspect whether the boxes follow the intended object before using them.

Align frames and coordinates

Process each pair of adjacent anchors within the same track. Start with short intervals—for example, no more than one second—and split at known occlusions or scene cuts. This interval length is a tuning choice for your application, not an API limit. Decode the actual intervening frames and retain their presentation timestamps. Model timestamps are in seconds; do not derive frame times from a guessed frame rate, especially for variable-frame-rate video. Map each anchor to its decoded frame and skip interpolation if both anchors fall in the same frame. The local trackers operate in pixels. Convert the model’s 0–1000 coordinates using the decoded frame’s dimensions. For lower processing cost, use one fixed crop covering both anchor boxes with padding, resize that crop consistently for the whole interval, and transform both anchor boxes into its coordinate system. Keep enough surrounding area for the object’s path to remain inside the crop. Save the crop and resize transform so you can map results back to the full frame.

Layer 1: follow local motion

Use sparse Lucas–Kanade optical flow as the first pass. Detect feature points inside the starting box, follow them through successive frames, and estimate how their motion changes the box. A practical implementation:

Select features inside the object with Shi–Tomasi corner detection.
Track those points into the next frame with pyramidal Lucas–Kanade flow.
Track them back to the previous frame. Reject points that do not return near their starting position.
Fit a robust translation/scale/rotation transform to the surviving points, rejecting outliers. Apply the accumulated transform to the original anchor box corners.
Lower confidence when too few points survive, points disagree, or motion and size change implausibly.

Run this twice for each interval: forward from the earlier model box and backward from the later one. Reset tracking state at the next anchor pair, so errors do not accumulate through the entire video. OpenCV’s Lucas–Kanade example shows feature detection and forward/backward verification using goodFeaturesToTrack() and calcOpticalFlowPyrLK(). You still need to turn feature motion into box estimates and a confidence score.

Layer 2: recover with appearance

If optical flow is missing or weak on any interior frame, run an appearance tracker such as NanoTrack over that interval. It matches an object template against nearby frames, providing another source of evidence when feature points are scarce or lost. Initialize one tracker with the earlier model box and another with the later box. Run them in opposite directions, with independent templates and state. Start from the model anchors, rather than using a drifting flow box as the new reference. Keep the endpoints fixed. OpenCV’s TrackerNano provides initialization, frame updates, and a tracking score. It requires separate backbone and localization model files, linked from that reference. Tracker scores and thresholds depend on the implementation; calibrate them on representative videos before combining them with optical-flow confidence. Appearance tracking can also drift onto a similar object. A high score alone does not prove identity, and running the second layer does not guarantee every gap can be filled.

Combine and render the results

Use the first and last decoded frame presentation timestamps as start_time and end_time. At an intermediate frame time t, let u = (t - start_time) / (end_time - start_time). Weight the forward prediction by (1 - u) × forward_confidence and the backward prediction by u × backward_confidence. This favors the nearer anchor while retaining evidence from both directions. Check agreement before accepting a combined box. Intersection over union (IoU) measures the overlap of the two boxes; low overlap should reduce confidence. When predictions agree, blend their centers and sizes. Blending width and height in log space gives a smooth transition in scale. Preserve the original model boxes exactly at the endpoints. Use the same agreement check across the two tracking layers: keep reliable flow, combine agreeing flow and appearance estimates, and accept a conflicting winner only when its confidence is strong enough and clearly higher. Otherwise, leave that frame without a box. Missing reverse evidence should lower confidence too. This pseudocode summarizes the orchestration. The tracker and fusion operations are application code, not Perceptron SDK functions:

for each adjacent anchor pair in one track:
    if the interval is too long or crosses a known visibility break:
        keep the anchors and leave the interior unavailable
        continue

    frames = decode the interval with its real presentation timestamps
    flow = track forward and backward from the two anchor boxes

    if any interior flow estimate is missing or has low confidence:
        appearance = track forward and backward with independent templates
    else:
        appearance = unavailable

    for each interior frame:
        combine trustworthy estimates using time, confidence, and box agreement
        if evidence is ambiguous, store no box for that frame

    preserve both original anchor boxes

Keep each output’s timestamp, box, confidence, and origin: model, flow, appearance, or fused. These local estimates are not additional model observations. Do not draw a straight-line fallback through rejected frames or extrapolate beyond the first and last anchors. Render using the video’s media time and the displayed video’s dimensions, including any letterbox offsets. Keep missing intervals missing during playback. This avoids showing a smooth but unsupported path after tracking has failed.

Recover a gap with new model observations

When both layers fail, select a frame near the middle of the uncertain interval and ask Perceptron to locate the object again. A nearby clear reference frame plus its known box can help identify the target. Send the reference and target as separate images, validate the returned target box, and associate it with the target frame’s original video timestamp. Those images have their own request-local asset indices; they are not the original video’s selector. See Multiple assets. If you used crops, map the coordinates back before adding the new anchor. Re-run the shorter intervals around an accepted anchor. If the object is occluded or identity remains uncertain, keep the gap. Start by reviewing a few difficult clips: fast motion, low texture, occlusion, camera cuts, and similar-looking objects. Adjust interval length and confidence gates based on whether boxes stay on the intended object, not merely whether the overlay looks smooth.

Get Started

Capabilities

Developer Guides

Scaling & deployment

Best practices

High Fidelity Object Tracking

Get model waypoints

Align frames and coordinates

Layer 1: follow local motion

Layer 2: recover with appearance

Combine and render the results

Recover a gap with new model observations

​Get model waypoints

​Align frames and coordinates

​Layer 1: follow local motion

​Layer 2: recover with appearance

​Combine and render the results

​Recover a gap with new model observations

Get model waypoints

Align frames and coordinates

Layer 1: follow local motion

Layer 2: recover with appearance

Combine and render the results

Recover a gap with new model observations