> ## Documentation Index > Fetch the complete documentation index at: https://docs.perceptron.inc/llms.txt > Use this file to discover all available pages before exploring further. # High Fidelity Object Tracking > Turn model waypoints into detailed video tracks with optical flow and appearance tracking. Use Perceptron Mk1.5 to identify an object and return timestamped boxes, then follow the object through the frames between those boxes. This combines the model's understanding of **what to track** with local tracking of **how it moves**. The workflow has two interpolation layers: **optical flow first**, followed by **appearance tracking where flow is weak**. Both use the video's pixels to estimate intermediate positions. A straight line between model boxes cannot capture a bounce or sudden turn. | Stage | What it contributes | | ---------------------------- | -------------------------------------------------------------------------------------- | | Perceptron video request | Sparse `` waypoints that establish the object and anchor boxes. | | Layer 1: optical flow | Motion estimates from visual features between nearby frames. | | Layer 2: appearance tracking | Additional estimates from matching the object's appearance when flow loses confidence. | The two local layers run in your application. They are not extra API parameters or built-in SDK interpolation methods. Start with one object and a short video; extend to multiple tracks once the result is reliable on your footage. ## Get model waypoints Install `perceptron>=0.4.0`, set `PERCEPTRON_API_KEY`, and save a short video containing a basketball as `clip.mp4`. Adapt the target description to your own footage: ```python theme={null} from perceptron import Client, video client = Client(provider="perceptron") response = client.chat.completions.create( model="perceptron-mk1.5", messages=[{ "role": "user", "content": [ video("clip.mp4"), {"type": "text", "text": ( "Track the basketball while it is visible. Return one track with " "mention=\"basketball\" and asset_idx=\"0\". Use point_box children " "with timestamps in seconds and coordinates normalized to 0–1000. " "Include waypoints around changes in direction. Do not invent " "positions when the ball is occluded or off-screen." )}, ], }], vision_config={"annotation_format": "box"}, reasoning_effort="high", max_completion_tokens=4096, ) if not response.complete or response.finish_reason != "stop" or response.tool_calls: raise RuntimeError(f"Incomplete tracking answer: {response.finish_reason}") print(response.text) ``` `video()` encodes a local file inline. For larger or reused videos, [upload once](/perceptron-mk1.5/guides/files) and pass the returned file to `video()`. See [Video tracking](/perceptron-mk1.5/capabilities/video-tracking) for the annotation format and streaming. Treat the model's boxes as anchors. For several objects, keep each `` separate; identical labels do not establish that two tracks are the same object. `asset_idx` is optional: an omitted selector here resolves to the only supplied video, asset `0`. Run this after the request above. It expects one complete box track and produces `anchors`, sorted by time: ```python theme={null} import math from perceptron import BoundingBox tracks = response.annotations(strict=True).tracks if len(tracks) != 1 or not tracks[0].complete: raise ValueError("Expected one complete track; inspect the model answer") anchors = [] for waypoint in tracks[0].points: if not isinstance(waypoint, BoundingBox) or response.resolve_asset_idx(waypoint) != 0: raise ValueError("Expected boxes on the supplied video") t = waypoint.t x1, y1 = waypoint.top_left.x, waypoint.top_left.y x2, y2 = waypoint.bottom_right.x, waypoint.bottom_right.y if t is None or not all(math.isfinite(v) for v in (t, x1, y1, x2, y2)): raise ValueError("Each anchor needs finite coordinates and a timestamp") if t < 0 or not (0 <= x1 < x2 <= 1000 and 0 <= y1 < y2 <= 1000): raise ValueError("Invalid timestamp or normalized box") anchors.append({"t": t, "box": [x1, y1, x2, y2]}) anchors.sort(key=lambda anchor: anchor["t"]) if len(anchors) < 2 or any(a["t"] == b["t"] for a, b in zip(anchors, anchors[1:])): raise ValueError("Interpolation needs at least two distinct anchor times") print(anchors) ``` `strict=True` raises reported parse errors; it is not a complete markup validator. Retain the raw answer and apply the [annotation validation guidance](/perceptron-mk1.5/guides/rendering-annotations). Also check anchor times against the video's duration and inspect whether the boxes follow the intended object before using them. ## Align frames and coordinates Process each pair of adjacent anchors within the same track. Start with short intervals—for example, no more than one second—and split at known occlusions or scene cuts. This interval length is a tuning choice for your application, not an API limit. Decode the actual intervening frames and retain their presentation timestamps. Model timestamps are in seconds; do not derive frame times from a guessed frame rate, especially for variable-frame-rate video. Map each anchor to its decoded frame and skip interpolation if both anchors fall in the same frame. The local trackers operate in pixels. Convert the model's 0–1000 coordinates using the decoded frame's dimensions. For lower processing cost, use one fixed crop covering both anchor boxes with padding, resize that crop consistently for the whole interval, and transform both anchor boxes into its coordinate system. Keep enough surrounding area for the object's path to remain inside the crop. Save the crop and resize transform so you can [map results back to the full frame](/perceptron-mk1.5/concepts/coordinates#map-a-crop-back-to-the-original-image). ## Layer 1: follow local motion Use sparse Lucas–Kanade optical flow as the first pass. Detect feature points inside the starting box, follow them through successive frames, and estimate how their motion changes the box. A practical implementation: 1. Select features inside the object with Shi–Tomasi corner detection. 2. Track those points into the next frame with pyramidal Lucas–Kanade flow. 3. Track them back to the previous frame. Reject points that do not return near their starting position. 4. Fit a robust translation/scale/rotation transform to the surviving points, rejecting outliers. Apply the accumulated transform to the original anchor box corners. 5. Lower confidence when too few points survive, points disagree, or motion and size change implausibly. Run this twice for each interval: forward from the earlier model box and backward from the later one. Reset tracking state at the next anchor pair, so errors do not accumulate through the entire video. OpenCV's [Lucas–Kanade example](https://github.com/opencv/opencv/blob/4.x/samples/python/lk_track.py) shows feature detection and forward/backward verification using `goodFeaturesToTrack()` and `calcOpticalFlowPyrLK()`. You still need to turn feature motion into box estimates and a confidence score. ## Layer 2: recover with appearance If optical flow is missing or weak on any interior frame, run an appearance tracker such as **NanoTrack** over that interval. It matches an object template against nearby frames, providing another source of evidence when feature points are scarce or lost. Initialize one tracker with the earlier model box and another with the later box. Run them in opposite directions, with independent templates and state. Start from the model anchors, rather than using a drifting flow box as the new reference. Keep the endpoints fixed. OpenCV's [TrackerNano](https://docs.opencv.org/4.x/d8/d69/classcv_1_1TrackerNano.html) provides initialization, frame updates, and a tracking score. It requires separate backbone and localization model files, linked from that reference. Tracker scores and thresholds depend on the implementation; calibrate them on representative videos before combining them with optical-flow confidence. Appearance tracking can also drift onto a similar object. A high score alone does not prove identity, and running the second layer does not guarantee every gap can be filled. ## Combine and render the results Use the first and last decoded frame presentation timestamps as `start_time` and `end_time`. At an intermediate frame time `t`, let `u = (t - start_time) / (end_time - start_time)`. Weight the forward prediction by `(1 - u) × forward_confidence` and the backward prediction by `u × backward_confidence`. This favors the nearer anchor while retaining evidence from both directions. Check agreement before accepting a combined box. Intersection over union (IoU) measures the overlap of the two boxes; low overlap should reduce confidence. When predictions agree, blend their centers and sizes. Blending width and height in log space gives a smooth transition in scale. Preserve the original model boxes exactly at the endpoints. Use the same agreement check across the two tracking layers: keep reliable flow, combine agreeing flow and appearance estimates, and accept a conflicting winner only when its confidence is strong enough and clearly higher. Otherwise, leave that frame without a box. Missing reverse evidence should lower confidence too. This **pseudocode** summarizes the orchestration. The tracker and fusion operations are application code, not Perceptron SDK functions: ```text theme={null} for each adjacent anchor pair in one track: if the interval is too long or crosses a known visibility break: keep the anchors and leave the interior unavailable continue frames = decode the interval with its real presentation timestamps flow = track forward and backward from the two anchor boxes if any interior flow estimate is missing or has low confidence: appearance = track forward and backward with independent templates else: appearance = unavailable for each interior frame: combine trustworthy estimates using time, confidence, and box agreement if evidence is ambiguous, store no box for that frame preserve both original anchor boxes ``` Keep each output's timestamp, box, confidence, and origin: `model`, `flow`, `appearance`, or `fused`. These local estimates are not additional model observations. Do not draw a straight-line fallback through rejected frames or extrapolate beyond the first and last anchors. Render using the video's media time and the displayed video's dimensions, including any letterbox offsets. Keep missing intervals missing during playback. This avoids showing a smooth but unsupported path after tracking has failed. ## Recover a gap with new model observations When both layers fail, select a frame near the middle of the uncertain interval and ask Perceptron to locate the object again. A nearby clear reference frame plus its known box can help identify the target. Send the reference and target as separate images, validate the returned target box, and associate it with the target frame's original video timestamp. Those images have their own request-local asset indices; they are not the original video's selector. See [Multiple assets](/perceptron-mk1.5/guides/multiple-assets). If you used crops, map the coordinates back before adding the new anchor. Re-run the shorter intervals around an accepted anchor. If the object is occluded or identity remains uncertain, keep the gap. Start by reviewing a few difficult clips: fast motion, low texture, occlusion, camera cuts, and similar-looking objects. Adjust interval length and confidence gates based on whether boxes stay on the intended object, not merely whether the overlay looks smooth.