In-context learning (video)

Use a reference image to show Perceptron Mk1.5 what to look for in a video. The request combines the example, an explanation of which features matter, and a query video. This can help with product matching, finding a familiar object, or reviewing footage for a visually defined condition. The example guides the current request. Keep it in the supplied conversation when asking follow-up questions that depend on it.

Find a reference product on a shelf

This example supplies an image of a cereal package, then asks whether a matching package is visible in a shelf video. It asks for temporal evidence and permits an uncertain answer when packaging details cannot be read clearly. Visibility in this recording does not establish a store’s current inventory. Install the perceptron>=0.4.0 Python package and set PERCEPTRON_API_KEY before running:

import os

from perceptron import Client, image, video

client = Client(api_key=os.environ["PERCEPTRON_API_KEY"])
asset_base = (
    "https://raw.githubusercontent.com/perceptron-ai-inc/perceptron/"
    "main/cookbook/_shared/assets/capabilities/in-context-learning-video"
)
response = client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=[{
        "role": "user",
        "content": [
            image(f"{asset_base}/mini_wheats.jpeg"),
            {"type": "text", "text": (
                "Asset 0 is the reference product. Look for a matching cereal "
                "package using its visible branding, product name, and package "
                "design. Similar colors alone do not establish a match."
            )},
            video(f"{asset_base}/cereal_short.mp4"),
            {"type": "text", "text": (
                "Asset 1 is the shelf video to inspect. Is a package matching "
                "the reference visible? Explain the visible evidence and cite "
                "the matching interval or intervals with clip annotations, "
                "asset_idx=1, and timestamps in explicit seconds. "
                "Only cite intervals in the video, not the reference image. "
                "If no match is visible, say that. If resolution or occlusion "
                "prevents a confident comparison, explain the uncertainty "
                "instead of claiming the product is absent."
            )},
        ],
    }],
    reasoning_effort="high",
    max_completion_tokens=2048,
    vision_config={"annotation_format": "clip"},
)
choice = response.choices[0]
if choice.finish_reason != "stop":
    raise RuntimeError(f"Incomplete comparison answer: {choice.finish_reason}")
print(choice.message.content or "")

The reference image is asset 0; the query video is asset 1. Temporal annotations must therefore select asset 1. This is an illustrative matching interval, not measured output for the sample video:

<clip mention="matching reference package visible" asset_idx="1" t="2.0 seconds 4.5 seconds" />

Validate the selected asset before using a timestamp in a player. If you add more references or retain earlier media in the conversation, update the selectors according to multiple-asset ordering.

Make the comparison precise

State whether you want the same class, the same product design, or the same individual object. Those are different matching tasks.
Use a reference where the distinguishing features are visible. If it contains several objects, identify the intended one with a reviewed box, as in image in-context learning.
For several reference concepts, label each one and request separate evidence for each. Do not assume every reference has a match in the query video.
Treat a missed or obscured match as uncertainty about the supplied footage. Selected frames do not show every instant of the source video.

Choose the output you need

Request clip annotations when the result should seek to supporting evidence. Request video tracks when you need the matching object’s position over time; the track belongs to the video asset, not the reference image. If a track has an uncertain interval, request new model observations around the gap before continuing local tracking. Keep crop coordinates and source timestamps associated with their own assets when combining the results.

Get Started

Capabilities

Developer Guides

Scaling & deployment

Best practices

In-context learning (video)

Find a reference product on a shelf

Make the comparison precise

Choose the output you need

​Find a reference product on a shelf

​Make the comparison precise

​Choose the output you need

Find a reference product on a shelf

Make the comparison precise

Choose the output you need