Video understanding

Perceptron Mk1.5 can describe actions, answer questions about a clip, and identify when events happen. Send a video_url content part, or upload a video through the Files API. Explore the dedicated guides for video Q&A, moments and intervals, reference-image video search, and object tracking.

Ask about a video

Install perceptron>=0.4.0 and set PERCEPTRON_API_KEY. The video() helper accepts a URL, local file, or uploaded file reference.

import os

from perceptron import Client, video

client = Client(api_key=os.environ["PERCEPTRON_API_KEY"])
video_url = (
    "https://raw.githubusercontent.com/perceptron-ai-inc/perceptron/"
    "main/cookbook/_shared/assets/tutorials/isaac_frame_by_frame/surf.mp4"
)
messages = [{
    "role": "user",
    "content": [
        video(video_url),
        {"type": "text", "text": "Describe the surfer's actions in chronological order."},
    ],
}]

response = client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=messages,
    reasoning_effort="high",
    max_completion_tokens=2048,
)
print(response.choices[0].message.content)

For long or visually complex footage, narrow the question to one event or supply the relevant excerpt. Frame sampling can miss brief events; avoid treating a missing observation as proof that an event never occurred.

Analyze video soundtracks

Videos are processed as frames only by default. Set vision_config.enable_audio_in_video: true when speech, sound effects, or other audio matters to your question. Run this after creating client above; this basketball sample contains an audio track:

soundtrack_video_url = (
    "https://raw.githubusercontent.com/perceptron-ai-inc/perceptron/"
    "main/cookbook/_shared/assets/capabilities/video-clipping/mj_shot_short.mp4"
)
response = client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=[{
        "role": "user",
        "content": [
            video(soundtrack_video_url),
            {"type": "text", "text": (
                "Describe the visible action and the audible speech or sounds. "
                "Distinguish what you see from what you hear, and say when speech is unclear."
            )},
        ],
    }],
    reasoning_effort="high",
    max_completion_tokens=2048,
    vision_config={"enable_audio_in_video": True},
)
choice = response.choices[0]
if choice.finish_reason != "stop":
    raise RuntimeError(f"Incomplete audiovisual answer: {choice.finish_reason}")
print(choice.message.content or "")

The flag applies to every byte-backed video in the request, including remote or data video_url parts and uploaded video_file_id parts. When omitted or set to false, the soundtrack is not processed. It does not enable or disable standalone audio inputs. Preselected video_frames contain no soundtrack, so the flag has no effect on those frames. A video with no audio stream, or no soundtrack overlapping the sampled video window, is processed as frames only. A video with an encoded silent audio track still consumes audio tokens when enable_audio_in_video is explicitly set to true. An audio decode failure or audio-token limit violation still returns an error; enabling sound does not silently discard those failures. Audio uses approximately 750 encoder tokens per minute, plus timestamp tokens in the prompt. The per-item audio limit is 16,384 encoder tokens, roughly 21.8 minutes; video frames, other inputs, and the requested output also need to fit the shared context budget. Check usage.prompt_tokens_details.audio_tokens for the audio encoder contribution when returned. See Audio and tokenization for formats and budgeting.

Locate an event in time

Use annotation_format: "clip" when your application needs a moment or interval. Run this after the setup above:

response = client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=[{
        "role": "user",
        "content": [
            video(video_url),
            {"type": "text", "text": (
                "Find the intervals where the surfer is riding a wave. "
                "Return clip annotations with timestamps in seconds."
            )},
        ],
    }],
    reasoning_effort="high",
    max_completion_tokens=2048,
    vision_config={"annotation_format": "clip"},
)
choice = response.choices[0]
if choice.finish_reason != "stop":
    raise RuntimeError(f"Incomplete clipping answer: {choice.finish_reason}")
print(choice.message.content or "")

An illustrative interval looks like:

<clip mention="riding a wave" asset_idx="0" t="1.0 seconds 4.5 seconds" />

The example values explain the format; they are not measured output for this video. The first time is the start and the second is the end. A single timestamp identifies a moment. See annotation format.

Choose a video workflow

Goal	Workflow
Understand speech or sounds alongside the scene	Enable video soundtracks.
Find when something happens	Request `clip` annotations.
Follow an object’s position	Request video tracks with timed spatial annotations.
Produce a denser object track	Combine model waypoints with optical flow and appearance tracking in High Fidelity Object Tracking.
Compare videos or use reference images	Send multiple assets in one conversation.
Ask several independent questions	Use Multilook to share the context.

Send timestamped frames

Use video_frames when you have already selected frames from a video. Provide at least two frames with the same image dimensions, each with an image_url and a non-negative integer timestamp_ms. Keep frames in timestamp order; equal timestamps are allowed. URLs can be HTTP(S) URLs or supported image data URLs. The SDK’s video_frames() helper accepts (image, timestamp_ms) pairs and encodes local frames for you. Save frames from 0 and 1.5 seconds of your clip as frame-000.png and frame-1500.png; change the filenames and timestamps to match your selected frames. Run this after creating client above:

from perceptron import video_frames

response = client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=[{
        "role": "user",
        "content": [
            video_frames([
                ("frame-000.png", 0),
                ("frame-1500.png", 1500),
            ]),
            {"type": "text", "text": (
                "Describe the change visible between these frames. "
                "Use clip annotations with timestamps in seconds."
            )},
        ],
    }],
    vision_config={"annotation_format": "clip"},
    max_completion_tokens=1024,
)
if response.finish_reason != "stop":
    raise RuntimeError(f"Incomplete frame comparison: {response.finish_reason}")
print(response.text or "")

The entire video_frames group is one asset for asset_idx. Each supplied frame consumes one media unit, so this example uses one asset index and two of the request’s 256 media units. Sending those frames as separate image_url content parts would instead create two image assets. See multiple assets. Timestamps are preserved, not automatically rebased: 1500 milliseconds corresponds to 1.5 seconds in annotations. If these frames came from an excerpt starting at 60 seconds and you supplied excerpt-relative timestamps 0 and 1500, add the 60-second offset when mapping results back to the source. If you supplied source-relative timestamps 60000 and 61500, the timeline is already 60 and 61.5 seconds; do not add the offset again. Keep this mapping with your media, and check finish_reason before using the answer. Selected frames provide sparse observations; they do not establish what happened between them. See recover a gap with new model observations when you need more observations in an uncertain tracking interval.

Get Started

Capabilities

Developer Guides

Scaling & deployment

Best practices

Ask about a video

Analyze video soundtracks

Locate an event in time

Choose a video workflow

Send timestamped frames

​Ask about a video

​Analyze video soundtracks

​Locate an event in time

​Choose a video workflow

​Send timestamped frames

Ask about a video

Analyze video soundtracks

Locate an event in time

Choose a video workflow

Send timestamped frames