> ## Documentation Index
> Fetch the complete documentation index at: https://docs.perceptron.inc/llms.txt
> Use this file to discover all available pages before exploring further.

# Video understanding

> Understand video frames and soundtracks, and identify moments or event intervals.

Perceptron Mk1.5 can describe actions, answer questions about a clip, and identify when events happen. Send a `video_url` content part, or upload a video through the [Files API](/perceptron-mk1.5/guides/files).

Explore the dedicated guides for [video Q\&A](/perceptron-mk1.5/capabilities/video-qa), [moments and intervals](/perceptron-mk1.5/capabilities/video-clipping), [reference-image video search](/perceptron-mk1.5/capabilities/in-context-learning-video), and [object tracking](/perceptron-mk1.5/capabilities/video-tracking).

## Ask about a video

Install `perceptron>=0.4.0` and set `PERCEPTRON_API_KEY`. The `video()` helper accepts a URL, local file, or uploaded file reference.

```python theme={null}
import os

from perceptron import Client, video

client = Client(api_key=os.environ["PERCEPTRON_API_KEY"])
video_url = (
    "https://raw.githubusercontent.com/perceptron-ai-inc/perceptron/"
    "main/cookbook/_shared/assets/tutorials/isaac_frame_by_frame/surf.mp4"
)
messages = [{
    "role": "user",
    "content": [
        video(video_url),
        {"type": "text", "text": "Describe the surfer's actions in chronological order."},
    ],
}]

response = client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=messages,
    reasoning_effort="high",
    max_completion_tokens=2048,
)
print(response.choices[0].message.content)
```

For long or visually complex footage, narrow the question to one event or supply the relevant excerpt. Frame sampling can miss brief events; avoid treating a missing observation as proof that an event never occurred.

## Analyze video soundtracks

Videos are processed as frames only by default. Set **`vision_config.enable_audio_in_video: true`** when speech, sound effects, or other audio matters to your question. Run this after creating `client` above; this basketball sample contains an audio track:

```python theme={null}
soundtrack_video_url = (
    "https://raw.githubusercontent.com/perceptron-ai-inc/perceptron/"
    "main/cookbook/_shared/assets/capabilities/video-clipping/mj_shot_short.mp4"
)
response = client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=[{
        "role": "user",
        "content": [
            video(soundtrack_video_url),
            {"type": "text", "text": (
                "Describe the visible action and the audible speech or sounds. "
                "Distinguish what you see from what you hear, and say when speech is unclear."
            )},
        ],
    }],
    reasoning_effort="high",
    max_completion_tokens=2048,
    vision_config={"enable_audio_in_video": True},
)
choice = response.choices[0]
if choice.finish_reason != "stop":
    raise RuntimeError(f"Incomplete audiovisual answer: {choice.finish_reason}")
print(choice.message.content or "")
```

The flag applies to every byte-backed video in the request, including remote or data `video_url` parts and uploaded `video_file_id` parts. When omitted or set to `false`, the soundtrack is not processed. It does not enable or disable standalone audio inputs. Preselected `video_frames` contain no soundtrack, so the flag has no effect on those frames.

A video with no audio stream, or no soundtrack overlapping the sampled video window, is processed as frames only. A video with an encoded silent audio track still consumes audio tokens when `enable_audio_in_video` is explicitly set to `true`. An audio decode failure or audio-token limit violation still returns an error; enabling sound does not silently discard those failures.

Audio uses approximately **750 encoder tokens per minute**, plus timestamp tokens in the prompt. The per-item audio limit is **16,384 encoder tokens**, roughly **21.8 minutes**; video frames, other inputs, and the requested output also need to fit the shared context budget. Check `usage.prompt_tokens_details.audio_tokens` for the audio encoder contribution when returned. See [Audio](/perceptron-mk1.5/capabilities/audio) and [tokenization](/perceptron-mk1.5/guides/tokenization) for formats and budgeting.

## Locate an event in time

Use `annotation_format: "clip"` when your application needs a moment or interval. Run this after the setup above:

```python theme={null}
response = client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=[{
        "role": "user",
        "content": [
            video(video_url),
            {"type": "text", "text": (
                "Find the intervals where the surfer is riding a wave. "
                "Return clip annotations with timestamps in seconds."
            )},
        ],
    }],
    reasoning_effort="high",
    max_completion_tokens=2048,
    vision_config={"annotation_format": "clip"},
)
choice = response.choices[0]
if choice.finish_reason != "stop":
    raise RuntimeError(f"Incomplete clipping answer: {choice.finish_reason}")
print(choice.message.content or "")
```

An illustrative interval looks like:

```html theme={null}
<clip mention="riding a wave" asset_idx="0" t="1.0 seconds 4.5 seconds" />
```

The example values explain the format; they are not measured output for this video. The first time is the start and the second is the end. A single timestamp identifies a moment. See [annotation format](/perceptron-mk1.5/concepts/annotations#clips).

## Choose a video workflow

| Goal                                            | Workflow                                                                                                                                                      |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Understand speech or sounds alongside the scene | [Enable video soundtracks](#analyze-video-soundtracks).                                                                                                       |
| Find when something happens                     | Request `clip` annotations.                                                                                                                                   |
| Follow an object's position                     | Request [video tracks](/perceptron-mk1.5/capabilities/video-tracking) with timed spatial annotations.                                                         |
| Produce a denser object track                   | Combine model waypoints with optical flow and appearance tracking in [High Fidelity Object Tracking](/perceptron-mk1.5/guides/high-fidelity-object-tracking). |
| Compare videos or use reference images          | Send [multiple assets](/perceptron-mk1.5/guides/multiple-assets) in one conversation.                                                                         |
| Ask several independent questions               | Use [Multilook](/perceptron-mk1.5/guides/multilook) to share the context.                                                                                     |

## Send timestamped frames

Use `video_frames` when you have already selected frames from a video. Provide at least two frames with the same image dimensions, each with an `image_url` and a non-negative integer `timestamp_ms`. Keep frames in timestamp order; equal timestamps are allowed. URLs can be HTTP(S) URLs or supported image data URLs.

The SDK's `video_frames()` helper accepts `(image, timestamp_ms)` pairs and encodes local frames for you. Save frames from 0 and 1.5 seconds of your clip as `frame-000.png` and `frame-1500.png`; change the filenames and timestamps to match your selected frames. Run this after creating `client` above:

```python theme={null}
from perceptron import video_frames

response = client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=[{
        "role": "user",
        "content": [
            video_frames([
                ("frame-000.png", 0),
                ("frame-1500.png", 1500),
            ]),
            {"type": "text", "text": (
                "Describe the change visible between these frames. "
                "Use clip annotations with timestamps in seconds."
            )},
        ],
    }],
    vision_config={"annotation_format": "clip"},
    max_completion_tokens=1024,
)
if response.finish_reason != "stop":
    raise RuntimeError(f"Incomplete frame comparison: {response.finish_reason}")
print(response.text or "")
```

The entire `video_frames` group is one asset for `asset_idx`. Each supplied frame consumes one media unit, so this example uses one asset index and two of the request's 256 media units. Sending those frames as separate `image_url` content parts would instead create two image assets. See [multiple assets](/perceptron-mk1.5/guides/multiple-assets).

Timestamps are preserved, not automatically rebased: `1500` milliseconds corresponds to `1.5 seconds` in annotations. If these frames came from an excerpt starting at 60 seconds and you supplied excerpt-relative timestamps `0` and `1500`, add the 60-second offset when mapping results back to the source. If you supplied source-relative timestamps `60000` and `61500`, the timeline is already 60 and 61.5 seconds; do not add the offset again. Keep this mapping with your media, and check `finish_reason` before using the answer.

Selected frames provide sparse observations; they do not establish what happened between them. See [recover a gap with new model observations](/perceptron-mk1.5/guides/high-fidelity-object-tracking#recover-a-gap-with-new-model-observations) when you need more observations in an uncertain tracking interval.