> ## Documentation Index
> Fetch the complete documentation index at: https://docs.perceptron.inc/llms.txt
> Use this file to discover all available pages before exploring further.

# Frame-by-frame tutorial

<Card icon="play" title="Run in Colab" href="https://colab.research.google.com/github/perceptron-ai-inc/perceptron/blob/main/cookbook/recipes/tutorials/isaac_frame_by_frame/isaac_frame_by_frame.ipynb">
  Step through this example interactively
</Card>

In this tutorial, we'll use the Perceptron SDK to run Isaac across an MP4 video of a surfing scene and stitch annotated frames back into a video. You can download the Jupyter notebook [here](https://github.com/perceptron-ai-inc/perceptron/blob/main/cookbook/recipes/tutorials/isaac_frame_by_frame/isaac_frame_by_frame.ipynb).

<Steps>
  <Step title="Install dependencies and configure the SDK">
    Install the SDK, OpenCV, Pillow, and tqdm. Export an API key so `configure()` can pick it up.

    ```bash theme={null}
    uv pip install perceptron opencv-python "pillow<12.0" tqdm
    export PERCEPTRON_API_KEY="sk_live_..."
    ```

    Create `frame_by_frame.py` and add the imports + configuration block:

    ```python frame_by_frame.py theme={null}
    from pathlib import Path

    import cv2
    from PIL import Image, ImageDraw
    from tqdm import tqdm

    from perceptron import configure, image, perceive, text

    configure(
        provider="perceptron",
        api_key="YOUR_API_KEY",
    )

    VIDEO_PATH = Path("surf.mp4")
    FRAMES_DIR = Path("frames")
    ANNOTATIONS_DIR = Path("frames_annotated")
    OUTPUT_VIDEO = Path("surf_annotated.mp4")

    FRAMES_DIR.mkdir(exist_ok=True)
    ANNOTATIONS_DIR.mkdir(exist_ok=True)
    ```
  </Step>

  <Step title="Download the sample video">
    Grab the shared surfing clip (or point the script at your own MP4).

    ```bash theme={null}
    curl -L -o surf.mp4 \
      https://raw.githubusercontent.com/perceptron-ai-inc/perceptron/main/cookbook/_shared/assets/tutorials/isaac_frame_by_frame/surf.mp4
    ```
  </Step>

  <Step title="Extract frames (tune the stride)">
    We sample one JPG every `stride` frames so long clips stay manageable.

    ```python theme={null}
    def extract_frames(video_path: Path, dest_dir: Path, stride: int = 3) -> list[Path]:
        cap = cv2.VideoCapture(str(video_path))
        if not cap.isOpened():
            raise RuntimeError(f"Failed to open video: {video_path}")

        saved: list[Path] = []
        idx = 0
        while True:
            ok, frame = cap.read()
            if not ok:
                break
            if idx % stride == 0:
                frame_path = dest_dir / f"frame_{idx:05d}.jpg"
                cv2.imwrite(str(frame_path), frame)
                saved.append(frame_path)
            idx += 1
        cap.release()
        print(f"Extracted {len(saved)} frames (stride={stride})")
        return saved

    FRAME_PATHS = extract_frames(VIDEO_PATH, FRAMES_DIR, stride=3)
    if not FRAME_PATHS:
        raise RuntimeError("No frames extracted; check the video and stride settings.")
    ```

    *Tuning tip*: decrease `stride` for smoother playback, increase it when you just need periodic samples.
  </Step>

  <Step title="Detect surfers in every frame">
    The `@perceive` helper wraps Isaac 0.1 so we can send each frame plus a natural-language instruction. The loop captures the raw answer, counts boxes, converts them to pixel coordinates, and draws overlays.

    ```python theme={null}
    PROMPT = "Find every surfer and surfboard in the frame. Return one bounding box per item."

    @perceive(expects="box", allow_multiple=True)
    def detect_surfers(frame_path: str):
        frame = image(frame_path)
        return frame + text(PROMPT)

    all_detections = []

    for frame_path in tqdm(FRAME_PATHS, desc="Detecting frames"):
        result = detect_surfers(str(frame_path))

        img = Image.open(frame_path).convert("RGB")
        draw = ImageDraw.Draw(img)
        pixel_boxes = result.points_to_pixels(width=img.width, height=img.height) or []

        all_detections.append(
            {
                "frame": frame_path.name,
                "boxes_count": len(pixel_boxes),
                "text": result.text,
            }
        )

        for box in pixel_boxes:
            draw.rectangle(
                [
                    int(box.top_left.x),
                    int(box.top_left.y),
                    int(box.bottom_right.x),
                    int(box.bottom_right.y),
                ],
                outline="lime",
                width=3,
            )
            label = box.mention or getattr(box, "label", None) or "surfer"
            draw.text(
                (int(box.top_left.x), max(int(box.top_left.y) - 18, 0)),
                label,
                fill="lime",
            )

        img.save(ANNOTATIONS_DIR / frame_path.name)
    ```

    * `result.points_to_pixels()` keeps the normalized → pixel conversion consistent.
    * The `all_detections` list becomes a quick audit trail (counts + captions per frame).
  </Step>

  <Step title="Stitch annotated frames back to MP4">
    OpenCV writes the annotated JPGs back into a video using the original resolution and your preferred FPS.

    ```python theme={null}
    def stitch_video(frame_dir: Path, output_path: Path, fps: int = 10) -> None:
        frames = sorted(frame_dir.glob("frame_*.jpg"))
        if not frames:
            raise RuntimeError("No annotated frames found to stitch.")

        sample = cv2.imread(str(frames[0]))
        height, width = sample.shape[:2]
        fourcc = cv2.VideoWriter_fourcc(*"mp4v")
        writer = cv2.VideoWriter(str(output_path), fourcc, fps, (width, height))

        for frame_path in frames:
            frame = cv2.imread(str(frame_path))
            writer.write(frame)
        writer.release()
        print(f"Saved annotated video to {output_path}")

    stitch_video(ANNOTATIONS_DIR, OUTPUT_VIDEO, fps=10)
    ```
  </Step>

  <Step title="Run the pipeline">
    Execute the script end-to-end and inspect the outputs. You should see:

    * `frames/` and `frames_annotated/` populated with numbered JPGs.
    * `surf_annotated.mp4` in the project root.

    ```bash theme={null}
    python frame_by_frame.py
    open surf_annotated.mp4  # macOS (use your OS viewer)
    ```
  </Step>
</Steps>

Feel free to tweak the prompt and target classes to match your production objects. We'd be delighted to hear about your project [on Discord](https://discord.gg/fgBeaACQzE).