Object detection

Ask Perceptron Mk1.5 to locate objects through chat completions. Name the categories and request the geometry your application needs: boxes for extents, points for centers, or polygons for boundaries.

Detect helmets and vests

Install perceptron>=0.4.0 and set PERCEPTRON_API_KEY. This example uses the public PPE image and asks for one box per visible helmet or safety vest.

import os
from pathlib import Path

from perceptron import Client, image

client = Client(api_key=os.environ["PERCEPTRON_API_KEY"])
image_url = (
    "https://raw.githubusercontent.com/perceptron-ai-inc/perceptron/"
    "main/cookbook/_shared/assets/capabilities/detection/ppe_line.webp"
)
response = client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=[{
        "role": "user",
        "content": [
            image(image_url),
            {"type": "text", "text": (
                "Detect visible helmets and safety vests. Return one flat point_box "
                "annotation per physical item, using mention='helmet' or mention='vest' "
                "and asset_idx=0. Use normalized coordinates. Do not duplicate an "
                "item or include objects outside these categories. If none are visible, "
                "say so without inventing a box. Do not wrap boxes in collections or tracks."
            )},
        ],
    }],
    reasoning_effort="high",
    max_completion_tokens=2048,
    vision_config={"annotation_format": "box"},
)
choice = response.choices[0]
if choice.finish_reason != "stop" or choice.message.tool_calls:
    raise RuntimeError(f"Incomplete detections: {choice.finish_reason}")
annotations = choice.message.content or ""
print(annotations)
Path("response.txt").write_text(annotations, encoding="utf-8")

Detections are annotation markup in message.content, not a separate JSON detections field. For example:

<point_box mention="helmet" asset_idx="0"> (420,100) (570,240) </point_box>
<point_box mention="vest" asset_idx="0"> (350,280) (650,760) </point_box>

These values illustrate the format and are not measured detections from this image. Coordinates are normalized to 0–1000, with top-left followed by bottom-right. asset_idx="0" identifies the input image. Validate the label, selector, coordinate range, and box ordering before using a detection. Follow rendering annotations to draw these boxes. For a center-point workflow, change the annotation format to "point" and ask for each object’s center; use "polygon" when an outline better represents the target.

Return counts as structured data

For applications that need numbers, request a constrained count response. Install jsonschema, then run this after the client and image setup above:

import json

from jsonschema import validate

count_schema = {
    "type": "object",
    "properties": {
        "helmet": {"type": "integer", "minimum": 0},
        "vest": {"type": "integer", "minimum": 0},
    },
    "required": ["helmet", "vest"],
    "additionalProperties": False,
}
response = client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=[{
        "role": "user",
        "content": [
            image(image_url),
            {"type": "text", "text": (
                "Count visible helmets and safety vests, counting each physical item "
                "once. Count the items, not the people. Return zero for a category "
                "with no visible instances. Return the requested JSON counts."
            )},
        ],
    }],
    reasoning_effort="high",
    max_completion_tokens=1024,
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "ppe_counts", "strict": True, "schema": count_schema},
    },
)
choice = response.choices[0]
if choice.finish_reason != "stop" or choice.message.tool_calls:
    raise RuntimeError("The count response did not complete")
counts = json.loads(choice.message.content or "")
validate(instance=counts, schema=count_schema)
print(counts)

This is a separate request, so its counts can differ from the first request’s detections. If totals must match the displayed overlay exactly, count the validated annotations by category in your application. A valid schema ensures the expected structure; it does not establish that every item was detected. Occlusion and small objects can affect counts.

Define the target precisely

Use clear category descriptions and explain borderline cases: “high-visibility safety vest” is more specific than “vest.” For subtle targets, supply a labeled reference image through in-context image learning. With several images, request asset_idx on every annotation and select the corresponding dimensions when rendering. For objects moving through a video, use video tracking. Separate detections from independent images do not create persistent object identities.

Choose the endpoint

The examples above select perceptron-mk1.5 on /v1/chat/completions and return normalized annotation markup. The separate Detect API provides a managed detection workflow with its own supported model and target-image pixel coordinates. Its request and response formats differ from the chat-completion examples on this page.

Get Started

Capabilities

Developer Guides

Scaling & deployment

Best practices

Detect helmets and vests

Return counts as structured data

Define the target precisely

Choose the endpoint

​Detect helmets and vests

​Return counts as structured data

​Define the target precisely

​Choose the endpoint

Detect helmets and vests

Return counts as structured data

Define the target precisely

Choose the endpoint