Image understanding

Send an image with a natural-language instruction to describe a scene, answer a question, or read visible text. Be specific about the region and the output you need. For OCR, ask the model to preserve reading order and mark unreadable text instead of guessing. The quickstart shows a complete image question. You can use the same request with prompts such as:

Task	Example instruction
Captioning	“Describe the scene in two sentences, including the main objects and their arrangement.”
Visual Q&A	“Which objects are on the desk? Use only details visible in the image.”
OCR	“Transcribe the visible labels from top to bottom. Mark unreadable text as unknown.”
Comparison	“Compare the objects in asset 0 and asset 1. Describe the visible differences.”

For comparisons and demonstrations with reference images, see multiple assets. For machine-readable extracted fields, use structured outputs.

Explore image tasks

Task	Guide
Answer questions with visible evidence	Image Q&A
Write concise, detailed, or grounded descriptions	Image captioning
Locate and count selected object categories	Object detection
Transcribe text, extract fields, or preserve tables	OCR
Supply labeled visual demonstrations	In-context learning

The shared Detect API provides a separate detection endpoint with pixel-coordinate results. The task guides above use Mk1.5 chat completions.

Locate objects

Install perceptron>=0.4.0 and set PERCEPTRON_API_KEY. Choose vision_config.annotation_format to request points, boxes, or polygons. This example requests boxes around people:

import os

from perceptron import Client, image

client = Client(api_key=os.environ["PERCEPTRON_API_KEY"])

response = client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=[{
        "role": "user",
        "content": [
            image(
                "https://raw.githubusercontent.com/perceptron-ai-inc/perceptron/"
                "main/cookbook/_shared/assets/capabilities/detection/ppe_line.webp"
            ),
            {"type": "text", "text": "Locate each visible person with a bounding box."},
        ],
    }],
    max_completion_tokens=2048,
    vision_config={"annotation_format": "box"},
)
choice = response.choices[0]
if choice.finish_reason != "stop":
    raise RuntimeError(f"Incomplete grounding answer: {choice.finish_reason}")
print(choice.message.content or "")

The answer is markup in message.content, not a separate JSON array of detections. A box uses <point_box> with top-left and bottom-right coordinates on a normalized 0–1000 grid. See the annotation reference before rendering the response.

Inspect details with tools

If a task needs another image, a crop, or information from an external system, your application can provide a function for that operation. The model requests the function; your code validates and executes it, then returns the result. See building tool agents for returning image results and mapping crop coordinates back to the original image.

Get Started

Capabilities

Developer Guides

Scaling & deployment

Best practices

Explore image tasks

Locate objects

Inspect details with tools

​Explore image tasks

​Locate objects

​Inspect details with tools

Explore image tasks

Locate objects

Inspect details with tools