> ## Documentation Index
> Fetch the complete documentation index at: https://docs.perceptron.inc/llms.txt
> Use this file to discover all available pages before exploring further.

# Image understanding

> Ask visual questions, extract text, and ground objects in images.

Send an image with a natural-language instruction to describe a scene, answer a question, or read visible text. Be specific about the region and the output you need. For OCR, ask the model to preserve reading order and mark unreadable text instead of guessing.

The [quickstart](/perceptron-mk1.5/index) shows a complete image question. You can use the same request with prompts such as:

| Task        | Example instruction                                                                      |
| ----------- | ---------------------------------------------------------------------------------------- |
| Captioning  | “Describe the scene in two sentences, including the main objects and their arrangement.” |
| Visual Q\&A | “Which objects are on the desk? Use only details visible in the image.”                  |
| OCR         | “Transcribe the visible labels from top to bottom. Mark unreadable text as unknown.”     |
| Comparison  | “Compare the objects in asset 0 and asset 1. Describe the visible differences.”          |

For comparisons and demonstrations with reference images, see [multiple assets](/perceptron-mk1.5/guides/multiple-assets). For machine-readable extracted fields, use [structured outputs](/perceptron-mk1.5/capabilities/structured-outputs).

## Explore image tasks

| Task                                                | Guide                                                                           |
| --------------------------------------------------- | ------------------------------------------------------------------------------- |
| Answer questions with visible evidence              | [Image Q\&A](/perceptron-mk1.5/capabilities/image-qa)                           |
| Write concise, detailed, or grounded descriptions   | [Image captioning](/perceptron-mk1.5/capabilities/image-captioning)             |
| Locate and count selected object categories         | [Object detection](/perceptron-mk1.5/capabilities/object-detection)             |
| Transcribe text, extract fields, or preserve tables | [OCR](/perceptron-mk1.5/capabilities/ocr)                                       |
| Supply labeled visual demonstrations                | [In-context learning](/perceptron-mk1.5/capabilities/in-context-learning-image) |

The shared [Detect API](/capabilities/detect) provides a separate detection endpoint with pixel-coordinate results. The task guides above use Mk1.5 chat completions.

## Locate objects

Install `perceptron>=0.4.0` and set `PERCEPTRON_API_KEY`. Choose `vision_config.annotation_format` to request points, boxes, or polygons. This example requests boxes around people:

```python theme={null}
import os

from perceptron import Client, image

client = Client(api_key=os.environ["PERCEPTRON_API_KEY"])

response = client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=[{
        "role": "user",
        "content": [
            image(
                "https://raw.githubusercontent.com/perceptron-ai-inc/perceptron/"
                "main/cookbook/_shared/assets/capabilities/detection/ppe_line.webp"
            ),
            {"type": "text", "text": "Locate each visible person with a bounding box."},
        ],
    }],
    max_completion_tokens=2048,
    vision_config={"annotation_format": "box"},
)
choice = response.choices[0]
if choice.finish_reason != "stop":
    raise RuntimeError(f"Incomplete grounding answer: {choice.finish_reason}")
print(choice.message.content or "")
```

The answer is markup in `message.content`, not a separate JSON array of detections. A box uses `<point_box>` with top-left and bottom-right coordinates on a normalized 0–1000 grid. See the [annotation reference](/perceptron-mk1.5/concepts/annotations) before rendering the response.

## Inspect details with tools

If a task needs another image, a crop, or information from an external system, your application can provide a function for that operation. The model requests the function; your code validates and executes it, then returns the result. See [building tool agents](/perceptron-mk1.5/guides/tool-agents) for returning image results and mapping crop coordinates back to the original image.