> ## Documentation Index
> Fetch the complete documentation index at: https://docs.perceptron.inc/llms.txt
> Use this file to discover all available pages before exploring further.

# Rendering annotations

> Parse image boxes with the SDK, resolve their asset selectors, and draw them on the original image.

A grounded answer contains annotation text in `message.content`. To draw it, first check that the response completed, parse the requested geometry, and resolve its asset selector to the correct image. See [annotation format](/perceptron-mk1.5/concepts/annotations) for the complete grammar and [coordinates](/perceptron-mk1.5/concepts/coordinates) for scaling and crop transforms.

## Save a completed grounding answer

The [image Q\&A](/perceptron-mk1.5/capabilities/image-qa), [captioning](/perceptron-mk1.5/capabilities/image-captioning), [detection](/perceptron-mk1.5/capabilities/object-detection), and [OCR](/perceptron-mk1.5/capabilities/ocr) guides include grounded examples. Request `point_box` elements with labels in `mention` for this walkthrough. You can request explicit `asset_idx` values to help map boxes to images, but the attribute is optional and the model may omit it even when requested. Save the completed output as `response.txt`, and keep the exact input image locally.

For example, after a non-streaming request has returned `response`:

```python theme={null}
from pathlib import Path

choice = response.choices[0]
if choice.finish_reason != "stop" or choice.message.tool_calls:
    raise RuntimeError("Grounding did not return a completed answer")
Path("response.txt").write_text(choice.message.content or "", encoding="utf-8")
```

For streaming, accumulate the answer and require successful completion before saving it. Receiving a closing box tag does not establish that the whole response succeeded.

## Draw image boxes

Install `perceptron>=0.4.0` and `pillow` and save the following as `render_boxes.py`. Provide the index of the image you want to render and the index of the last asset available when the saved answer was produced. For a request with one image, both indices are `0`:

```bash theme={null}
pip install "perceptron>=0.4.0" pillow
python render_boxes.py image.png response.txt --asset-idx 0 --last-asset-idx 0 --output annotated.png
```

For a request with two images, render the second image with:

```bash theme={null}
python render_boxes.py second-image.png response.txt --asset-idx 1 --last-asset-idx 1 --output annotated.png
```

The parser preserves a missing `asset_idx` as `None`; the renderer resolves it to `--last-asset-idx`. After checking inherited selectors, an omitted selector refers to the **last asset available when the answer was produced**. An explicit selector, including `0`, takes precedence. Count supplied media in message history and tool results up to that answer. Do not include media added later in the conversation when rendering an earlier answer. To render the first image from the two-image request, use `--asset-idx 0 --last-asset-idx 1`; boxes without selectors still belong to asset `1` and are filtered out.

The SDK parser handles boxes and inherited collection selectors. This renderer accepts a small, explicit subset: static `point_box` elements, optionally grouped in `collection` elements, with only `mention` and `asset_idx` attributes. Write coordinates as `(x1,y1) (x2,y2)`, quote attribute values, and XML-escape label characters such as `&` and `"`. Tracks, timestamps, and other geometry need a different renderer.

`strict=True` raises reported SDK parse errors; it is not a complete syntax or geometry validator. The example checks each annotation fragment with Python's XML parser, rejects leftover annotation markup, and validates coordinates and selectors before drawing.

```python theme={null}
import argparse
import math
import re
import xml.etree.ElementTree as ET
from pathlib import Path

from perceptron import collect_annotations, resolve_asset_idx
from PIL import Image, ImageDraw, ImageOps


def parse_boxes(text):
    annotations = collect_annotations(text, strict=True)
    tag_names = ("point_box", "point", "polygon", "clip", "collection", "track")
    coordinate = r"\(\s*[0-9]+\s*,\s*[0-9]+\s*\)"
    box_body = rf"\s*{coordinate}\s+{coordinate}\s*"
    for segment in annotations.parsed:
        if segment["kind"] == "text":
            remaining = segment["text"]
            tag_start = re.search(r"</?(?:point_box|point|polygon|clip|collection|track)\b", remaining, re.I)
            partial = re.search(r"</?([a-z_]+)$", remaining.rstrip(), re.I)
            if tag_start or (partial and any(tag.startswith(partial[1].lower()) for tag in tag_names)):
                raise ValueError("Unparsed annotation markup remains in the answer")
            continue
        span = segment["span"]
        try:
            root = ET.fromstring(text[span["start"]:span["end"]])
        except ET.ParseError as error:
            raise ValueError("Annotation markup must be valid XML") from error
        for node in root.iter():
            if node.tag not in {"point_box", "collection"}:
                raise ValueError("Expected static image boxes only")
            if set(node.attrib) - {"mention", "asset_idx"}:
                raise ValueError("Only mention and asset_idx attributes are supported")
            selector = node.get("asset_idx")
            if selector is not None and re.fullmatch(r"[0-9]+", selector) is None:
                raise ValueError("asset_idx must be a non-negative integer when present")
            if node.tag == "point_box":
                if len(node) or re.fullmatch(box_body, node.text or "") is None:
                    raise ValueError("A box must contain exactly two coordinate pairs")
            elif (node.text or "").strip() or any((child.tail or "").strip() for child in node):
                raise ValueError("Collections must contain only annotation elements")
    if annotations.tracks or annotations.points or annotations.polygons or annotations.clips:
        raise ValueError("Expected static image boxes only")
    for box in annotations.boxes:
        if box.t is not None:
            raise ValueError("Timestamped boxes need a video frame")
        x1, y1 = box.top_left.x, box.top_left.y
        x2, y2 = box.bottom_right.x, box.bottom_right.y
        if not all(math.isfinite(value) for value in (x1, y1, x2, y2)):
            raise ValueError("Coordinates must be finite")
        if not (0 <= x1 < x2 <= 1000 and 0 <= y1 < y2 <= 1000):
            raise ValueError("Box corners must be ordered and within 0–1000")
    return annotations.boxes


def render(image_path, response_path, asset_idx, last_asset_idx, output_path):
    if last_asset_idx < 0:
        raise ValueError("last_asset_idx must be non-negative")
    if not 0 <= asset_idx <= last_asset_idx:
        raise ValueError("asset_idx must be between 0 and last_asset_idx")
    boxes = parse_boxes(Path(response_path).read_text(encoding="utf-8"))
    selected = [
        box for box in boxes
        if resolve_asset_idx(box, n_assets=last_asset_idx + 1) == asset_idx
    ]
    with Image.open(image_path) as source:
        canvas = ImageOps.exif_transpose(source).convert("RGB")
    width, height = canvas.size
    draw = ImageDraw.Draw(canvas)
    for box in selected:
        x1, y1 = box.top_left.x, box.top_left.y
        x2, y2 = box.bottom_right.x, box.bottom_right.y
        pixels = (
            min(width - 1, round(x1 * width / 1000)),
            min(height - 1, round(y1 * height / 1000)),
            min(width - 1, round(x2 * width / 1000)),
            min(height - 1, round(y2 * height / 1000)),
        )
        draw.rectangle(pixels, outline="red", width=3)
        draw.text((pixels[0], max(0, pixels[1] - 12)), box.mention or "object", fill="red")
    canvas.save(output_path)
    print(f"Drew {len(selected)} boxes for asset {asset_idx} in {output_path}")


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("image", type=Path)
    parser.add_argument("response", type=Path)
    parser.add_argument("--asset-idx", type=int, required=True)
    parser.add_argument("--last-asset-idx", type=int, required=True,
                        help="Index of the last asset available when this answer was produced")
    parser.add_argument("--output", type=Path, default=Path("annotated.png"))
    args = parser.parse_args()
    render(args.image, args.response, args.asset_idx, args.last_asset_idx, args.output)
```

Within this supported subset, malformed markup, empty or duplicate selectors, unsupported attributes, and invalid coordinates raise an error rather than silently changing the prediction. Keep the original model output for inspection if parsing fails; do not repair an unfinished answer into an apparent detection.

A successful run can draw zero boxes when the answer contains no detections for that asset. Inspect the answer and your asset mapping to distinguish an absent object from a selector mismatch. A valid box and a successful drawing do not establish that the model identified the correct object.

## Extend to other geometry

For points and polygons, apply the same asset selection and [coordinate conversion](/perceptron-mk1.5/concepts/coordinates) to each coordinate. For collections, resolve inherited asset selectors before rendering children. For tracks, first select the video and timestamp; the [tracking guide](/perceptron-mk1.5/capabilities/video-tracking#turn-waypoints-into-an-overlay) explains sparse observations and interpolation.

Use image drawing primitives for geometry and plain text for labels. Do not insert the model's annotation string as executable HTML; ordinary HTML parsers also treat `track` as a void element and can lose its children.