Coordinate system

Mk1.5 chat-completion annotations use a 0–1000 grid on each image or video frame. The origin is the top-left corner, x increases to the right, and y increases downward. (500,500) is the center, whatever the image dimensions.

(0,0) ───────────────────── (1000,0)
  │                             │
  │          (500,500)          │
  │                             │
(0,1000) ────────────────── (1000,1000)

Keep normalized geometry together with its optional asset_idx and, for video observations, its timestamp. Resolve an omitted selector through inheritance or the last asset available when the annotation was produced. The annotation reference defines the tags and inheritance rules; multiple assets explains which input each selector identifies.

Convert between normalized and pixel coordinates

Use the dimensions of the selected image as it was supplied to the model:

def to_pixels(x, y, width, height):
    return x * width / 1000, y * height / 1000


def to_normalized(pixel_x, pixel_y, width, height):
    if width <= 0 or height <= 0:
        raise ValueError("Image dimensions must be positive")
    return pixel_x * 1000 / width, pixel_y * 1000 / height


print(to_pixels(500, 500, 1920, 1080))  # (960.0, 540.0)
print(to_normalized(960, 540, 1920, 1080))  # (500.0, 500.0)

Apply the conversion to both corners of a box and to every vertex of a polygon. Keep fractional coordinates until the drawing or indexing step. A coordinate of 1000 denotes the image boundary; if a drawing library needs an integer pixel index, clamp the final index to width - 1 or height - 1. For example, the illustrative normalized box (250,250) (750,750) becomes (480,270) (1440,810) on a 1920×1080 image. On a 640×360 thumbnail of the same image it becomes (160,90) (480,270).

Choose the correct asset

Identify the asset before choosing dimensions. Use asset_idx when present, including an inherited value. Otherwise, use the last asset available when the annotation was produced. If the only asset is a 1920×1080 image, a box without a selector uses that image’s dimensions, even if a later tool result adds a 512×512 crop. A new box produced after the crop defaults to the crop when its selector is neither explicit nor inherited. Repeating an image in the conversation creates another asset occurrence. For a video track, the asset selector identifies the video and t identifies the observation time. Spatial coordinates describe that frame. Keep frame timestamps separate from frame numbers: sampling does not guarantee that one model observation corresponds to one source frame. When rendering a thumbnail, use the size of its displayed image content. If the viewer letterboxes the image, add the content rectangle’s offset after scaling; do not scale into the surrounding margins. Keep EXIF orientation and any explicit rotations consistent between the input and the rendered image.

Map a crop back to the original image

A crop returned by a tool is a separate asset. Its coordinates describe the crop, so first convert them to crop pixels and add the crop’s offset in the original:

def crop_point_to_image(x, y, left, top, crop_width, crop_height, image_width, image_height):
    if min(crop_width, crop_height, image_width, image_height) <= 0:
        raise ValueError("Image and crop dimensions must be positive")
    pixel_x = left + x * crop_width / 1000
    pixel_y = top + y * crop_height / 1000
    return pixel_x * 1000 / image_width, pixel_y * 1000 / image_height


# Center of a 400×200 crop beginning at (100,50) in a 1000×500 image.
print(crop_point_to_image(500, 500, 100, 50, 400, 200, 1000, 500))
# (300.0, 300.0) in the original image's normalized coordinates.

Apply this to every point being mapped, including both box corners. This formula assumes the crop has only been cropped and resized. Rotated or perspective-transformed crops need the corresponding transform. See High Fidelity Object Tracking for keeping video frames, timestamps, and boxes aligned during local tracking.

Validate before drawing or measuring

Require finite coordinates in the expected 0–1000 range. Keep invalid predictions available for inspection rather than silently hiding them with clamping.
Require ordered box corners: x1 < x2 and y1 < y2 for a nonempty box.
Resolve inherited selectors on collections and tracks before selecting a canvas. Preserve selector 0; it is a valid value.
Compare geometry only after resolving its asset, timestamp, orientation, and coordinate frame. Normalized coordinates alone do not make boxes from different images comparable.

Intersection-over-union can be computed in normalized coordinates when both boxes describe the same image and coordinate frame. Converting both through the same axis-aligned resize leaves the ratio unchanged. For a complete response-to-image example, see Rendering annotations. The shared Detect API has its own pixel-coordinate response contract; use the contract of the endpoint that produced your result.

Get Started

Capabilities

Developer Guides

Scaling & deployment

Best practices

Convert between normalized and pixel coordinates

Choose the correct asset

Map a crop back to the original image

Validate before drawing or measuring

​Convert between normalized and pixel coordinates

​Choose the correct asset

​Map a crop back to the original image

​Validate before drawing or measuring

Convert between normalized and pixel coordinates

Choose the correct asset

Map a crop back to the original image

Validate before drawing or measuring