In-context learning (image)

In-context learning gives Perceptron Mk1.5 examples of the visual concept you want to find. Supply a reference image, a label, and a box around the reference object, then ask for matching objects in a target image. The examples guide that request; they do not train or permanently update the model. Use one reference for a specific object or several labeled references to distinguish classes. This is useful when a visual example communicates the appearance more clearly than a description alone.

Detect from labeled examples

This example uses reviewed boxes around a cat and a dog in the sample reference images. It runs once with only the dog reference, then with both classes. The target image shows a dog; providing a cat reference does not mean a cat must appear in the answer. Install the perceptron>=0.4.0 Python package and set PERCEPTRON_API_KEY before running:

import os
from html import escape

from perceptron import Client, image

client = Client(api_key=os.environ["PERCEPTRON_API_KEY"])
asset_base = (
    "https://raw.githubusercontent.com/perceptron-ai-inc/perceptron/"
    "main/cookbook/_shared/assets/in-context-learning/multi"
)
cat_reference = {
    "url": f"{asset_base}/classA.jpg",
    "label": "cat",
    "box": (316, 136, 703, 906),
}
dog_reference = {
    "url": f"{asset_base}/classB.webp",
    "label": "dog",
    "box": (161, 48, 666, 980),
}
target_url = f"{asset_base}/cat_dog_input.png"


def detect_from_examples(examples, target_url):
    content = [{"type": "text", "text": (
        "The following images and box annotations are labeled references. "
        "Use them to recognize the same classes in the final target image."
    )}]
    for asset_idx, example in enumerate(examples):
        x1, y1, x2, y2 = example["box"]
        label = escape(example["label"], quote=True)
        content.extend([
            image(example["url"]),
            {"type": "text", "text": (
                f'Reference: <point_box mention="{label}" asset_idx="{asset_idx}"> '
                f"({x1},{y1}) ({x2},{y2}) </point_box>"
            )},
        ])

    target_idx = len(examples)
    content.extend([
        image(target_url),
        {"type": "text", "text": (
            f"This is the target image, asset_idx={target_idx}. "
            "Find all visible instances of the reference classes here. "
            "Return flat point_box annotations, without collections, using the reference labels and "
            f"asset_idx={target_idx}, with normalized 0–1000 coordinates. "
            "Do not annotate the reference images. Do not assume every class "
            "is present; say when no matching object is visible."
        )},
    ])
    response = client.chat.completions.create(
        model="perceptron-mk1.5",
        messages=[{"role": "user", "content": content}],
        reasoning_effort="high",
        max_completion_tokens=2048,
        vision_config={"annotation_format": "box"},
    )
    choice = response.choices[0]
    if choice.finish_reason != "stop":
        raise RuntimeError(f"Incomplete detection answer: {choice.finish_reason}")
    return choice.message.content or ""


print("One reference:")
print(detect_from_examples([dog_reference], target_url))

print("Multiple classes:")
print(detect_from_examples([cat_reference, dog_reference], target_url))

In the first call, the dog reference is asset 0 and the target is asset 1. In the second, the cat is asset 0, the dog is asset 1, and the target is asset 2. Each call has its own supplied conversation, so its numbering starts again at zero. See multiple assets before adapting this to a conversation with earlier media. The answer is annotation markup in message.content. Check that each returned box refers to the target asset before drawing it. The annotation reference explains coordinate conversion, collections, and validation. To draw the boxes, save one call’s returned string as response.txt, download the target image, and follow rendering annotations. Use --asset-idx 1 --last-asset-idx 1 for the single-reference answer or --asset-idx 2 --last-asset-idx 2 for the multiclass answer. Keep the two answers separate because their indices differ. When asset_idx is omitted and no selector is inherited, it defaults to the last asset available when that answer was produced, which is the target image in both examples.

Prepare your own references

Replace each reference URL, label, and box together. The sample coordinates belong only to the sample images. Use the normalized 0–1000 grid, with the top-left and bottom-right corners enclosing the object you intend to teach. If you obtain candidate boxes from a previous model response, inspect and correct them before using them as examples.

Choose references with a clear view of the object, including the features that distinguish it from similar objects.
Use consistent class labels across examples and the target request. Multiple views of one class should share a label.
Include confusing alternatives as separately labeled references when distinguishing them matters.
Keep the target image separate from the examples, and request output only for its asset_idx.

For the same reference-to-query pattern across time, see in-context learning for video. For ordinary detection without examples, see object detection.

Get Started

Capabilities

Developer Guides

Scaling & deployment

Best practices

In-context learning (image)

Detect from labeled examples

Prepare your own references

​Detect from labeled examples

​Prepare your own references

Detect from labeled examples

Prepare your own references