Building a tool agent

A tool agent is an application that repeats the function-calling loop: ask the model, execute its calls, return the results, and ask again. Perceptron Mk1.5 can combine those results with images and videos already in the conversation.

Design tools around useful steps

Give each tool a specific job and a short description of when to use it. Prefer inputs your application can validate and results the model can use directly.

Application	Example tools	Workflow
Product matching	`search_catalog`, `get_product`	Identify an item in an image, retrieve candidates, and compare specifications or reference images.
Visual inspection	`get_inspection_record`, `get_reference_image`	Compare a new image with the approved reference and retrieve relevant inspection history.
Document research	`search_documents`, `read_document`	Search a collection, read relevant passages, and answer with source references.

These are function designs for your application to implement. Include source IDs, titles, and relevant excerpts in retrieval results, and ask the model to cite those IDs. Return an explicit empty result when nothing matches. Separate searching from reading so a broad search does not fill the conversation with entire documents. When a tool returns a long document, extract passages relevant to the question before adding the result. If you use another model request to summarize a document, count that request against the same time and token budgets as the main loop.

Bound the whole workflow

Choose budgets before starting the loop. The values below are illustrative application settings, not API limits.

Budget	Example	Where to enforce it
Model rounds	4	Before each model request
Total tool calls	6	Before dispatch, counting every call in a parallel group
Concurrent executions	2	In your tool runner
End-to-end time	90 seconds	Across model calls, tools, and retries together
Completion tokens	2,048 per request	`max_completion_tokens` on every model request
Result size	A few relevant passages or images	Before appending each result

Use a single deadline based on a monotonic clock. Recompute the remaining time before each request and tool execution, including after waiting for a concurrency slot. Set each operation’s timeout to at most the remaining time and stop when it reaches zero. A timeout on an individual request alone does not bound a multi-round workflow. Add token usage across every model request to track the workflow’s cost. Each new request includes the accumulated history, so input usage can grow even when the latest tool result is short. Stop initiating calls when your budget is spent; reserve room for a final answer if you need one.

Return failures as results

An unknown function, invalid argument, timeout, or empty lookup should produce a clear result for that call. This gives the model a chance to choose another step or explain the limitation. The function-calling example already does this for invalid names and arguments. For example, this is a tool-result message for a call that timed out; replace call_from_response with the actual returned ID:

{
  "role": "tool",
  "tool_call_id": "call_from_response",
  "content": "{\"error\":\"The document lookup timed out. No document was retrieved.\"}"
}

When a batch exceeds your remaining budget, return a result explaining that limit for each skipped call. Before making another model request, every call in the previous assistant message must have a result. Keep failures separate from successful data so the model does not treat an error as evidence. To request a final answer after the budget is exhausted, keep the tool declarations and set tool_choice: "none". Tell the model to answer from the results already available. If it still returns calls, stop execution and report that the workflow reached its limit; "none" is not an execution guard. Run independent calls concurrently when useful, but preserve their IDs and append all results before asking the model to continue. Stable result ordering also keeps image indices predictable. Retry decisions belong in the application; avoid automatically repeating an action that may already have succeeded.

Return an image from a tool

Tool results can contain text and image parts. This lets a catalog lookup return a product photograph or a document lookup return an existing page image. A bare image URL inside a text string is just text; use an image_url content part to provide the image to the model. This example compares an observed product with its reference photograph. Save two PNG images as observed.png and reference.png in your working directory, install perceptron>=0.4.0, and set PERCEPTRON_API_KEY. Save the script as compare_product.py and run python compare_product.py. The application declares get_reference_image(product_id) and maps the product ID to a local file. The observed image is asset_idx=0; the image returned by the tool becomes asset_idx=1. The script makes at most two model requests: one to request the reference and one to compare the images.

import json
from pathlib import Path

from perceptron import Client, function_tool, image


client = Client(timeout=60.0)
REFERENCE_IMAGES = {"product-001": Path("reference.png")}
TOOLS = [function_tool(
    "get_reference_image",
    description="Return the reference photograph for a product ID.",
    parameters={
        "type": "object",
        "properties": {
            "product_id": {"type": "string", "enum": ["product-001"]},
        },
        "required": ["product_id"],
        "additionalProperties": False,
    },
)]


def execute_tool(call):
    function = call["function"]
    if function["name"] != "get_reference_image":
        raise ValueError("Unknown function.")
    arguments = json.loads(function["arguments"])
    if not isinstance(arguments, dict) or set(arguments) != {"product_id"}:
        raise ValueError("Expected only product_id in the arguments.")
    product_id = arguments["product_id"]
    if not isinstance(product_id, str) or product_id not in REFERENCE_IMAGES:
        raise ValueError("Unknown product ID.")
    return {
        "role": "tool",
        "tool_call_id": call["id"],
        "content": [
            {
                "type": "text",
                "text": f"Reference photograph for {product_id} (asset_idx=1).",
            },
            image(REFERENCE_IMAGES[product_id]),
        ],
    }


def complete(messages, tool_choice):
    return client.chat.completions.create(
        model="perceptron-mk1.5",
        messages=messages,
        tools=TOOLS,
        tool_choice=tool_choice,
        parallel_tool_calls=False,
        reasoning_effort="high",
        max_completion_tokens=2048,
    ).choices[0]


def run():
    messages = [{
        "role": "user",
        "content": [
            image("observed.png"),
            {
                "type": "text",
                "text": (
                    "This is an observed product-001 (asset_idx=0). "
                    "Call get_reference_image for product-001, then compare the "
                    "observed item with the reference photograph. Describe visible "
                    "matches and differences. Include asset_idx on any spatial annotations."
                ),
            },
        ],
    }]
    choice = complete(messages, "auto")
    assistant = choice.message.to_dict()
    calls = assistant.get("tool_calls") or []
    if choice.finish_reason != "tool_calls" or len(calls) != 1:
        raise RuntimeError("Expected one completed reference-image call.")

    messages.append(assistant)
    messages.append(execute_tool(calls[0]))
    final = complete(messages, "none")
    if final.finish_reason != "stop" or final.message.tool_calls:
        raise RuntimeError("The model did not complete its comparison.")
    print(final.message.content or "")


if __name__ == "__main__":
    run()

The second request includes the complete assistant message, including any reasoning_content, and the image result with its matching tool_call_id. The comparison can now refer to both the user’s photograph and the retrieved reference. tool_choice: "none" requests a final answer; the script stops if the model asks for another tool instead. The SDK encodes these local images inline. Pass an HTTPS URL to image(url) for a remote image, or use image(file_id="file-...") for an image already uploaded with the Files API. Tool results support text and images; other media belong in user messages. Images returned by tools become new assets in conversation order. If the user supplied one image and the first tool result adds two images, their indices are 0, 1, and 2. A later annotation such as <point_box asset_idx="2">...</point_box> refers to the second tool-returned image, with coordinates relative to that image’s dimensions. Adding those images does not change the bindings of annotations produced before the tool result. Maintain the same ordering when replaying history. Indices refer to occurrences of media, so returning the same image again adds another asset position. Include a short description next to each image, especially when returning several candidates. See Working with multiple assets for allocation rules and Annotations for rendering coordinates.

Combine retrieval with visual evidence

For a product-matching workflow, send the user’s image and ask the model to search your catalog, inspect the returned candidates, and explain the match using both visible features and retrieved specifications. Keep the user’s original image and each candidate as separate assets. Ask for asset_idx on spatial annotations to help your viewer select the right image. If it is neither explicit nor inherited, the annotation refers to the last asset available when it was produced, including images returned by tools up to that point. For video inspection, keep the video in the conversation while tools retrieve reference material. Ask for timed track annotations when the answer needs to identify an object’s path, and cite retrieved records separately from visual observations. The application chooses which tool results to expose to the model and which annotations to render for the user.

Get Started

Capabilities

Developer Guides

Scaling & deployment

Best practices

Design tools around useful steps

Bound the whole workflow

Return failures as results

Return an image from a tool

Combine retrieval with visual evidence

​Design tools around useful steps

​Bound the whole workflow

​Return failures as results

​Return an image from a tool

​Combine retrieval with visual evidence

Design tools around useful steps

Bound the whole workflow

Return failures as results

Return an image from a tool

Combine retrieval with visual evidence