Task prompts
Use these as starting points, then evaluate them on representative inputs.
For spatial and temporal output, set
vision_config.annotation_format to the value in the table. These prompts request model-generated annotations; they do not guarantee exhaustive detection, correct counts, or continuous visibility.
Set the API controls explicitly
track is a container in annotation markup, not an annotation_format value. Use clip for moments or intervals, and spatial annotations inside tracks for positions over time. Coordinates use the normalized 0–1000 grid; generated annotation timestamps use seconds. See annotations.
Use reasoning_effort for new requests and leave the older thinking override unset. Reasoning and grounding are separate choices: do not assume all spatial tasks require reasoning off, or that a higher effort guarantees a better answer. See reasoning and tokenization for budgeting.
Ground a specific question
SetPERCEPTRON_API_KEY, then send the image, an observable task, and the requested annotation format together:
choices[0].message.content and inspect finish_reason before parsing it. Native annotations remain markup within that text; the response does not turn them into a separate detections array.
Provide examples and identify the target
When an object is hard to describe, supply a reference image and, when available, a verified annotation showing the intended region. Keep category labels consistent between demonstrations and the query. Accurate examples are more useful than many uncertain ones; review a model-generated exemplar before reusing it as a reference. Label the role of each input explicitly: for example, “Asset 0 is the reference object; locate matching objects in asset 1.” Requestasset_idx in annotations and preserve the media ordering when resending history. The attribute is optional, and the model may omit it despite the request; without an explicit or inherited selector, the annotation refers to the last asset passed to the model. A reference image plus a target is useful comparison context; an annotated demonstration additionally shows the desired geometry or answer format. See multiple assets.
For video, describe a visible event precisely and distinguish a single moment from a duration. Ask for separate clips for repeated events. If you provide a product reference followed by a video, request evidence in the video rather than an annotation on the reference image. Keep input timestamp_ms values distinct from generated timestamps in seconds. For denser object tracks, follow High Fidelity Object Tracking to combine model waypoints with optical flow and appearance tracking.
Choose a final-answer format
“Return JSON” is an instruction, not a response constraint. Use a JSON Schemaresponse_format when your application requires known fields and types, then validate the completed answer locally. Include an unknown or empty value for information that may not be visible. For document transcription, ask explicitly for reading order, tables, headings, or Markdown rather than relying on an unstated layout convention.
Keep native annotation markup and a constrained JSON report as distinct output contracts. Function arguments are another contract: function strict does not enforce their schema. A request cannot declare non-empty tools together with JSON Schema or regex constrained output. Finish the tool loop, then request a constrained report without tool declarations; completed tool history can remain. See structured outputs.