MCP

Perceptron’s hosted MCP server lets an AI assistant ask questions about media, generate captions, detect objects, and read text. Connect directly over Streamable HTTP for image, video, audio, and reasoning-effort controls. The optional local npm bridge also uploads local files, with the version limits described below.

Before you begin

You need an MCP client and a Perceptron API key. Direct hosted connections do not require Node.js. The local npm bridge requires Node.js 24 or later.

Create an API key

Get your key from the Perceptron platform

Join Discord

Get help and see what others are building

Connect directly to hosted MCP

Add a remote server in a client that supports Streamable HTTP and custom authorization headers:

Setting	Value
Transport	Streamable HTTP
URL	`https://api.perceptron.inc/mcp`
Header name	`Authorization`
Header value	`Bearer YOUR_API_KEY`

Replace YOUR_API_KEY with your Perceptron key using the client’s secret or private configuration. Do not commit credentials with the project. Your MCP client controls when tools can run and any approval prompts it shows. Connect the server and refresh its tools in your client. The examples in this section are hosted MCP tool arguments, not chat-completions request bodies. Hosted calls accept accessible media URLs or supported data URLs; they cannot read a path on your computer.

Select a model

The hosted service defaults to perceptron-mk1.5 when model is omitted. This replaces the previous Mk1 default; pass "model": "perceptron-mk1" to keep the previous model. These examples set "model": "perceptron-mk1.5" explicitly so the choice remains stable if the default changes again. A previous tool call does not select the model for the next call. The model used by your assistant is also independent of the model used by a Perceptron tool. Model discovery is available through MCP resources at perceptron://models/{modelId}.

Choose a hosted tool

Tool	Inputs	Task controls
`question`	`media_url`, `question`; `modality`: image, video, or audio	Optional `output_format`: point, box, polygon, or clip
`caption`	`media_url`; `modality`: image, video, or audio	Optional `style`: concise (default) or detailed; `output_format`
`detect`	`media_url`; `modality`: image or video	Optional `classes`: a list of object labels
`ocr`	`image_url`, `mode`; image-only	`mode`: plain, markdown, or html; optional `prompt`
`generate_upload_urls`	`files`: file names, MIME types, and byte lengths	Returns URLs for uploading media
`generate_download_urls`	`object_keys` from the upload response	Returns media URLs to use in an inference tool

For hosted question, caption, and detect calls, modality defaults to image. Set it explicitly when sending video or audio. Each inference call accepts one media input. All four inference tools accept optional model, max_tokens, temperature, and reasoning_effort settings.

Ask about audio

Call question with modality: "audio". Replace the example URL with an accessible WAV, MP3, or FLAC recording:

{
  "media_url": "https://example.com/recording.wav",
  "modality": "audio",
  "model": "perceptron-mk1.5",
  "question": "Summarize the speech and describe any other audible sounds.",
  "reasoning_effort": "high",
  "max_tokens": 2048
}

caption also accepts audio. detect and ocr do not accept audio inputs. Omit spatial output formats for an audio-only question.

Include a video’s soundtrack

Video soundtracks are disabled by default. To analyze sound together with frames, set enable_audio_in_video: true on a hosted question or caption call with modality: "video":

{
  "media_url": "https://raw.githubusercontent.com/perceptron-ai-inc/perceptron/main/cookbook/_shared/assets/capabilities/video-clipping/mj_shot_short.mp4",
  "modality": "video",
  "model": "perceptron-mk1.5",
  "question": "Describe what happens and what can be heard in this clip.",
  "enable_audio_in_video": true,
  "reasoning_effort": "high",
  "max_tokens": 2048
}

This sample contains a soundtrack; replace its URL with your video as needed. Setting the flag to true on an image or audio-only input returns a tool error. The flag is available only on question and caption. The audio limits and usage rules also apply to recordings and soundtracks sent through MCP.

Set reasoning effort

The four hosted inference tools accept reasoning_effort: none, minimal, low, medium, or high. Any value other than none enables reasoning. The older boolean reasoning field is deprecated; omit it when using reasoning_effort. The JSON result inside an MCP text block uses content for the answer and reasoning for reasoning when returned. More reasoning consumes output budget and does not guarantee a more accurate answer. See Reasoning for guidance on choosing an effort level.

Upload media for a hosted call

For a file on your computer, your application can use the hosted upload helpers:

Call generate_upload_urls with a files array. Each entry contains file_name, content_type, and content_length in bytes.
Upload the actual bytes to each returned upload_url using HTTP PUT and the matching Content-Type. Upload files sequentially.
Call generate_download_urls with the returned object_key values in its object_keys array, then pass a returned download_url as media_url or image_url to the chosen inference tool.

For audio uploads, use audio/wav for WAV, audio/mpeg for MP3, or audio/flac for FLAC. Existing image and video MIME types are image/jpeg, image/png, image/webp, video/mp4, and video/webm. A MIME type must match the file’s actual encoding. The hosted server does not upload a local path automatically. The application’s ability to read and upload files depends on the client it uses.

Use the local npm bridge

The currently published bridge, @perceptron-ai/mcp-server@0.2.0, supports image and video inputs. It does not support audio inputs, enable_audio_in_video, or reasoning_effort. Version 0.3.0 with these additions is forthcoming. Use the direct hosted connection above for those features until that version is published.

With Node.js 24 or later installed, add a local server using the command npx and arguments -y, @perceptron-ai/mcp-server@0.2.0. Supply PERCEPTRON_API_KEY in its environment. For clients that use an mcpServers configuration:

{
  "mcpServers": {
    "perceptron": {
      "command": "npx",
      "args": ["-y", "@perceptron-ai/mcp-server@0.2.0"],
      "env": {
        "PERCEPTRON_API_KEY": "YOUR_API_KEY"
      }
    }
  }
}

Replace YOUR_API_KEY using your client’s private configuration. Restart or reconnect the server after changing its environment. The bridge connects to the hosted MCP endpoint above and uses the same hosted model default.

Call a bridge tool

Version 0.2.0 exposes question, caption, detect, and ocr, plus a list_models helper. Unlike the hosted schema, the bridge requires an explicit modality of "image" or "video" for question, caption, and detect. ocr remains image-only and requires image_url and mode. For example, call question with:

{
  "media_url": "https://raw.githubusercontent.com/perceptron-ai-inc/perceptron/main/cookbook/_shared/assets/capabilities/qna/studio_scene.webp",
  "modality": "image",
  "model": "perceptron-mk1.5",
  "question": "Describe this coastal scene and the visible objects.",
  "max_tokens": 1024
}

The bridge accepts optional generation settings including max_tokens, temperature, and the older reasoning boolean. Use the hosted connection to set reasoning_effort. For example, call ocr with these arguments, replacing the path with your receipt image:

{
  "image_url": "/absolute/path/to/receipt.png",
  "mode": "markdown",
  "model": "perceptron-mk1.5",
  "max_tokens": 2048
}

OCR modes are plain, markdown, and html. Choose a mode explicitly.

Supply a local file

The bridge accepts HTTP(S) URLs, supported base64 data URLs, and local image or video paths. Prefer an absolute path: a relative path is resolved from the server process’s working directory, which may differ from your project directory. For a local path, the bridge reads the file and uploads its bytes before making the vision request. This keeps the base64 payload out of the assistant’s conversation, but the media is still sent to the hosted service. A remote URL must be accessible to that service.

Handle results

For both connection methods, inspect a tool’s isError flag before treating its content as a successful answer. Results contain text and may include grounded annotations. Use the chat API when you need multi-asset conversation histories, native tracking, or streaming. MCP vision tools are separate from Mk1.5’s native function calling. To give Mk1.5 functions from your own application, declare them in a chat request’s tools field and execute its returned calls in your application.

Troubleshoot the connection

A hosted connection fails: check that the client supports Streamable HTTP and sends the bearer authorization header to /mcp.
The local bridge does not start: check node --version is 24 or later, npx is on the client’s executable path, and PERCEPTRON_API_KEY is set in the server environment.
The wrong model answers: inspect the actual tool arguments and include model: "perceptron-mk1.5" in each call.
Audio or effort settings do not work through the bridge: use the hosted connection; bridge version 0.2.0 does not expose these controls.
A local file fails: hosted calls need a URL or data URL. For the bridge, use an existing absolute path readable by its process, with a file extension matching its media format.
The service returns an error: inspect the tool error and use Error messages to distinguish request problems from transient failures.

See the MCP server repository for package details and client-specific configuration examples.

Get Started

Capabilities

Developer Guides

Scaling & deployment

Best practices

Before you begin

Create an API key

Join Discord

Connect directly to hosted MCP

Select a model

Choose a hosted tool

Ask about audio

Include a video’s soundtrack

Set reasoning effort

Upload media for a hosted call

Use the local npm bridge

Call a bridge tool

Supply a local file

Handle results

Troubleshoot the connection

​Before you begin

Create an API key

Join Discord

​Connect directly to hosted MCP

​Select a model

​Choose a hosted tool

​Ask about audio

​Include a video’s soundtrack

​Set reasoning effort

​Upload media for a hosted call

​Use the local npm bridge

​Call a bridge tool

​Supply a local file

​Handle results

​Troubleshoot the connection

Before you begin

Connect directly to hosted MCP

Select a model

Choose a hosted tool

Ask about audio

Include a video’s soundtrack

Set reasoning effort

Upload media for a hosted call

Use the local npm bridge

Call a bridge tool

Supply a local file

Handle results

Troubleshoot the connection