> ## Documentation Index
> Fetch the complete documentation index at: https://docs.perceptron.inc/llms.txt
> Use this file to discover all available pages before exploring further.

# MCP

> Connect an MCP client to Mk1.5 image, video, audio, and reasoning tools.

Perceptron's hosted MCP server lets an AI assistant ask questions about media, generate captions, detect objects, and read text. Connect directly over Streamable HTTP for image, video, audio, and reasoning-effort controls. The optional local npm bridge also uploads local files, with the version limits described below.

## Before you begin

You need an MCP client and a Perceptron API key. Direct hosted connections do not require Node.js. The local npm bridge requires **Node.js 24 or later**.

<CardGroup cols={2}>
  <Card title="Create an API key" icon="key" href="https://platform.perceptron.inc/">
    Get your key from the Perceptron platform
  </Card>

  <Card title="Join Discord" icon="discord" href="https://discord.gg/fgBeaACQzE">
    Get help and see what others are building
  </Card>
</CardGroup>

## Connect directly to hosted MCP

Add a remote server in a client that supports Streamable HTTP and custom authorization headers:

| Setting      | Value                            |
| ------------ | -------------------------------- |
| Transport    | Streamable HTTP                  |
| URL          | `https://api.perceptron.inc/mcp` |
| Header name  | `Authorization`                  |
| Header value | `Bearer YOUR_API_KEY`            |

Replace `YOUR_API_KEY` with your Perceptron key using the client's secret or private configuration. Do not commit credentials with the project. Your MCP client controls when tools can run and any approval prompts it shows.

Connect the server and refresh its tools in your client. The examples in this section are **hosted MCP tool arguments**, not chat-completions request bodies. Hosted calls accept accessible media URLs or supported data URLs; they cannot read a path on your computer.

### Select a model

The hosted service defaults to `perceptron-mk1.5` when `model` is omitted. This replaces the previous Mk1 default; pass `"model": "perceptron-mk1"` to keep the previous model. These examples set `"model": "perceptron-mk1.5"` explicitly so the choice remains stable if the default changes again.

A previous tool call does not select the model for the next call. The model used by your assistant is also independent of the model used by a Perceptron tool. Model discovery is available through MCP resources at `perceptron://models/{modelId}`.

### Choose a hosted tool

| Tool                     | Inputs                                                      | Task controls                                                    |
| ------------------------ | ----------------------------------------------------------- | ---------------------------------------------------------------- |
| `question`               | `media_url`, `question`; `modality`: image, video, or audio | Optional `output_format`: point, box, polygon, or clip           |
| `caption`                | `media_url`; `modality`: image, video, or audio             | Optional `style`: concise (default) or detailed; `output_format` |
| `detect`                 | `media_url`; `modality`: image or video                     | Optional `classes`: a list of object labels                      |
| `ocr`                    | `image_url`, `mode`; image-only                             | `mode`: plain, markdown, or html; optional `prompt`              |
| `generate_upload_urls`   | `files`: file names, MIME types, and byte lengths           | Returns URLs for uploading media                                 |
| `generate_download_urls` | `object_keys` from the upload response                      | Returns media URLs to use in an inference tool                   |

For hosted `question`, `caption`, and `detect` calls, `modality` defaults to `image`. Set it explicitly when sending video or audio. Each inference call accepts one media input. All four inference tools accept optional `model`, `max_tokens`, `temperature`, and `reasoning_effort` settings.

### Ask about audio

Call `question` with `modality: "audio"`. Replace the example URL with an accessible WAV, MP3, or FLAC recording:

```json theme={null}
{
  "media_url": "https://example.com/recording.wav",
  "modality": "audio",
  "model": "perceptron-mk1.5",
  "question": "Summarize the speech and describe any other audible sounds.",
  "reasoning_effort": "high",
  "max_tokens": 2048
}
```

`caption` also accepts audio. `detect` and `ocr` do not accept audio inputs. Omit spatial output formats for an audio-only question.

### Include a video's soundtrack

Video soundtracks are disabled by default. To analyze sound together with frames, set `enable_audio_in_video: true` on a hosted `question` or `caption` call with `modality: "video"`:

```json theme={null}
{
  "media_url": "https://raw.githubusercontent.com/perceptron-ai-inc/perceptron/main/cookbook/_shared/assets/capabilities/video-clipping/mj_shot_short.mp4",
  "modality": "video",
  "model": "perceptron-mk1.5",
  "question": "Describe what happens and what can be heard in this clip.",
  "enable_audio_in_video": true,
  "reasoning_effort": "high",
  "max_tokens": 2048
}
```

This sample contains a soundtrack; replace its URL with your video as needed. Setting the flag to `true` on an image or audio-only input returns a tool error. The flag is available only on `question` and `caption`. The [audio limits and usage rules](/perceptron-mk1.5/capabilities/audio#understand-usage-and-limits) also apply to recordings and soundtracks sent through MCP.

### Set reasoning effort

The four hosted inference tools accept `reasoning_effort`: `none`, `minimal`, `low`, `medium`, or `high`. Any value other than `none` enables reasoning. The older boolean `reasoning` field is deprecated; omit it when using `reasoning_effort`.

The JSON result inside an MCP text block uses `content` for the answer and `reasoning` for reasoning when returned. More reasoning consumes output budget and does not guarantee a more accurate answer. See [Reasoning](/perceptron-mk1.5/capabilities/thinking) for guidance on choosing an effort level.

### Upload media for a hosted call

For a file on your computer, your application can use the hosted upload helpers:

1. Call `generate_upload_urls` with a `files` array. Each entry contains `file_name`, `content_type`, and `content_length` in bytes.
2. Upload the actual bytes to each returned `upload_url` using HTTP `PUT` and the matching `Content-Type`. Upload files sequentially.
3. Call `generate_download_urls` with the returned `object_key` values in its `object_keys` array, then pass a returned `download_url` as `media_url` or `image_url` to the chosen inference tool.

For audio uploads, use `audio/wav` for WAV, `audio/mpeg` for MP3, or `audio/flac` for FLAC. Existing image and video MIME types are `image/jpeg`, `image/png`, `image/webp`, `video/mp4`, and `video/webm`. A MIME type must match the file's actual encoding.

The hosted server does not upload a local path automatically. The application's ability to read and upload files depends on the client it uses.

## Use the local npm bridge

<Note>
  The currently published bridge, `@perceptron-ai/mcp-server@0.2.0`, supports image and video inputs. It does not support audio inputs, `enable_audio_in_video`, or `reasoning_effort`. Version `0.3.0` with these additions is forthcoming. Use the direct hosted connection above for those features until that version is published.
</Note>

With Node.js 24 or later installed, add a local server using the command `npx` and arguments `-y`, `@perceptron-ai/mcp-server@0.2.0`. Supply `PERCEPTRON_API_KEY` in its environment. For clients that use an `mcpServers` configuration:

```json theme={null}
{
  "mcpServers": {
    "perceptron": {
      "command": "npx",
      "args": ["-y", "@perceptron-ai/mcp-server@0.2.0"],
      "env": {
        "PERCEPTRON_API_KEY": "YOUR_API_KEY"
      }
    }
  }
}
```

Replace `YOUR_API_KEY` using your client's private configuration. Restart or reconnect the server after changing its environment. The bridge connects to the hosted MCP endpoint above and uses the same hosted model default.

### Call a bridge tool

Version `0.2.0` exposes `question`, `caption`, `detect`, and `ocr`, plus a `list_models` helper. Unlike the hosted schema, the bridge requires an explicit `modality` of `"image"` or `"video"` for `question`, `caption`, and `detect`. `ocr` remains image-only and requires `image_url` and `mode`.

For example, call `question` with:

```json theme={null}
{
  "media_url": "https://raw.githubusercontent.com/perceptron-ai-inc/perceptron/main/cookbook/_shared/assets/capabilities/qna/studio_scene.webp",
  "modality": "image",
  "model": "perceptron-mk1.5",
  "question": "Describe this coastal scene and the visible objects.",
  "max_tokens": 1024
}
```

The bridge accepts optional generation settings including `max_tokens`, `temperature`, and the older `reasoning` boolean. Use the hosted connection to set `reasoning_effort`.

For example, call `ocr` with these arguments, replacing the path with your receipt image:

```json theme={null}
{
  "image_url": "/absolute/path/to/receipt.png",
  "mode": "markdown",
  "model": "perceptron-mk1.5",
  "max_tokens": 2048
}
```

OCR modes are `plain`, `markdown`, and `html`. Choose a mode explicitly.

### Supply a local file

The bridge accepts HTTP(S) URLs, supported base64 data URLs, and local image or video paths. Prefer an absolute path: a relative path is resolved from the server process's working directory, which may differ from your project directory.

For a local path, the bridge reads the file and uploads its bytes before making the vision request. This keeps the base64 payload out of the assistant's conversation, but the media is still sent to the hosted service. A remote URL must be accessible to that service.

## Handle results

For both connection methods, inspect a tool's `isError` flag before treating its content as a successful answer. Results contain text and may include grounded annotations. Use the [chat API](/perceptron-mk1.5/guides/python/request-basics) when you need multi-asset conversation histories, [native tracking](/perceptron-mk1.5/capabilities/video-tracking), or [streaming](/perceptron-mk1.5/capabilities/video-tracking#stream-a-tracking-answer).

MCP vision tools are separate from Mk1.5's native [function calling](/perceptron-mk1.5/guides/tool-calling). To give Mk1.5 functions from your own application, declare them in a chat request's `tools` field and execute its returned calls in your application.

## Troubleshoot the connection

* **A hosted connection fails:** check that the client supports Streamable HTTP and sends the bearer authorization header to `/mcp`.
* **The local bridge does not start:** check `node --version` is 24 or later, `npx` is on the client's executable path, and `PERCEPTRON_API_KEY` is set in the server environment.
* **The wrong model answers:** inspect the actual tool arguments and include `model: "perceptron-mk1.5"` in each call.
* **Audio or effort settings do not work through the bridge:** use the hosted connection; bridge version `0.2.0` does not expose these controls.
* **A local file fails:** hosted calls need a URL or data URL. For the bridge, use an existing absolute path readable by its process, with a file extension matching its media format.
* **The service returns an error:** inspect the tool error and use [Error messages](/perceptron-mk1.5/guides/error-messages) to distinguish request problems from transient failures.

See the [MCP server repository](https://github.com/perceptron-ai-inc/mcp-server) for package details and client-specific configuration examples.