> ## Documentation Index
> Fetch the complete documentation index at: https://docs.perceptron.inc/llms.txt
> Use this file to discover all available pages before exploring further.

# Audio understanding

> Understand speech and sounds, choose audio inputs, and manage usage and limits.

Perceptron Mk1.5 accepts audio in chat-completion requests and returns text. Use it to transcribe speech, summarize a recording, or answer questions about audible events. Audio can appear alongside text, images, and videos in the same conversation. The model does not generate audio.

| Task                                                                      | Example instruction                                                            |
| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| Audio understanding                                                       | “Describe the audible events in order, then summarize any speech.”             |
| [Audio Q\&A](/perceptron-mk1.5/capabilities/audio-qa)                     | “What reason does the speaker give for delaying the delivery?”                 |
| [Audio transcription](/perceptron-mk1.5/capabilities/audio-transcription) | “Transcribe the speech. Mark unclear words as \[unclear] instead of guessing.” |
| [Audio clipping](/perceptron-mk1.5/capabilities/audio-clipping)           | “Locate the first audible alarm and return its start and end times.”           |

For a video's soundtrack, use [video with audio](/perceptron-mk1.5/capabilities/video-understanding#analyze-video-soundtracks). Standalone audio inputs do not require `enable_audio_in_video`.

## Understand a local recording

Install `perceptron>=0.4.0`, set `PERCEPTRON_API_KEY` as shown in the [quickstart](/perceptron-mk1.5/index), and put a short WAV recording at `./recording.wav`. The `audio()` helper detects WAV, MP3, or FLAC from local file contents and encodes the recording inline. It does not upload the file to the Files API.

```python theme={null}
import os

from perceptron import Client, audio


def print_audio_usage(usage):
    if usage is None:
        print("Usage was not included in this response.")
        return
    print(f"Input tokens: {usage.prompt_tokens}")
    print(f"Output tokens: {usage.completion_tokens}")
    if usage.audio_tokens is None:
        print("Audio token breakdown was not reported.")
    else:
        print(f"Audio tokens, included in input: {usage.audio_tokens}")


client = Client(
    provider="perceptron",
    api_key=os.environ["PERCEPTRON_API_KEY"],
)
audio_part = audio("recording.wav")
messages = [{
    "role": "user",
    "content": [
        audio_part,
        {"type": "text", "text": (
            "Describe the audible events in order, then summarize any speech. "
            "Distinguish what you can hear from guesses about its source, "
            "and say when a sound or word is unclear."
        )},
    ],
}]
response = client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=messages,
    reasoning_effort="none",
    max_completion_tokens=2048,
)
choice = response.choices[0]
if choice.finish_reason != "stop" or choice.message.tool_calls:
    raise RuntimeError(f"Expected a completed answer, got: {choice.finish_reason}")
print(choice.message.content or "")
print_audio_usage(response.usage)
```

Check `choices[0].finish_reason`: `stop` indicates a completed answer, while `length` means the output may be incomplete. Choose an output budget appropriate to the requested detail. For word-for-word speech text, use the [transcription guide](/perceptron-mk1.5/capabilities/audio-transcription).

On the wire, the helper sends local audio as `input_audio`: `data` is **bare base64**, without a `data:` prefix, and `format` is `wav`, `mp3`, or `flac`. The API accepts these format names case-insensitively. Renaming a file does not convert its encoding.

## Choose an audio input

The three input forms all belong in a **user message's content array**. They are not supported as audio content in system, assistant, or tool-result messages.

| Content-part type | Use it for                                                                              |
| ----------------- | --------------------------------------------------------------------------------------- |
| `input_audio`     | A short local recording encoded as bare base64, as above.                               |
| `audio_url`       | An HTTP(S) URL accessible to the API, or a base64 data URL.                             |
| `audio_file_id`   | An audio file already uploaded through the [Files API](/perceptron-mk1.5/guides/files). |

All three accept WAV, MP3, and FLAC. For uploads and data URLs, use `audio/wav`, `audio/mpeg`, and `audio/flac`, respectively. The Files API also accepts aliases such as `audio/x-wav` and `audio/x-flac`. For URL inputs, the actual container determines the format; an incorrect MIME label does not convert unsupported audio.

### Send a remote URL or data URL

For a remotely hosted recording, replace `audio_part` in the first example with `audio("https://your-host.example/recording.wav")` before constructing `messages`. Use your actual HTTP(S) URL, accessible to the API; a path such as `/home/me/recording.wav` is a local file, not a remote URL. The SDK passes remote URLs through without downloading the recording itself.

The helper also accepts an existing data URL. This fragment uses the same local recording:

```python theme={null}
import base64
from pathlib import Path

encoded = base64.b64encode(Path("recording.wav").read_bytes()).decode("ascii")
audio_part = audio("data:audio/wav;base64," + encoded)
```

Both inline forms count toward the [20 MiB JSON request limit](/perceptron-mk1.5/guides/scaling). Use a remote URL or an uploaded file ID for larger recordings.

### Reference an uploaded audio file

Upload the recording through [Files](/perceptron-mk1.5/guides/files), then pass the returned file object to `audio()`. Replace the `audio_part` assignment in the first example with:

```python theme={null}
uploaded = client.files.upload("recording.wav")
audio_part = audio(uploaded)
```

For a file uploaded earlier, use `audio(file_id="file-...")` with its actual ID. The helper emits an `audio_file_id` content part. The file must belong to the organization making the request. Uploading once saves repeated transfer of the file bytes; the recording still contributes input tokens each time it is analyzed. See the [Files guide](/perceptron-mk1.5/guides/files) for upload limits and deletion.

## Stream the text response

Streaming delivers generated text incrementally; it does not turn this endpoint into a live microphone stream. Supply the recording with the request as above.

Run this after defining `client`, `messages`, and `print_audio_usage` in the Python example. The code consumes the whole stream, including a possible final usage chunk with an empty `choices` array:

```python theme={null}
with client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=messages,
    reasoning_effort="none",
    max_completion_tokens=1024,
    stream=True,
    stream_options={"include_usage": True},
) as stream:
    for chunk in stream:
        for choice in chunk.choices:
            if choice.delta is not None:
                if choice.delta.tool_calls:
                    raise RuntimeError("Expected a text answer, but received a tool call")
                print(choice.delta.content or "", end="", flush=True)
    response = stream.get_final_completion()

if not response.complete or response.finish_reason != "stop" or response.tool_calls:
    raise RuntimeError(f"Incomplete answer: {response.finish_reason}")
print()
print_audio_usage(response.usage)
```

The SDK assembles the final completion, including usage from a chunk with no choices. It raises on API errors or a stream that ends before its completion marker. If iteration raises or the final completion is unsuccessful, treat displayed text as incomplete.

## Use the command line

The SDK installation includes the `perceptron` command. With `PERCEPTRON_API_KEY` set, ask about the same local recording:

```bash theme={null}
perceptron question recording.wav \
  "Describe the audible events in order, then summarize any speech." \
  --provider perceptron --model perceptron-mk1.5 \
  --reasoning-effort none --format json
```

For a local video such as `recording.mp4`, explicitly enable its soundtrack:

```bash theme={null}
perceptron question recording.mp4 \
  "What happens in this video, and what speech or sounds accompany it?" \
  --provider perceptron --model perceptron-mk1.5 \
  --reasoning-effort high --audio-in-video --format json
```

`--audio-in-video` enables soundtrack processing, including encoded silence; `--no-audio-in-video` explicitly disables it. Omitting both leaves the server default of disabled. These flags do not affect standalone audio inputs. JSON output includes `finish_reason` and reported usage; check completion before relying on the answer. Use the Python API when you need to set an explicit output budget.

## Understand usage and limits

`usage.prompt_tokens_details.audio_tokens` reports the audio portion of `prompt_tokens`. It includes standalone recordings and video soundtracks when enabled. **Do not add it to `prompt_tokens` again.** For chat completions, an explicit `0` reports no audio tokens; a missing or null breakdown means that count was not reported. Multilook also uses zero when the upstream breakdown is absent, so its zero does not prove audio was skipped. When several media inputs are present, this aggregate does not identify which individual soundtracks contributed audio.

The current **Perceptron Mk1.5** deployment accepts at most **16,384 audio tokens per item**. Audio uses approximately **750 tokens per minute**, making that roughly **21.8 minutes per recording**. Treat this as an estimate: the enforced token count is authoritative, and the limit is model-specific. Video soundtracks use the same per-item audio budget.

The per-item limit is separate from the full [context and output limits](/perceptron-mk1.5/models/perceptron-mk1.5#specifications). All recordings, visual inputs, text, history, and requested output must fit the context together. Base64 request size and Files upload limits are separate constraints too. Each standalone audio content part consumes one of the request's 256 media units; follow [asset ordering](/perceptron-mk1.5/guides/multiple-assets) when combining modalities.

An over-limit item returns HTTP 400 with `error.code: "audio_token_limit_exceeded"`. Audio is rejected rather than truncated; shorten or split it. The error message reports the active limit, and `error.param` can be null. This rejection occurs before streaming begins, so a streaming request can receive a JSON error instead of an event stream. See [Error messages](/perceptron-mk1.5/guides/error-messages) and [Tokenization](/perceptron-mk1.5/guides/tokenization).

## Ask several questions about the same recording

[Multilook](/perceptron-mk1.5/guides/multilook) accepts the same audio parts in shared user context or structured prompts. Use it for independent questions about one recording, such as a summary and a list of stated decisions. Audio usage is reported in the call-level `usage.prompt_tokens_details.audio_tokens`; per-prompt usage contains completion tokens only.

Inspect every result: an audio-limit failure can appear in `results[].error` while the Multilook HTTP response is 200. Request-level validation errors still fail the request. For an interactive follow-up conversation or a tool workflow, use ordinary chat completions.