Audio understanding

Perceptron Mk1.5 accepts audio in chat-completion requests and returns text. Use it to transcribe speech, summarize a recording, or answer questions about audible events. Audio can appear alongside text, images, and videos in the same conversation. The model does not generate audio.

Task	Example instruction
Audio understanding	“Describe the audible events in order, then summarize any speech.”
Audio Q&A	“What reason does the speaker give for delaying the delivery?”
Audio transcription	“Transcribe the speech. Mark unclear words as [unclear] instead of guessing.”
Audio clipping	“Locate the first audible alarm and return its start and end times.”

For a video’s soundtrack, use video with audio. Standalone audio inputs do not require enable_audio_in_video.

Understand a local recording

Install perceptron>=0.4.0, set PERCEPTRON_API_KEY as shown in the quickstart, and put a short WAV recording at ./recording.wav. The audio() helper detects WAV, MP3, or FLAC from local file contents and encodes the recording inline. It does not upload the file to the Files API.

import os

from perceptron import Client, audio


def print_audio_usage(usage):
    if usage is None:
        print("Usage was not included in this response.")
        return
    print(f"Input tokens: {usage.prompt_tokens}")
    print(f"Output tokens: {usage.completion_tokens}")
    if usage.audio_tokens is None:
        print("Audio token breakdown was not reported.")
    else:
        print(f"Audio tokens, included in input: {usage.audio_tokens}")


client = Client(
    provider="perceptron",
    api_key=os.environ["PERCEPTRON_API_KEY"],
)
audio_part = audio("recording.wav")
messages = [{
    "role": "user",
    "content": [
        audio_part,
        {"type": "text", "text": (
            "Describe the audible events in order, then summarize any speech. "
            "Distinguish what you can hear from guesses about its source, "
            "and say when a sound or word is unclear."
        )},
    ],
}]
response = client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=messages,
    reasoning_effort="none",
    max_completion_tokens=2048,
)
choice = response.choices[0]
if choice.finish_reason != "stop" or choice.message.tool_calls:
    raise RuntimeError(f"Expected a completed answer, got: {choice.finish_reason}")
print(choice.message.content or "")
print_audio_usage(response.usage)

Check choices[0].finish_reason: stop indicates a completed answer, while length means the output may be incomplete. Choose an output budget appropriate to the requested detail. For word-for-word speech text, use the transcription guide. On the wire, the helper sends local audio as input_audio: data is bare base64, without a data: prefix, and format is wav, mp3, or flac. The API accepts these format names case-insensitively. Renaming a file does not convert its encoding.

Choose an audio input

The three input forms all belong in a user message’s content array. They are not supported as audio content in system, assistant, or tool-result messages.

Content-part type	Use it for
`input_audio`	A short local recording encoded as bare base64, as above.
`audio_url`	An HTTP(S) URL accessible to the API, or a base64 data URL.
`audio_file_id`	An audio file already uploaded through the Files API.

All three accept WAV, MP3, and FLAC. For uploads and data URLs, use audio/wav, audio/mpeg, and audio/flac, respectively. The Files API also accepts aliases such as audio/x-wav and audio/x-flac. For URL inputs, the actual container determines the format; an incorrect MIME label does not convert unsupported audio.

Send a remote URL or data URL

For a remotely hosted recording, replace audio_part in the first example with audio("https://your-host.example/recording.wav") before constructing messages. Use your actual HTTP(S) URL, accessible to the API; a path such as /home/me/recording.wav is a local file, not a remote URL. The SDK passes remote URLs through without downloading the recording itself. The helper also accepts an existing data URL. This fragment uses the same local recording:

import base64
from pathlib import Path

encoded = base64.b64encode(Path("recording.wav").read_bytes()).decode("ascii")
audio_part = audio("data:audio/wav;base64," + encoded)

Both inline forms count toward the 20 MiB JSON request limit. Use a remote URL or an uploaded file ID for larger recordings.

Reference an uploaded audio file

Upload the recording through Files, then pass the returned file object to audio(). Replace the audio_part assignment in the first example with:

uploaded = client.files.upload("recording.wav")
audio_part = audio(uploaded)

For a file uploaded earlier, use audio(file_id="file-...") with its actual ID. The helper emits an audio_file_id content part. The file must belong to the organization making the request. Uploading once saves repeated transfer of the file bytes; the recording still contributes input tokens each time it is analyzed. See the Files guide for upload limits and deletion.

Stream the text response

Streaming delivers generated text incrementally; it does not turn this endpoint into a live microphone stream. Supply the recording with the request as above. Run this after defining client, messages, and print_audio_usage in the Python example. The code consumes the whole stream, including a possible final usage chunk with an empty choices array:

with client.chat.completions.create(
    model="perceptron-mk1.5",
    messages=messages,
    reasoning_effort="none",
    max_completion_tokens=1024,
    stream=True,
    stream_options={"include_usage": True},
) as stream:
    for chunk in stream:
        for choice in chunk.choices:
            if choice.delta is not None:
                if choice.delta.tool_calls:
                    raise RuntimeError("Expected a text answer, but received a tool call")
                print(choice.delta.content or "", end="", flush=True)
    response = stream.get_final_completion()

if not response.complete or response.finish_reason != "stop" or response.tool_calls:
    raise RuntimeError(f"Incomplete answer: {response.finish_reason}")
print()
print_audio_usage(response.usage)

The SDK assembles the final completion, including usage from a chunk with no choices. It raises on API errors or a stream that ends before its completion marker. If iteration raises or the final completion is unsuccessful, treat displayed text as incomplete.

Use the command line

The SDK installation includes the perceptron command. With PERCEPTRON_API_KEY set, ask about the same local recording:

perceptron question recording.wav \
  "Describe the audible events in order, then summarize any speech." \
  --provider perceptron --model perceptron-mk1.5 \
  --reasoning-effort none --format json

For a local video such as recording.mp4, explicitly enable its soundtrack:

perceptron question recording.mp4 \
  "What happens in this video, and what speech or sounds accompany it?" \
  --provider perceptron --model perceptron-mk1.5 \
  --reasoning-effort high --audio-in-video --format json

--audio-in-video enables soundtrack processing, including encoded silence; --no-audio-in-video explicitly disables it. Omitting both leaves the server default of disabled. These flags do not affect standalone audio inputs. JSON output includes finish_reason and reported usage; check completion before relying on the answer. Use the Python API when you need to set an explicit output budget.

Understand usage and limits

usage.prompt_tokens_details.audio_tokens reports the audio portion of prompt_tokens. It includes standalone recordings and video soundtracks when enabled. Do not add it to prompt_tokens again. For chat completions, an explicit 0 reports no audio tokens; a missing or null breakdown means that count was not reported. Multilook also uses zero when the upstream breakdown is absent, so its zero does not prove audio was skipped. When several media inputs are present, this aggregate does not identify which individual soundtracks contributed audio. The current Perceptron Mk1.5 deployment accepts at most 16,384 audio tokens per item. Audio uses approximately 750 tokens per minute, making that roughly 21.8 minutes per recording. Treat this as an estimate: the enforced token count is authoritative, and the limit is model-specific. Video soundtracks use the same per-item audio budget. The per-item limit is separate from the full context and output limits. All recordings, visual inputs, text, history, and requested output must fit the context together. Base64 request size and Files upload limits are separate constraints too. Each standalone audio content part consumes one of the request’s 256 media units; follow asset ordering when combining modalities. An over-limit item returns HTTP 400 with error.code: "audio_token_limit_exceeded". Audio is rejected rather than truncated; shorten or split it. The error message reports the active limit, and error.param can be null. This rejection occurs before streaming begins, so a streaming request can receive a JSON error instead of an event stream. See Error messages and Tokenization.

Ask several questions about the same recording

Multilook accepts the same audio parts in shared user context or structured prompts. Use it for independent questions about one recording, such as a summary and a list of stated decisions. Audio usage is reported in the call-level usage.prompt_tokens_details.audio_tokens; per-prompt usage contains completion tokens only. Inspect every result: an audio-limit failure can appear in results[].error while the Multilook HTTP response is 200. Request-level validation errors still fail the request. For an interactive follow-up conversation or a tool workflow, use ordinary chat completions.

Get Started

Capabilities

Developer Guides

Scaling & deployment

Best practices

Understand a local recording

Choose an audio input

Send a remote URL or data URL

Reference an uploaded audio file

Stream the text response

Use the command line

Understand usage and limits

Ask several questions about the same recording

​Understand a local recording

​Choose an audio input

​Send a remote URL or data URL

​Reference an uploaded audio file

​Stream the text response

​Use the command line

​Understand usage and limits

​Ask several questions about the same recording

Understand a local recording

Choose an audio input

Send a remote URL or data URL

Reference an uploaded audio file

Stream the text response

Use the command line

Understand usage and limits

Ask several questions about the same recording