Skip to main content
Use Cases
  • Analyzing uploaded images or PDFs without a separate preprocessing pipeline.
  • Generating images from text prompts through the same API and key.
  • Transcribing audio recordings or voice inputs for downstream processing.
  • Extracting structured data from scanned documents, receipts, or screenshots.

The AI Gateway supports all input and output modalities through a single OpenAI-compatible API, including the Responses API. All endpoints share the same base URL, authentication, and AI Gateway features: fallbacks, caching, load balancing, and retries. All endpoints use the same base URL and authentication:
This page covers sending and generating each modality end to end. Which models accept image, PDF, or audio input, the content part contract, and URL versus base64 transport are owned by Sending files to models. For the catalog by endpoint, see Supported Models or the Providers page.

Image input

Analyze images alongside text. Pass image URLs or base64-encoded files in chat/completions messages.

PDF input

Send PDF documents for extraction and analysis. Supported natively by compatible models.

Image generation

Generate, edit, and vary images using DALL-E 2, DALL-E 3, and GPT Image 1.

Audio

Convert text to speech, transcribe audio files, and translate audio to English.

Image input

Analyze images alongside text using POST /v3/router/responses or POST /v3/router/chat/completions. Pass images as public URLs or base64-encoded data in the message content array. For the file lifecycle, see the Files API.

Supported formats

Detail levels

Set detail in the image_url object:

Patterns

Multiple images:
OCR and text extraction:
Structured output:

Limitations

PDF input

Send PDF documents directly in messages for analysis and content extraction using POST /v3/router/responses or POST /v3/router/chat/completions.
PDF input support varies by model. See the Supported Models page and check your provider’s documentation for PDF capability. For the file lifecycle and for turning uploaded documents into knowledge base datasources, see the Files API.
The input_file and file content part fields are specified once, in Sending files to models.

Use cases

Limitations

Image generation

Generate images from a text prompt using POST /v3/router/images/generations.
For the full and up-to-date list of supported image models, see Image models on the Supported Models page.

Parameters

Set response_format to url to receive a hosted image link, or b64_json to receive the image inline as a base64-encoded string.
Streaming image generation (stream: true) has these constraints:
  • n must be 1
  • response_format is not supported
  • size must be 1024x1024, 1024x1536, 1536x1024, or auto
Requests outside these constraints return a 400 error before the stream opens.

Image editing

Modify an existing image using a prompt and an optional mask with POST /v3/router/images/edits.

Image variations

Generate variations of an existing image with POST /v3/router/images/variations. See Image models for which models support variations.

Fallbacks and reliability

Image endpoints support the same fallbacks and retry parameters as chat completions:
TypeScript

Audio

The AI Gateway exposes three OpenAI-compatible audio endpoints. All support fallbacks, load balancing, and retries.

Text to speech

Convert text to audio using POST /v3/router/audio/speech.
For the full and up-to-date list of TTS models, see Text-to-Speech models on the Supported Models page.
Streaming: Process audio chunks in real time as they arrive, useful for low-latency playback pipelines.
Parameters: Voices:

Transcription

Transcribe an audio file to text using POST /v3/router/audio/transcriptions.
For the full and up-to-date list of transcription models, see Speech-to-Text models on the Supported Models page.
Parameters:

Translation

The OpenAI translation endpoint only supports openai/whisper-1. gpt-4o-transcribe and gpt-4o-mini-transcribe do not support translation.
Transcribe and translate audio to English using POST /v3/router/audio/translations. The output is always in English regardless of the source language.
Translation supports the same response_format and temperature parameters as transcription.