Skip to main content
The major multimodal models (GPT-4o, Claude 4.x, Gemini 2.5) all support image input. You can have the model describe an image, recognize objects, do OCR, generate alt text, or answer questions about an image.

​
Supported models

ModelOpenAI protocolClaude protocolGemini protocol
gpt-5.5, gpt-5.4-mini✅——
o3 series✅——
claude-haiku-4-5, sonnet, opus✅ (auto-converted)✅—
gemini-3.1-pro-preview / flash-lite✅ (auto-converted)—✅
qwen-vl-*✅——

​
OpenAI protocol

​
Via URL

{
  "model": "gpt-5.5",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "What animal is in the image?"},
      {"type": "image_url", "image_url": {
        "url": "https://example.com/cat.jpg",
        "detail": "high"
      }}
    ]
  }]
}
detail values: low (fast, cheap) / high (high resolution, expensive) / auto (default).

​
Via base64

import base64
with open("photo.jpg", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

resp = client.chat.completions.create(
    model="gpt-5.5",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "Describe this image"},
        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}}
    ]}]
)
Supported formats: image/png / jpeg / gif / webp.

​
Claude protocol

{
  "model": "claude-opus-4-8",
  "max_tokens": 1024,
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "OCR this image"},
      {"type": "image", "source": {
        "type": "url",
        "url": "https://example.com/doc.png"
      }}
    ]
  }]
}
Or base64: 
{"type": "image", "source": {
  "type": "base64",
  "media_type": "image/png",
  "data": "<base64>"
}}

​
Gemini protocol

{
  "contents": [{
    "role": "user",
    "parts": [
      {"text": "What's happening in the image?"},
      {"inline_data": {"mime_type": "image/png", "data": "<base64>"}}
    ]
  }]
}
Gemini also supports file_data (an already-uploaded file) plus PDF / video parts: 
{"file_data": {"mime_type": "application/pdf", "file_uri": "https://..."}}

​
Multiple images

Put multiple image objects in the same message: 
{
  "role": "user",
  "content": [
    {"type": "text", "text": "Compare the differences between the two images"},
    {"type": "image_url", "image_url": {"url": "https://.../a.jpg"}},
    {"type": "image_url", "image_url": {"url": "https://.../b.jpg"}}
  ]
}
  • GPT-4o: up to ~50 images per call.
  • Claude: up to ~20 images per call.
  • Gemini: up to ~3000 images or 1 long video per call.

​
Billing

Image input is converted to tokens by a pixels → tokens strategy and counted in prompt_tokens; the exact rules depend on the upstream:
  • OpenAI: detail: low ≈ 85 tokens/image; high is estimated as 512×512 tiles × 170 tokens.
  • Claude: (width × height) / 750 tokens; 1024×1024 ≈ 1400 tokens.
  • Gemini: a fixed 258 tokens per image.
The console log shows the image portion of the actual prompt_tokens.

​
Best practices

  • Keep images clear: blur or low resolution significantly lowers recognition accuracy.
  • Resize when necessary: scaling large images to a 1024-2048 long edge saves tokens.
  • Ask specific questions: don’t just say “describe the image” — asking specific questions (objects / positions / numbers / text / colors) works better.
  • Use specialized models for OCR: for document OCR, prefer claude-opus-4-8 or gemini-3.1-pro-preview; for Chinese documents, consider qwen-vl-max.