Skip to main content
OpenPAI sets RPM / TPM / concurrency ceilings per user / token at the gateway. High-QPS scenarios require throttling and retries on the client. For the full rules, see Rate limits.

​
Simple client throttling (Python)

Use asyncio.Semaphore to control concurrency: 
import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(api_key="sk-...", base_url="https://openp.ai/v1")
sem = asyncio.Semaphore(32)  # at most 32 concurrent requests

async def call(prompt: str):
    async with sem:
        return await client.chat.completions.create(
            model="gpt-5.5",
            messages=[{"role": "user", "content": prompt}],
        )

tasks = [call(f"task {i}") for i in range(1000)]
results = await asyncio.gather(*tasks)

​
Exponential backoff

import asyncio, random

async def call_with_retry(prompt, max_retries=5):
    delay = 1
    for attempt in range(max_retries):
        try:
            return await client.chat.completions.create(
                model="gpt-5.5",
                messages=[{"role": "user", "content": prompt}],
            )
        except openai.RateLimitError as e:
            retry_after = float(e.response.headers.get("Retry-After", delay))
            await asyncio.sleep(retry_after + random.uniform(0, 0.5))
            delay = min(delay * 2, 30)
    raise RuntimeError("max retries exceeded")

​
Token-bucket rate limiting

aiolimiter implements a client-side token bucket in Python: 
from aiolimiter import AsyncLimiter

# 200 requests per minute
rate = AsyncLimiter(max_rate=200, time_period=60)

async def call(prompt):
    async with rate:
        return await client.chat.completions.create(...)

​
Use the response headers

Every response carries remaining-quota headers, enabling adaptive throttling: 
remaining_req = int(resp.response.headers.get("x-ratelimit-remaining-requests", "100"))
remaining_tok = int(resp.response.headers.get("x-ratelimit-remaining-tokens", "1000000"))

# proactively sleep when remaining < 10%
if remaining_req < 20:
    await asyncio.sleep(2)

​
Batching tips

  • Merge short requests: concatenate several independent prompts into one large request (use a system prompt to delimit tasks).
  • Embed many texts at once: input accepts an array, up to ~2048 items per call.
  • Rerank many documents at once: documents can hold tens to hundreds at a time.
  • Streaming + early stop: if the answer length is controllable, have the model output a JSON header first and use a stop sequence to end early on the client.

​
Increase your quota

If the strategies above still can’t meet your needs, consider:
  1. Top up to raise your account level for higher quotas.
  2. Apply for a dedicated quota and isolated channel via enterprise sales.

​
Monitoring

  • Console → Usage statistics: watch daily peak RPM / TPM.
  • Client logs: print the Retry-After and x-ratelimit-* headers to locate the rate-limit bottleneck.
  • Integrate Prometheus / Grafana for your own monitoring: report spend / tokens / latency for each call.