> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openp.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Rate limits & concurrency

> How to achieve stable high QPS under OpenPAI's rate limits

OpenPAI sets RPM / TPM / concurrency ceilings per user / token at the gateway.
High-QPS scenarios require throttling and retries on the client. For the full rules, see [Rate limits](/en/api-reference/rate-limits).

## Simple client throttling (Python)

Use `asyncio.Semaphore` to control concurrency:

```python theme={null}
import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(api_key="sk-...", base_url="https://openp.ai/v1")
sem = asyncio.Semaphore(32)  # at most 32 concurrent requests

async def call(prompt: str):
    async with sem:
        return await client.chat.completions.create(
            model="gpt-5.5",
            messages=[{"role": "user", "content": prompt}],
        )

tasks = [call(f"task {i}") for i in range(1000)]
results = await asyncio.gather(*tasks)
```

## Exponential backoff

```python theme={null}
import asyncio, random

async def call_with_retry(prompt, max_retries=5):
    delay = 1
    for attempt in range(max_retries):
        try:
            return await client.chat.completions.create(
                model="gpt-5.5",
                messages=[{"role": "user", "content": prompt}],
            )
        except openai.RateLimitError as e:
            retry_after = float(e.response.headers.get("Retry-After", delay))
            await asyncio.sleep(retry_after + random.uniform(0, 0.5))
            delay = min(delay * 2, 30)
    raise RuntimeError("max retries exceeded")
```

## Token-bucket rate limiting

`aiolimiter` implements a client-side token bucket in Python:

```python theme={null}
from aiolimiter import AsyncLimiter

# 200 requests per minute
rate = AsyncLimiter(max_rate=200, time_period=60)

async def call(prompt):
    async with rate:
        return await client.chat.completions.create(...)
```

## Use the response headers

Every response carries remaining-quota headers, enabling adaptive throttling:

```python theme={null}
remaining_req = int(resp.response.headers.get("x-ratelimit-remaining-requests", "100"))
remaining_tok = int(resp.response.headers.get("x-ratelimit-remaining-tokens", "1000000"))

# proactively sleep when remaining < 10%
if remaining_req < 20:
    await asyncio.sleep(2)
```

## Batching tips

* **Merge short requests**: concatenate several independent prompts into one large request (use a system prompt to delimit tasks).
* **Embed many texts at once**: `input` accepts an array, up to \~2048 items per call.
* **Rerank many documents at once**: `documents` can hold tens to hundreds at a time.
* **Streaming + early stop**: if the answer length is controllable, have the model output a JSON header first and use a stop sequence to end early on the client.

## Increase your quota

If the strategies above still can't meet your needs, consider:

1. **Top up** to raise your account level for higher quotas.
2. Apply for a dedicated quota and isolated channel via **enterprise sales**.

## Monitoring

* Console → **Usage statistics**: watch daily peak RPM / TPM.
* Client logs: print the `Retry-After` and `x-ratelimit-*` headers to locate the rate-limit bottleneck.
* Integrate Prometheus / Grafana for your own monitoring: report spend / tokens / latency for each call.