Rate Limits

Cap requests per minute, tokens per minute, and in-flight concurrency per user path, provider, or model.

Overview

Rate limits cap request rate, token rate, and in-flight concurrency. Every rule has a scope that decides what it protects:

• User path (consumer control) — e.g. /team/alpha can send 100 requests per minute, consume 50,000 tokens per minute, and hold 10 requests in flight, across everything under that path.
• Provider (upstream protection) — e.g. send provider openai at most 500 requests per minute across all consumers and models, so the gateway never blows through your provider quota.
• Model — e.g. openai/gpt-4o may consume 90,000 tokens per minute; a bare id like gpt-4o caps the model across every provider.

The scopes behave differently on breach. A user path over its limit gets 429 with an OpenAI-shaped error, an accurate Retry-After header, and x-ratelimit-* headers — switching models would not help, the consumer is the subject. A saturated provider or model is instead treated like an unavailable target: virtual-model load balancing and failover route around it while another target has capacity, and only when no viable target remains does the client get 429.

Limits apply to every model endpoint: chat completions, responses, messages, embeddings, audio, passthrough, realtime sessions, and batch submission.

Use rate limits for traffic control and budgets for spend control: budgets cap dollars over hours-to-months from durable usage records, rate limits cap request and token velocity over seconds-to-a-day with fast in-memory counters.

Enable rate limits

Rate limits are enabled by default and cost nothing until rules exist:

RATE_LIMITS_ENABLED=true

Token limits (max_tokens) are counted from usage records, so they require usage tracking:

USAGE_ENABLED=true

Request and concurrency limits work without usage tracking. If token rules are configured while usage tracking is off, GoModel starts and logs a warning; the token limits are not enforced.

Create rate limits

In the dashboard:

Rate Limits -> Create Rate Limit

Pick the scope (user path, provider, or model), the subject, a period, and the caps. Dashboard-created rules are marked manual. Rules loaded from YAML or environment variables are marked config and are read-only in the dashboard.

The Models page has a gauge button on every model row and provider header that opens the same limits for that subject in context: the model’s own rules, the provider rules it shares, and the global (root user-path) rules — each with live usage — plus add and edit actions.

Editing a rule’s scope, subject, or period moves it: the rule is recreated under the new key and its live counters restart. Editing only the caps keeps the counters.

Seed rules from YAML:

rate_limits:
  enabled: true
  user_paths:
    - path: "/team/alpha"
      limits:
        - period: "minute"
          max_requests: 100
          max_tokens: 50000
        - period: "day"
          max_requests: 10000
        - period: "concurrent"
          max_requests: 10
  providers:
    - name: "openai"
      limits:
        - period: "minute"
          max_requests: 500
        - period: "concurrent"
          max_requests: 50
  models:
    - model: "openai/gpt-4o"
      limits:
        - period: "minute"
          max_tokens: 90000

Or use environment variables. User-path rules use SET_RATE_LIMIT_<PATH> (the suffix maps to a user path exactly like SET_BUDGET_*; __ separates path segments); provider rules use SET_PROVIDER_RATE_LIMIT_<NAME> (underscores become hyphens, like provider-instance env vars):

SET_RATE_LIMIT_TEAM__ALPHA="rpm=100,tpm=50000,rpd=10000,concurrent=10"
SET_RATE_LIMIT_="rpm=1000"
SET_PROVIDER_RATE_LIMIT_OPENAI="rpm=500,concurrent=50"

Compact names map to periods: rpm/tpm per minute, rph/tph per hour, rpd/tpd per day, and concurrent for the in-flight cap. A JSON array of rule objects is also accepted for custom windows:

SET_RATE_LIMIT_TEAM__ALPHA='[{"period_seconds":300,"max_requests":50}]'

Model rules have no env form (model ids are not env-name safe); declare them in YAML or through the admin API and dashboard.

Supported named periods:

Period	Seconds	Meaning
minute	60	requests/tokens per minute
hour	3600	requests/tokens per hour
day	86400	requests/tokens per day
concurrent	0	max in-flight requests right now

Custom windows use period_seconds directly.

How matching and counting work

Rule paths apply to the configured path and its descendants, exactly like budgets: a rule on /team matches /team/app but not /team-alpha. A rule on / applies to all traffic.

Provider rules match the resolved provider instance by name. Model rules match the executed model: a provider-qualified subject (openai/gpt-4o) covers one provider’s model, a bare subject (gpt-4o) covers that model on any provider. Matching is case-insensitive.

A rule owns one shared counter for its whole subject. A 100 rpm rule on /team allows 100 requests per minute across everything under /team combined; a 500 rpm rule on provider openai is shared by every consumer and model routed there. For per-key limits, bind each managed API key to its own user path and define rules there.

All matching rules are checked; the first exhausted limit rejects the request.

Provider and model limits route around saturation

A saturated provider or model behaves like one with stale inventory:

• Virtual-model load balancing skips the saturated target and picks the next available one, so an alias with several targets keeps serving. When every target is saturated, the alias resolves to its first target anyway so the client gets the honest 429 with Retry-After (or failover takes over) rather than an unavailable-model error, and the alias stays listed in /v1/models throughout.
• Failover rules take over for a saturated primary route: the primary provider is never called (it would happily serve the request and defeat the limit), the sweep starts immediately, and rate-saturated candidates are skipped along the way.
• Direct requests with no failover configured receive 429 with an honest Retry-After — the same answer the provider itself would eventually give, minus the wasted upstream call. Consumer (user path) breaches always return 429: switching targets cannot relieve them.

Token accounting charges the provider and model that actually executed the request (recorded on the usage entry), so the windows stay correct under aliasing and failover. Batch submissions skip provider/model rules — the batch file can mix models, so only user-path rules apply at submission.

Details worth knowing:

• Request windows use a sliding-window estimate, so bursts cannot double up at window boundaries.
• Token usage is counted after each response completes (streams included). One request can overshoot a token window before the counter catches up — the same behavior as other gateways that count tokens from responses.
• Response-cache hits are served before enforcement and count nothing.
• A realtime session counts as one request and holds one concurrency slot for the whole session. A batch submission counts as one request and holds no concurrency slot.
• Counters live in memory, per gateway instance: with N replicas the effective limit is about N times the configured value, and counters reset on restart. Use budgets for durable, cross-instance control.

Client behavior

Requests over a limit receive:

{
  "error": {
    "type": "rate_limit_error",
    "message": "rate limit exceeded for /team/alpha: minute request limit of 100 reached",
    "code": "rate_limit_exceeded",
    "param": null
  }
}

with status 429 and a Retry-After header holding the seconds until a retry would actually be admitted under the sliding window (which can extend past the next window boundary). OpenAI SDKs back off automatically on this shape.

Successful responses on limited paths carry OpenAI-style headers from the most-constrained matching rule:

x-ratelimit-limit-requests: 100
x-ratelimit-remaining-requests: 97
x-ratelimit-reset-requests: 42
x-ratelimit-limit-tokens: 50000
x-ratelimit-remaining-tokens: 31200
x-ratelimit-reset-tokens: 42

Admin API

curl -H "Authorization: Bearer $GOMODEL_MASTER_KEY" \
  http://localhost:8080/admin/rate-limits

Create or update a rule (user_path is shorthand for "scope": "user_path", "subject": ...):

curl -X PUT http://localhost:8080/admin/rate-limits \
  -H "Authorization: Bearer $GOMODEL_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "user_path": "/team/alpha",
    "limit_key": {"period": "minute"},
    "max_requests": 100,
    "max_tokens": 50000
  }'

Provider and model rules name their subject explicitly:

curl -X PUT http://localhost:8080/admin/rate-limits \
  -H "Authorization: Bearer $GOMODEL_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "scope": "provider",
    "subject": "openai",
    "limit_key": {"period": "minute"},
    "max_requests": 500
  }'

Delete a rule:

curl -X DELETE http://localhost:8080/admin/rate-limits \
  -H "Authorization: Bearer $GOMODEL_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"scope": "provider", "subject": "openai", "limit_key": {"period": "minute"}}'

Reset live counters for one rule (POST /admin/rate-limits/reset-one with {"scope": ..., "subject": ..., "period": ...}) or for all rules (POST /admin/rate-limits/reset with {"confirmation": "reset"}). Editing a rule never resets its counters; deleting one does.

See Admin Endpoints for the endpoint list.