> ## Documentation Index > Fetch the complete documentation index at: https://gomodel-feat-mcp-server.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Resilience > Tune retries and the per-provider circuit breaker, and learn when to reach for cross-model failover instead. GoModel wraps every upstream provider call with two resilience layers: 1. **Retry with exponential backoff** — repeats a failed request against the same provider with growing delays. 2. **Circuit breaker** — short-circuits calls to a provider that has been failing repeatedly, then probes once the timeout elapses. Both layers apply per provider. They do not switch to a different model or provider on failure. For cross-model failover, see [Failover](/features/failover). ## Defaults The defaults are tuned to be safe for most deployments. Override only what you need. | Setting | Default | Notes | | ------------------- | ------- | ---------------------------------------------- | | `max_retries` | `3` | Maximum retry attempts per request | | `initial_backoff` | `1s` | First retry wait | | `max_backoff` | `30s` | Upper cap on retry wait | | `backoff_factor` | `2.0` | Exponential multiplier between retries | | `jitter_factor` | `0.1` | Random jitter as a fraction of the backoff | | `failure_threshold` | `5` | Consecutive failures before the circuit opens | | `success_threshold` | `2` | Consecutive successes to close it again | | `timeout` | `30s` | How long the circuit stays open before probing | To disable a layer, zero out its trigger: `max_retries: 0` gives every request exactly one attempt, and `failure_threshold: 0` disables the circuit breaker entirely. Both work globally or per provider. ## What counts as a failure **Retries** fire on transport errors (connection refused, resets, DNS failures) and on `429`, `502`, `503`, and `504` responses. Other statuses — including `500` — are returned to the caller without retrying. Two paths retry less than the table suggests: * **Streaming** requests are never retried once dispatched, because partial data may already have been sent. * **Passthrough** requests are retried only when they are replay-safe: `GET`, `HEAD`, `OPTIONS`, `PUT`, or any request carrying an `Idempotency-Key` header. **The circuit breaker** counts transport errors and `5xx` responses (including non-retried `500`s) as failures. It deliberately ignores: * `429` rate limits — a rate-limited provider is up, so the backpressure signal is passed through instead of opening the circuit. The one exception is a `429` on a half-open probe, which reopens the circuit. * `4xx` responses — the provider answered; the request was at fault. * Requests canceled by the client — a disconnect says nothing about provider health. Client-side *timeouts* do count as failures. While the circuit is open, requests fail fast with a `503` and the message `circuit breaker is open - provider temporarily unavailable`. After `timeout` elapses, a single probe request is let through while concurrent requests keep failing fast; `success_threshold` consecutive successful probes close the circuit, and any probe failure reopens it. The circuit breaker is in-memory and per gateway process: each provider gets its own breaker, state resets on restart, and nothing is shared between replicas. ## Environment Variables These set the global defaults that apply to every provider unless overridden in YAML. ### Retry | Variable | Type | Default | Description | | ----------------------- | -------- | ------- | ------------------------------------------ | | `RETRY_MAX_RETRIES` | int | `3` | Maximum retry attempts per request | | `RETRY_INITIAL_BACKOFF` | duration | `1s` | First retry wait (e.g. `500ms`, `2s`) | | `RETRY_MAX_BACKOFF` | duration | `30s` | Upper cap on retry wait | | `RETRY_BACKOFF_FACTOR` | float | `2.0` | Exponential multiplier between retries | | `RETRY_JITTER_FACTOR` | float | `0.1` | Random jitter as a fraction of the backoff | ### Circuit Breaker | Variable | Type | Default | Description | | ----------------------------------- | -------- | ------- | ---------------------------------------------- | | `CIRCUIT_BREAKER_FAILURE_THRESHOLD` | int | `5` | Consecutive failures before opening | | `CIRCUIT_BREAKER_SUCCESS_THRESHOLD` | int | `2` | Consecutive successes to close again | | `CIRCUIT_BREAKER_TIMEOUT` | duration | `30s` | How long the circuit stays open before probing | ## YAML The same fields are available under the global `resilience:` block, and can be overridden per provider: ```yaml theme={null} resilience: retry: max_retries: 2 initial_backoff: 500ms max_backoff: 10s backoff_factor: 1.5 jitter_factor: 0.05 circuit_breaker: failure_threshold: 3 success_threshold: 1 timeout: 15s providers: anthropic: type: anthropic api_key: ${ANTHROPIC_API_KEY} resilience: retry: max_retries: 5 # Anthropic supports long requests — allow more retries ollama: type: ollama base_url: ${OLLAMA_BASE_URL:-http://localhost:11434/v1} resilience: circuit_breaker: failure_threshold: 10 # local service — tolerate more transient failures timeout: 5s ``` Only fields explicitly listed under a provider's `resilience:` block are overridden. Everything else inherits from the global section, which in turn inherits from the built-in defaults. Per-provider tuning **must** come from YAML. Environment variables set global defaults only — `RETRY_MAX_RETRIES` cannot target a single provider. See [config.yaml gotchas](/advanced/config-yaml#gotchas). ### Worked example Given the YAML above, the effective per-provider settings are: | Provider | max\_retries | failure\_threshold | cb timeout | | --------- | ---------------- | ------------------ | ----------------- | | openai | 2 (global) | 3 (global) | 15s (global) | | anthropic | **5** (override) | 3 (global) | 15s (global) | | ollama | 2 (global) | **10** (override) | **5s** (override) | `anthropic` and `ollama` inherit every field they did not explicitly override. ## Circuit breaker vs. dashboard provider health The dashboard's provider status (Healthy / Degraded / Unhealthy and the "Last checked" timestamp) is **not** the circuit breaker. Circuit breaker state is not shown in the dashboard and recovers on its own within `timeout` (default 30s) once the provider is back; when [metrics](/advanced/configuration#metrics) are enabled it is exported as the `gomodel_circuit_breaker_state` gauge (0 = closed, 1 = half-open, 2 = open). Dashboard health reflects **model discovery**: whether the provider's model inventory could last be fetched. Providers are re-checked: * at startup, * on every model registry refresh, controlled by `CACHE_REFRESH_INTERVAL` (seconds, default `3600` — hourly), * on the fast recheck loop, which re-probes **only** providers whose latest refresh failed, controlled by `PROVIDER_RECHECK_INTERVAL` (`cache.model.recheck_interval` in `config.yaml`; seconds, default `60`; `0` disables), and * on demand, when a request asks for a provider-qualified model (`provider/model`) that is missing from the registry. ## What happens while a provider is down When a provider's refresh fails, its previously discovered models are kept and marked stale, and the dashboard shows the provider as Degraded ("previous inventory is still available"): * **Direct requests** to its models still resolve and are sent to the provider, so callers get an honest `502`/`503` (and manual failover rules can fire) instead of a misleading "model not found". * **Virtual-model redirects** skip the provider's targets, so a load-balanced redirect keeps working through its healthy targets. The fast recheck loop re-probes the provider every `PROVIDER_RECHECK_INTERVAL` seconds, updating "Last checked" and restoring normal routing typically within a minute of the provider coming back. A provider that was already down at startup (nothing discovered yet) shows as Unhealthy until its first successful fetch. ## Failover vs. Resilience The retry and circuit breaker layers stay on a single provider. If you also want GoModel to try a different model or provider when the primary keeps failing, configure manual failover rules. See [Failover](/features/failover).