Rate Limiting & Backoff

TMT implements a multi-layer strategy to stay within API rate limits and recover gracefully from transient failures.

Layer 1 — Fixed Rate Limit Delay

When --rate-limit-ms <N> is provided, TmtClient inserts a mandatory sleep of at least N milliseconds between any two consecutive requests. This is enforced by a Mutex<Instant> (see TMT API Client).

Layer 2 — Exponential Backoff with Jitter

AsyncGlobalBackoffState tracks the history of failures and calculates a retry delay using jittered exponential backoff:

delay = min(BASE_DELAY * 2^failures, MAX_DELAY) + random_jitter

• Streak tracking — consecutive failures increase the exponent; a successful request resets the streak.
• Jitter — randomness is added to prevent thundering-herd problems when many tasks hit the API simultaneously.

Layer 3 — Retry-After Header Parsing

When the API responds with HTTP 429 (Too Many Requests), the response may include a Retry-After header specifying how many seconds to wait. TMT parses this header and uses its value as the minimum delay before the next attempt, overriding the backoff calculation if the server-supplied value is larger.

HTTP 429
Retry-After: 30

TMT will wait at least 30 seconds before retrying that request.

--max-retries

Non-rate-limit failures (e.g., network timeouts, HTTP 5xx) are retried up to --max-retries times (default: 4). Rate-limit (429) failures use backoff and do not count against this limit — they are retried indefinitely until the backoff delay is served.

Summary

Request fails
    │
    ├─ HTTP 429 ──▶  parse Retry-After ──▶  sleep ──▶  retry (no limit)
    │
    └─ Other error
           │
           ├─ retries remaining? ──▶  exponential backoff + jitter ──▶  retry
           │
           └─ retries exhausted ──▶  AppError::Network (propagated up)