Relay // reliable LLM API delivery
I rewrote retry-and-failover for LLM APIs in four projects, each time with a new bug. Relay is the version you import instead — same SDK shape, plus auto-retry, provider failover, and a cache that stops you paying twice.
Live stream from a relay account — same shape you see in your dashboard. Newest event types in at the top, older ones fade out of the bottom of the window.
— The reality
01 · Outage
Anthropic real uptime vs 99.9% SLA, 2026 YTD
Anthropic drops thirty minutes. Your agent stops mid-task. Your user closes the tab. Your churn ticks up. Repeat next month when OpenAI has its own bad day.
02 · Rate limit
the status code you see during every demo
Peak hours are exactly when you can't afford retries you don't have. Every 529 you don't catch is a request that quietly dies and a user who quietly leaves.
03 · Boilerplate
times the average AI dev rewrites their retry loop
Every project ships its own retry loop. Half of them get the jitter wrong. None of them failover to a backup provider — the model id wouldn't even match.
04 · Cost
of prompts repeated within the same hour, observed
Same system prompt, same user input — different request id. Without caching, every duplicate burns tokens you'll never see again on a bill that grows linearly.
— The fix
Replace @anthropic-ai/sdk with @relay-api/sdk. Same method signatures, same streaming. Retry, failover and caching become someone else's problem.
01// 47 lines of retry boilerplate, written badly.02async function callLLM(body) {03 let attempt = 0;04 while (attempt < 3) {05 try {06 return await anthropic.messages.create(body);07 } catch (e) {08 attempt++;09 await sleep(2 ** attempt * 1000);10 }11 }12 throw new Error('Gave up');13}14// Still no failover. No cache. No metrics.01// One line. Reliability handled.02const reply = await llm.messages.create(body);0304// What Relay just did for you:05// ✓ exponential backoff with jitter06// ✓ failover Anthropic → OpenAI on 5xx07// ✓ exact + semantic cache lookup08// ✓ streaming preserved end-to-end09// ✓ live metrics in your dashboardminutes Anthropic returned 5xx
Relay calls succeeded via failover to OpenAI
— What you get
When Anthropic returns 5xx, Relay routes the next attempt to OpenAI — or any chain you configure per request. Model ids are translated on the way out, so you keep writing claude-sonnet-4-6 either way.
Decorrelated jitter (the algorithm AWS uses internally) prevents retry storms when a whole region 429s at once. Configurable per request, surfaced on every response.
Exact-match on day one. Semantic caching with embeddings on Hobby and up. Stop paying twice for prompts that arrive twice — most agent loops hit the same system message four times a session.
Latency, cost, cache hit rate, retries, failover events — searchable, filterable, streaming as it happens. The data layer is open: read it through Supabase if you'd rather build your own.
You keep paying Anthropic and OpenAI directly — Relay never sees the invoice. Your provider keys live encrypted in Cloudflare KV and only decrypt inside the edge worker at request time. They are never written to logs.
Runs on Cloudflare Workers across 300+ cities. The proxy executes in the region geographically closest to your user, so we add ~28ms p95 — less than most Postgres connections.
— How it works
Swap @anthropic-ai/sdk for @relay-api/sdk. Same method signatures, same streaming. Your existing code keeps working — only the import line moves.
$ npm i @relay-api/sdk
Paste your Anthropic, OpenAI or Gemini keys at /signup. We encrypt them in Cloudflare KV at rest and never log plaintext. You can revoke any time.
$ anthropic ✓ openai ✓ gemini ✓
Every call hits a Cloudflare Worker in 300+ cities. Cache lookup, retry policy and failover happen before bytes leave the region nearest the user.
$ POST /v1/messages → ~28ms
Open the dashboard. Filter requests by status, model, latency or cost. The next outage shows as a green failover row, not a red incident.
$ uptime 99.99% failovers handled 137
— Pricing
You only pay for the proxy. Provider charges (Anthropic, OpenAI, Gemini) stay on your own card under BYOK.
Hack on a side project.
Indie devs shipping to real users.
Small teams in production.
When downtime costs real money.
Need more than 1M requests, on-prem, audit logs, SOC 2 questions? Email me directly — I'll write a quote inside 24h.
— FAQ
— Waitlist
I'm Alikhan. I'm hand-onboarding the first hundred builders myself — pairing on the integration, listening to the bugs that ate your evening, shipping fixes the same day.
Drop your email. You'll hear from me when access opens — usually within a week — with six months of Pro on the house and a short ask: what reliability pain do you wish would vanish first?
— Alikhan
— Join the waitlist