Most founders I meet still believe a sudden ten-thousand-dollar API bill is the price of “real AI traffic.” It almost never is. It is a loop. A button that retries. A useEffect that fires on every keystroke. A serverless function that calls itself because nobody put a ceiling on it.
Last monsoon a Pune founder forwarded me a screenshot at 6:14 a.m. His weekend demo had been shared in a WhatsApp group. By Monday morning the OpenAI rate-limit page was not the problem. The model had happily accepted every request. The bill was. That is when I stopped treating cost as a finance conversation and started treating it as an engineering one.
Rate limiting is a control that refuses work after a defined budget of requests, tokens, or concurrent jobs. If that sentence is not already in your architecture notes, you do not have an AI product. You have a demo with a credit card attached.
How a weekend prototype becomes a five-figure invoice
Vibe-coded stacks are generous. Tools like Bolt, v0, Lovable, and Cursor will happily wire a chat box straight to a model. That is useful on Friday. It is dangerous on Monday, because the generated front end retries on error, streams on mount, and has no notion of a user budget.
Vercel bills for function duration and invocations. According to Vercel’s published pricing documentation, fluid compute is metered. Pair that with an unthrottled OpenAI completion and you pay twice: once for the model, once for the function that sat open waiting for tokens.
The pattern I see in rescue reviews is boring and repeatable. A React client calls /api/chat on every render. The route has no identity check worth the name. There is no queue. There is no daily cap. Someone pastes the URL into a community Slack. A bot or a curious intern hits refresh. The loop does the rest.
The Open Web Application Security Project lists unrestricted resource consumption as API4 in the OWASP API Security Top 10 2023. That is the polite name for what just emptied the company card.
The three-lock model: refuse, defer, cap
I use a simple mental model on every AI SaaS we harden. Three locks. If any one is missing, the other two will not save you for long.
Think of it as a kitchen. Rate limiting is the waiter who will not take the twentieth order. The queue is the ticket rail so the grill is not cooking twenty steaks because one diner tapped the bell. The budget cap is the owner locking the walk-in freezer at midnight.
| A product that can spend without asking is not ambitious. It is unfinished. |
Why is Upstash Redis a good rate-limit store for serverless?
Upstash Redis is a good rate-limit store for serverless because it is HTTP-reachable, globally replicated, and billed per request, so a Vercel or Cloudflare function does not need a long-lived Redis TCP connection. The official Upstash ratelimiting guide documents sliding-window and token-bucket algorithms you can drop in front of any route.
We key the limiter on three things, in this order: authenticated user id, then a hashed IP, then a route name. User first. If you only key on IP, a shared office or a mobile carrier NAT will punish the wrong people. If you only key on user, an unauthenticated preview URL will still burn money.
A practical starting policy for a seed-stage chat product in 2026: 20 completions per user per hour, 200 per day, and a hard 8 concurrent jobs. Raise it when you have data. Do not start at infinity.
const ratelimit = new Ratelimit({ redis, limiter: Ratelimit.slidingWindow(20, "1 h"), prefix: "chat" });
That one line, on the server, is worth more than a week of prompt tuning. Put it in the route that actually calls the model, not in the client. The client can be rewritten by anyone with DevTools.
What does a BullMQ queue change about cost?
A BullMQ queue changes cost by turning a user click into a single job with retries, backoff, and a concurrency cap, instead of letting the browser fire the model on every render. BullMQ’s own job documentation is explicit: you control attempts, delay, and parallelism.
In the Pune case, the front end retried a failed stream six times. Each retry opened a new completion. After we put the work on a queue with attempts: 2 and backoff: { type: "exponential", delay: 2000 }, the same failure cost one extra job, not six extra invoices.
Workers should run in a long-lived process, not inside the incoming HTTP function. If you must stay serverless, use a dedicated consumer and keep the request handler to “enqueue and return a job id.” The user polls or subscribes. They do not hold a 40-second function open while tokens dribble in.
| Do not enqueue from the browser. If the queue endpoint is public, an attacker becomes your most productive employee. Authenticate, then enqueue on the server, then return only the job id. |
A worked repair: the WhatsApp-shared demo
Here is what we actually did on that Pune product. The stack was Next.js on Vercel, OpenAI chat completions, no auth on the preview URL, and a client hook that called the API in useEffect with an empty dependency array that was not, in fact, empty after the generator rewrote it.
Day one: we cut the preview URL. We added a session cookie. We put Upstash in front of /api/chat at 10 requests per hour per user. Ugly. Effective. The bleeding stopped before lunch.
Day two: we moved generation to BullMQ. The route accepted a prompt, wrote a job, returned { jobId }. A worker with concurrency 4 called the model. The UI subscribed to job status. Refreshing the page no longer minted a new completion.
Day three: we set a monthly organisation cap in the OpenAI dashboard and a matching check in our worker that reads a Redis counter of estimated tokens. When the counter crosses 80 percent of the cap, new jobs enter a “paused” state. Humans get a Slack ping. The model does not get a vote.
According to the OWASP API Security Project, resource consumption is a first-class risk, not a nice-to-have. The 2023 edition put it in the top ten for a reason. We did not need a new framework. We needed those three locks.
The founder asked if this would make the product feel slow. It made the product feel finished. Waiting two seconds for a queued job is better than waiting for a finance team to reverse a card.
What you can ship today
Do not start with a rewrite. Start with a ceiling. This afternoon, in this order:
- Turn on organisation usage limits at your model provider. Write down the number.
- Add Upstash rate limiting to the one route that calls the model. Key on user id.
- Kill any
useEffectthat talks to that route. Call it from an explicit user action. - Put a daily Redis counter next to the limiter. Alert at 50 percent and 80 percent.
- Move the completion onto BullMQ or an equivalent queue before you add a second model.
Data from the Vercel Functions runtime docs is clear that duration is a billable unit. Streaming a 2,000-token answer inside the request path is how you pay for waiting. Queue it.
If you want a partner to do this as a rescue, IndiaNIC’s production teams treat cost controls as part of hardening, not as a later “optimisation” sprint. We would rather ship an ugly limiter on day one than a beautiful chat that bankrupts a seed round.
The next useful conversation with your team is not “which model is cheaper.” It is “who is allowed to spend, how often, and what happens when they cannot.” Start that conversation before the next WhatsApp share.
Frequently asked questions
Will rate limiting make my AI product feel broken to real users?
Rate limiting makes an AI product feel broken only when the limit is a surprise. Show remaining quota in the UI, queue overflow work, and return a clear “try again in 12 minutes” instead of a generic 500. Users accept a ceiling. They do not accept a silent failure that still bills you.
Is a provider dashboard limit enough on its own?
A provider dashboard limit is not enough on its own because it stops the whole organisation, not the one user or bot that is looping. Use the dashboard as the last lock. Put per-user and per-route limits in your own Redis so one abusive session cannot freeze paying customers.
Can I stay fully serverless and still run BullMQ?
You can stay mostly serverless, but BullMQ workers need a process that stays alive to pull jobs. Many teams keep Vercel for the website and run a small worker on Railway, Fly.io, or a container. The request path stays cheap. The worker is where spend happens, on purpose.
