From Server 1 to N

Nine things that break the moment you add the second one. Your error tracker won’t show you any of them.

Mihir Jataniya · October 14, 2026 · 13 min read

You are at a startup. Small team, no senior engineer above you, and you own more of the system than your job title suggests. You wrote the auth, you wrote the upload endpoint, you set up the cron job. It all works.

Then the product gets picked up somewhere, and traffic does the thing traffic does.

Yesterday your single box sat at 30–40% CPU and you never thought about it. Today p99 latency has gone from 80 ms to four seconds, the event loop is saturated, and requests are queueing somewhere you cannot see. Nobody is going to rewrite anything this week. So the team makes the obvious call, the correct call, the one any engineer would make:

add a second server and put a load balancer in front of it.

You spin up an identical instance, point an nginx upstream at both with round-robin, and deploy. Capacity doubles. CPU drops. Latency comes back down.

And then the weird stuff starts.

Users report being randomly logged out. Someone’s avatar upload works, then 404s, then works again. A customer emails to ask why they received the same digest email five times. Your rate limiter, configured for 100 requests a minute, is letting someone through at 300. None of it appears in Sentry. None of it appears in your logs. Every health check is green.

Nothing is broken. Everything is working exactly as written. The code just made an assumption that stopped being true the moment there were two servers.

Here is the one question that finds all of it:

Does any variable in this process outlive a single request?

A variable created inside a handler and thrown away when the response is sent is fine. That is just work. A variable at module scope, or attached to the app, or captured in a closure, that still holds something after the response has gone out is state living on one specific machine, and it is about to become a bug.

Nine places it hides.

1. Sessions in memory

The default session store in most frameworks is memory. express-session without a store option will even warn you about it in the console, and everybody ignores the warning because it works.

app.use(session({ secret: process.env.SECRET }))   // MemoryStore

Server A creates the session. The next request lands on Server B, which has never heard of that session ID, so the user is anonymous again.

How often depends on how your balancer distributes work. With round-robin it is roughly every other connection. Browsers reuse a connection for many requests via keep-alive, so it does not alternate cleanly request by request. It shows up as the session working fine for a while and then vanishing when a new connection is opened. From the user’s side that is worse than a clean 50/50, because it looks random and they cannot describe it to you.

Fix: move sessions to a store every server can reach. Redis, with the session ID still in an httpOnly; Secure; SameSite=Lax cookie and the session data in Redis under that key. Three lines of config.

And do not reach for sticky sessions instead. Pinning each user to one server makes the symptom go away and keeps the actual problem. Now that server holds state nobody else has, so you cannot restart it without logging those users out. Which means you cannot deploy without logging those users out. You have traded a bug for a permanent constraint on how you ship code.

2. In-memory caches

const cache = new Map()

app.get('/products/:id', async (req, res) => {
  if (cache.has(req.params.id)) return res.json(cache.get(req.params.id))
  const p = await db.products.findById(req.params.id)
  cache.set(req.params.id, p)
  res.json(p)
})

Two servers, two caches, each filled by whichever requests happened to land there. They will disagree. Marketing changes a price and for a while Server A serves the new one and Server B serves the old one.

This one is usually fine, and that is the point of including it. Sixty seconds of a stale price is annoying, rarely fatal. Coordinating caches across servers costs far more than the inconsistency does, so the right move is to accept the divergence and put a bound on it:

const cache = new Map()   // value: { data, expiresAt }
const TTL = 60_000        // 60s of worst-case disagreement, by design

A TTL converts “inconsistent, unknown duration” into “inconsistent, at most 60 seconds.” That is a design decision you can defend in a review.

It stops being fine when the cache is authoritative: the only copy, or one where being wrong has consequences. A cached permission check returning yes after access was revoked is not an inconsistency. It is a security hole. If the cache is the only source of truth, it is not a cache. It is your database, and it is in RAM on one machine.

3. Rate limiters and counters

My favourite, because the arithmetic is a single line and most people have this bug right now.

const hits = new Map()   // userId -> count

function limit(req, res, next) {
  const n = (hits.get(req.user.id) || 0) + 1
  hits.set(req.user.id, n)
  if (n > 100) return res.status(429).send('Too many requests')
  next()
}

One server: 100 requests a minute. Correct.

Three servers behind round-robin: each sees a third of the traffic and each independently allows 100 before it starts rejecting.

3 servers × 100 each = 300 requests per minute

Your limit is now 300. You did not loosen it. You tripled it, silently, by deploying. Generalised: an in-memory limit of L on N servers is really N × L. It gets worse under autoscaling, because N changes through the day. The effective limit at 3 a.m. is not the one at peak.

And nothing errors. Nothing logs. The feature simply stops doing its job, and you find out when somebody abuses it.

Fix: one shared counter with an atomic increment.

const n = await redis.incr(key)
if (n === 1) await redis.expire(key, 60)
if (n > 100) return res.status(429).send('Too many requests')

INCR is atomic because Redis executes commands one at a time, so two servers incrementing the same key cannot interleave and lose a count. One counter, all servers, correct limit.

4. Files on local disk

app.post('/avatar', upload.single('file'), (req, res) => {
  // multer just wrote this to ./uploads/ on THIS machine
  res.json({ url: `/uploads/${req.file.filename}` })
})

The upload succeeds. The user reloads, the request lands on the other server, and the image 404s about half the time, at random, which is the hardest kind of bug to get a reproduction for.

Then it gets worse. On the next deploy the container is replaced, and container filesystems do not survive that. The file is not misplaced. It is gone.

Fix: object storage, S3 or equivalent. Your servers upload to it and store the returned key in your database. Nothing durable ever lives on a machine you are willing to terminate.

5. WebSocket and SSE connections

A WebSocket is a long-lived TCP connection to one specific process.

user A ── ws ──► Server 1      Server 2 ◄── ws ── user B
                    │               ▲
                    └───── ? ───────┘
                      no path exists

User A is connected to Server 1, user B to Server 2. A sends B a message. Server 1 does not hold B’s socket and has no way to reach it.

This is the one item you cannot fix by moving data, because the connection is the state and a TCP connection cannot be shared between processes.

Fix: keep the connection local and move the messaging. Server 1 publishes the message to a Redis pub/sub channel, every server subscribes, Server 2 receives it and writes to B’s socket. Socket.IO’s Redis adapter is exactly this and is about five lines to wire up.

6. Scheduled jobs

cron.schedule('0 0 * * *', sendDailyDigest)   // midnight

One server: sent once. Five servers: every server runs it, and every customer gets five identical emails at midnight.

This is the most dangerous item on the list, for a reason that has nothing to do with the code. It passes every test you have, because you test on one machine and on one machine it is perfect. It fails the first night after you scale, at midnight, to your entire customer base at once, while you are asleep. And the damage is reputational rather than technical, so no alert fires.

Fix, in order of preference:

  1. Take the scheduler out of the application. A single-instance worker, or your platform’s scheduler hitting one endpoint.
  2. If it must live in the app, make the instances race for a lock and let the losers skip:
const won = await redis.set('job:digest', instanceId, 'NX', 'EX', 300)
if (!won) return                  // another instance is doing it
await sendDailyDigest()

NX means “only set if absent”, so exactly one instance wins. EX matters too. Without an expiry, one crash leaves the lock held forever and the job never runs again.

Be honest about what this is. It is good enough to stop five instances sending five emails, which is the problem we have. It is not a correct mutual-exclusion lock. If the winning instance pauses for longer than the TTL, say a long GC pause or a hung network call, the key expires, another instance acquires it, and both run. For a digest email that is an acceptable risk you have chosen. For anything where double execution is genuinely unacceptable, you need either a fencing token or, better, to make the operation idempotent so running it twice does not matter.

7. In-process locks

let processing = false

async function handle() {
  if (processing) return
  processing = true
  try { await doTheThing() } finally { processing = false }
}

This genuinely prevents concurrent execution inside one process. Five servers means five independent flags and up to five concurrent executions.

The lock does not fail loudly. It just stops being a lock, and whatever race you wrote it to prevent comes back, now in production, now intermittent.

Fix: a distributed lock in Redis. But better, where you can: make the operation safe to run concurrently so it needs no lock at all. A conditional update does this for free, because the database decides the winner atomically:

UPDATE orders SET status = 'processing'
WHERE id = $1 AND status = 'pending'

If two servers run that simultaneously, one of them updates a row and the other updates zero rows. Check the affected-row count and the loser simply stops. No lock, no coordination, no lock expiry to get wrong.

8. setInterval and in-memory queues

const pending = []
setInterval(() => flush(pending), 5000)   // batch writes every 5s

First, why anyone writes this, because the pattern is not stupid. Every database round trip costs half a millisecond to a millisecond. A thousand INSERTs is a thousand round trips; batched, it is one. Buffering and flushing every five seconds is a real optimisation and it works.

Then the process dies with items still in the array. They are gone. No error, no log, no trace. The user’s action was accepted with a 200 and silently discarded.

Put a number on it:

200 events/s × 5 s buffer      =  1,000 items at risk per instance
3 instances, rolling restart   =  3,000 items lost per deploy
deploying 10 times a week      =  30,000 items a week, silently

On one server you restarted weekly and never noticed. Now rolling restarts feel safe so you ship far more often, and the buffer is per process, so N instances means N lossy buffers and nothing that knows what was dropped.

The obvious fix is to flush on shutdown. It has two sharp edges.

You have to wait for it. This is what people write:

process.on('SIGTERM', () => {
  flush(pending)        // returns a promise that nobody awaits
  process.exit(0)       // process is gone before the write lands
})

process.exit() does not wait for pending work. You wrote a flush handler, watched it run, and still lost the data. The working version:

process.on('SIGTERM', async () => {
  try { await flush(pending) } finally { process.exit(0) }
})

And you only get SIGTERM sometimes. Not from the OOM killer, docker kill, Kubernetes escalating to SIGKILL after the grace period, a reclaimed spot instance, or a crash. SIGKILL cannot be caught. The handler reduces losses; it does not remove them.

Learn the shape, because it wears several costumes: an in-memory job array with a worker loop, a debounced save, a setTimeout retrying a webhook in thirty seconds, an outbox of unshipped events. If the only record of the work lives in a variable, the work dies with the process.

Fixes, in the order I would try them:

  1. Write through immediately and batch at the database. Multi-row INSERT, COPY, or group commit. Costs a little throughput and the problem disappears instead of shrinking.
  2. Use a durable queue. SQS, Redis Streams, BullMQ. The item is persisted on arrival, so a dead process loses nothing. Redis is only as durable as its persistence config, so check it.
  3. If the buffer stays, bound the loss deliberately. Cap its size, flush on count as well as timer, await on SIGTERM, and write down what you accept losing.

Be honest about which case you are in. Losing five seconds of analytics pageviews is fine and worth saying out loud instead of engineering around. Losing five seconds of “your order has been placed” is not, and no buffer size makes it so.

9. Self-incremented metrics

let requestCount = 0
app.use((req, res, next) => { requestCount++; next() })

Five servers each report their own count, so unless something sums across instances your dashboard shows a fifth of the truth.

Usually a monitoring bug rather than a user-facing one, which sounds harmless until you notice what it means: your numbers are wrong at exactly the moment you need them to be right. You scale out because of a traffic spike, and the act of scaling out breaks the graph you are using to decide whether scaling out worked.

Fix: export per-instance counters and aggregate in your metrics backend. Prometheus and friends are built for this. Never add up application-level counters by hand.

The pattern

Read the nine together and they are all the same bug.

Your code assumed one process, one memory space, forever. That assumption is free on one server and silently false on two. Nothing throws, because from the code’s point of view nothing went wrong. The variable really was set, the file really was written, the flag really was flipped. It just happened on a machine that this request did not land on.

Which is why your error tracker is quiet. There is no error. There is a wrong answer.

So before you add that second server, grep for the question:

Does any variable in this process outlive a single request?

Every hit is either something to move, or something to consciously decide to leave alone. Both are fine. What is not fine is not having looked.

The bare minimum before you scale out

The nine above are the state audit. These are the four things to have alongside them, because having two servers is only half the job. The other half is being able to lose one.

Health checks, and two of them. /live answers “is this process alive” and touches nothing. /ready answers “can this instance serve traffic right now” and may check things that are true of this instance: its connection pool, its shutdown flag. It must not check shared dependencies like your database or Redis. If /ready pings Redis, a two-second Redis blip fails every server’s health check inside one interval, your load balancer has zero healthy backends, and a degraded feature becomes a total outage.

Graceful shutdown, in the right order. On SIGTERM: fail /ready, then keep serving normally for however long your load balancer takes to notice (interval × unhealthy threshold, often 10–30 seconds), then stop accepting connections, then drain in-flight requests, then exit. Skip the waiting step and the balancer routes live traffic at a closed socket for the length of its detection window.

Sticky sessions off, deliberately. Once sessions are in Redis you do not need them, and leaving them on quietly reintroduces item 1.

Trust your proxy by hop count, not by true. Behind a load balancer every request appears to come from the balancer’s IP, so your rate limiter sees one client. The fix is X-Forwarded-For, but that header is client-settable, and app.set('trust proxy', true) means “trust every hop”, which makes Express read the leftmost entry, the one the attacker controls. Set it to the number of proxies you actually own:

app.set('trust proxy', 1)   // one proxy in front of us

Now it counts one hop in from the right and takes the address your balancer actually observed. Trust exactly as many hops as you control, counted from the right, and never one more.

One last thing

Going from one server to two is not a deployment change. It is the point where you find out what you actually built, because every assumption about shared memory gets tested at once by real users.

The good news is that it is a finite list. You are not rearchitecting anything. You are moving a handful of variables to places more than one process can see, and writing down the one case where you have decided not to.

And once you have done it, the next ninety servers are boring. That is the whole return on the work.