Lesson 23 / 32
Deploys, Restarts and Graceful Shutdown
Restart real-time servers without dropping users abruptly or causing a reconnect storm.
By Rahul Agarwal · 15 years in web development
Every deploy disconnects someone
A REST deploy can drain in seconds because requests are short. A real-time server holds connections for hours, so every restart will disconnect users; the goal is to make that cheap. A graceful shutdown on SIGTERM has three steps. Stop accepting new connections, so the load balancer sends new clients elsewhere (fail the readiness probe first in Kubernetes). Send each client a close frame with 1001 Going Away, so clients know to reconnect, ideally after a short random delay. Then, after a deadline, terminate whatever has not closed and exit. Rolling deploys that restart one instance at a time spread the reconnects. Combine this with client-side jittered backoff and resume-from-sequence, and users see at most a brief "reconnecting" banner. Make sure the orchestrator's grace period (for example terminationGracePeriodSeconds) is longer than your shutdown deadline, or the process is killed mid-drain.
Graceful shutdown with ws
Close codes tell clients this is a planned restart. The timer is unref'd so it does not keep the process alive on its own if every socket closes early.
import { WebSocketServer } from 'ws';
const wss = new WebSocketServer({ port: 3000 });
process.on('SIGTERM', () => {
wss.close(); // 1. stop accepting new connections
for (const ws of wss.clients) {
ws.close(1001, 'server restarting'); // 2. 1001 = Going Away
}
// 3. Force-close anything that has not finished the close handshake.
setTimeout(() => {
for (const ws of wss.clients) ws.terminate();
process.exit(0);
}, 10_000).unref();
});Rehearse the reconnect storm
In staging, open thousands of test connections and restart the server. Watch CPU, auth service load and database queries during the reconnect wave. Resume logic that runs a heavy query per client is the usual culprit.
Quick check: Which close code should a server send to clients when it is shutting down for a deploy?
- 1000 Normal Closure, so clients stop for good
- 1006 Abnormal Closure
- 1001 Going Away
- 1009 Message Too Big
Answer
1001 Going Away — 1001 means the endpoint is going away, such as a server restart. 1006 can never be sent in a frame, and many clients treat 1000 as "do not reconnect".