Curl-loop evidence shows symlink-swap deploys still serve 502s on every release

The deploy that wasn't atomic

A writeup on dev.to examines a deployment setup many teams treat as the safe default on simple infrastructure: a Capistrano-style release tree on a single virtual machine, where each build lands in a timestamped directory and a current symlink is repointed at the new release before the service restarts. The author assumed this made deploys atomic — until running curl in a loop during a real deploy and watching every request fail with a 502 for roughly three seconds, the time the application needed to finish booting and bind its port again.

The core distinction the author draws: the symlink swap switches files atomically, but systemctl restart is a stop followed by a start, and between those two steps nothing is listening. A request that began against the old release finishes fine; requests during the gap have nowhere to go.

Even the symlink swap has traps

Before getting to the restart, the post flags that two of the three obvious ways to repoint the link are broken. ln -sf follows an existing symlink that resolves to a directory and quietly creates the new link inside the target. ln -sfn avoids that but is implemented as an unlink followed by a symlink call, leaving a brief window where the path resolves to nothing. The correct approach, according to the author, is to create a temporary link beside the target and move it into place with mv -Tf, which relies on rename(2) and is genuinely atomic.

Two mechanisms, one 502

The post identifies two shapes the outage takes. The first is the straightforward reverse proxy: nginx passes traffic to 127.0.0.1:3000, the process dies, the port closes, the connection is refused, and nginx returns 502.

The second catches people out. In a conventional PHP setup, nginx owns the document root and only .php requests reach php-fpm over a unix socket. The instinct is that nginx serving files itself makes a restart harmless — but if the fpm master runs under the same systemd unit as the app, a common choice so that logs, resource limits and lifecycle stay per-application, restarting the unit unlinks the socket file and nginx gets ENOENT until it returns. Static assets keep serving throughout, which makes the gap look like an application bug rather than a deployment one.

The nginx line that changes nothing

The reflex fix is proxy_next_upstream. The author explains it is inert next to a literal proxy_pass address, because that forms an upstream group with exactly one peer — there is no next server to try, so the same 502 comes back with a config line that suggests the problem was handled.

The version that works looks like a typo: an upstream block listing the same address twice, so a refused connection can be retried against the second entry. max_fails=0 matters too. With the default max_fails=1 and fail_timeout=10s, one refused connect marks both entries down for ten seconds, converting a 200 ms gap into a ten-second outage. The post also recommends answering with 503 plus a Retry-After header rather than 502, since a process that is starting is not a broken upstream, and the status code is the part clients, CDNs and health checkers can act on.

The caveat is explicit: this buys a retry, not zero downtime. It absorbs a sub-second gap, not a multi-second boot.

The actual fix: run both versions

The post's real answer is blue-green deployment driven by a systemd template unit, [email protected], plus a slots directory where slots/blue and slots/green each symlink to a release and carry their own port assignment. A deploy builds the new release without touching what is serving, points the standby slot at it, restarts only the standby service, and verifies it with curl on its own port rather than through the proxy, which still routes to the old version. The published text cuts off mid-sequence, so the final cutover steps are not shown in full.

Why it matters

The symlink swap is a long-standing, widely repeated recipe, often described as delivering zero-downtime deploys without containers or load balancers. This post measures the claim and finds it false in the common case: atomic file switching says nothing about process availability, and the resulting outage recovers on its own fast enough that engineers tend to file it as flaky rather than deploy-related. It also documents a trap in which the standard nginx mitigation does nothing — or, misconfigured, stretches the outage from milliseconds to ten seconds. For teams on plain VMs, the takeaway is that availability during deploys comes from running two instances and cutting over between them, not from how cleverly the filesystem pointer moves.