deniz.in

Markets

Weather

Loading weather

· via Hacker News – Front Page (native)

S2 replaced Route53 with its own authoritative DNS servers to support unlimited subdomains

S2 says Route53's API rate limits made unlimited per-customer subdomains impossible, so it built an authoritative DNS server that queries Spanner directly, fronted by Cloudflare's DNS Firewall.

S2 replaced Route53 with its own authoritative DNS servers to support unlimited subdomains

Why S2 moved off Route53

S2, a service whose storage model is built around streams and basins, has replaced AWS Route53 as the authoritative DNS provider for its customer subdomains — and it has published a detailed account of why running its own DNS servers became the only workable option. The post, which surfaced on the front page of Hacker News, is a rare blow-by-blow description of operating authoritative DNS at scale.

The trigger was a product decision. Basins, S2's top-level containers for streams and rough analogues of S3 buckets, each get their own subdomain of the form my-basin.b.s2.dev, backed by a CNAME record that routes clients to the correct cell in S2's architecture. S2 wanted to lift its limit of 100 basins per account after watching customers work around it. Unlimited basins implied unlimited DNS records — and those records needed strong consistency, so a freshly created basin would resolve immediately instead of tripping over a cached NXDOMAIN.

The Route53 bottleneck

According to the post, Route53 itself can hold very large numbers of records, but its API throttled at around five UPSERT operations per second — a figure S2 notes may have changed since, as AWS has recently updated its throttling documentation. The team learned this in practice when a bad actor rapidly created tens of thousands of basins, saturating S2's Route53 record quota. The resulting backlog took hours to process, with jobs deleting records while others tried to add them. Enforcing the 100-basin limit was the direct outcome of that incident.

Batching writes was considered and rejected: it would break instant resolvability, and legitimate usage could still hit the ceiling. The team also evaluated Google Cloud DNS, Cloudflare and Azure DNS, and found that all of them impose heavy rate limiting on API-driven record changes. That left one option — building a custom authoritative DNS server.

DNS as a view over the database

Rather than syncing records into a separate system, S2 made its DNS server answer queries with a live SELECT against its existing basin database, which runs on Google Cloud Spanner. DNS effectively became another client of the company's strongly consistent account database, eliminating a sync path and an extra moving part.

The team weighed writing a CoreDNS plugin against building on a library and chose the library route for flexibility. Although S2 mostly writes Rust, it picked Go for this project because Spanner tooling and the broader GCP ecosystem are more mature there, settling on the miekg/dns library. The server handles A, AAAA, NS, CNAME and SOA records. Correctness testing combined unit tests, RFC compliance checks using Ferret — a tool the team found through the SCALE paper on finding RFC compliance bugs in nameservers — and a post-deploy smoke test against live traffic.

Wildcard certificates for the basin zone require ACME DNS-01 challenges, and S2 chose not to serve those from its own server: the challenges are delegated via CNAME to a separate Route53 zone, which avoids dynamic TXT records and any changes to the certificate renewal pipeline.

Hardening a service exposed to the internet

The part that frightened the team most, per the post, is the NXDOMAIN flood: every query for a random non-existent name becomes a fresh database lookup, and the author regards it as essentially unstoppable as an attack. After looking for something equivalent to a reverse proxy in front of DNS serving, S2 found only two vendors that fit — Cloudflare and Akamai, both with no public pricing — and selected Cloudflare's DNS Firewall after running proofs of concept. The origins accept traffic only from Cloudflare's known IPs, and an automated ns-leak-test guards against regressions that might expose the load balancer addresses in SOA and NS responses.

For high availability, the DNS server runs on two GKE clusters in separate regions, both located close to Spanner, with multiple replicas behind TCP and UDP load balancers. The team also built the alerts, runbooks and late-night recovery procedures that a managed provider would otherwise absorb. The final delegation chain is genuinely tangled: a regular Cloudflare zone for the main domain, delegation of the basin subdomain to Cloudflare's DNS Firewall service, GKE origins behind it, plus a Route53 zone for certificates. Tuning DNS caching across clients, firewall and application is still on the internal TODO list.

Why it matters

Managed DNS looks unlimited until a product depends on per-customer subdomains, and S2's post documents that every major provider throttles API-driven record creation in ways that quietly cap that model. The write-up also offers a reusable design pattern — making authoritative DNS a thin, strongly consistent view over an existing database rather than a synced copy — and an honest accounting of what you take on when you leave a managed service: attack surface like NXDOMAIN floods, IP-leak regressions, and operational ownership around the clock. For any platform handing out subdomains per tenant, it is a practical blueprint, caveats included.

  • #dns
  • #cloud
  • #infrastructure
  • #aws
  • #google-cloud

Related posts