A Tableau heartbeat kept a Databricks serverless warehouse alive and ran up a $14k weekend bill

What happened

A first-person postmortem on dev.to details how a team's Databricks serverless SQL warehouse generated roughly $14,000 in charges over a single weekend. The author writes that a PagerDuty alert arrived at 2:14 AM on a Sunday saying the team had consumed 80% of its monthly cloud spend within 48 hours. The pipeline everyone initially suspected had shipped that Friday, passed CI, and landed its data on time. The failure mode was cost, not correctness.

Why the usual suspects didn't fit

According to the post, the author's first assumption was a runaway loop in a Python job. Cluster logs on the DBR 13.3 LTS environment showed nothing unusual, and neither did the Spark query history beyond standard daily ingestion. The billing dashboard made the scale obvious: three months of flat spend followed by a near-vertical spike on the warehouse named sql_warehouse_prod_v2.

The auto-stop setting initially looked like proof of innocence. The console showed a 10-minute auto-stop on the Serverless SQL Warehouse, which should mean the warehouse shuts down when idle. The author realised the mistake was watching cluster state instead of session state. The warehouse was never actually idle.

A heartbeat that reset the idle timer

The root cause was a hidden interaction between the team's Tableau integration and the serverless warehouse. A dashboard connection string, configured with catalog and schema settings, was issuing a system.information_schema query on a heartbeat interval of 8 minutes. Because the BI tool connected through a service principal with broad permissions, the session stayed alive.

Every heartbeat arrived before the 10-minute idle window expired, resetting the timer indefinitely. Databricks counted these pings as active queries, so the warehouse never reached its idle threshold and never stopped. The queries were sub-millisecond, which is why they never surfaced in the team's performance monitoring. The sizing made it worse: a Large warehouse, billed at a significantly higher DBU rate, was serving metadata lookups that a Starter warehouse could have carried. On a traditional cluster, resource contention or timeouts might have ended the situation naturally; serverless, designed to stay available, simply kept paying for readiness.

The fix

The immediate remediation, per the post, was to terminate the Tableau connection and set auto-stop to 1 minute so the warehouse would shut down as soon as the connection was severed. The team then split its workloads: a Serverless-Small warehouse now handles the heartbeat-heavy dashboarding traffic, while the Large warehouse is reserved for ad-hoc analyst queries and heavy ELT. The BI connection string was repointed at a specific Unity Catalog schema so it no longer triggers broad information_schema scanning. Custom tags for cost centre and owner were also added to warehouses, letting the team isolate DBU consumption in billing exports within minutes rather than waiting for aggregation.

Guardrails added afterwards

The team then introduced structural changes so a similar event cannot repeat silently:

• Warehouse sizing policy: no production warehouse above Medium without a documented exemption in the Terraform repository. Larger sizes require a pull request, and CI runs a cost estimate against the Databricks Billing API to flag the daily run-rate.
• Scoped service principals: BI tools now use read-only principals limited to specific schemas, preventing the catalog and information_schema queries that caused the hidden load.
• A budget alarm: a Lambda function polls Databricks billing data every 6 hours and fires a high-priority Slack alert if the daily burn rate deviates more than 20% from the 7-day rolling average.
• Auto-stop discipline: never longer than 5 minutes for non-critical workloads.

Why it matters

The story is a useful counterweight to the idea that serverless removes operational burden. Serverless takes away node management, but as the author frames it, teams still own session lifecycle. Idle-based shutdowns are only as trustworthy as the definition of idle, and lightweight clients, such as BI heartbeats, connection keepalives or monitoring probes, can quietly void them while remaining invisible in performance dashboards. The lesson generalises beyond Databricks: elasticity converts operational mistakes directly into invoices, so spend has to be monitored as telemetry, with variance-based alerts rather than absolute thresholds alone. For teams adopting serverless data platforms, the checklist from this incident is short: size warehouses to real load, scope service principal permissions narrowly, tag resources for fast attribution, and alert on the rate of change in burn, not just totals.