deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

Terraform null default silently disabled RDS deletion protection on 14 databases

A wrapper module passing null for an RDS setting meant Terraform never applied the platform module's default, so deletion protection defaulted to false — and a destroy removed the QA database.

Terraform null default silently disabled RDS deletion protection on 14 databases

What happened

In August, an engineer who believed he was working in a personal sandbox ran terraform destroy against what was actually the shared QA environment, and the QA database was among the resources removed. According to a postmortem by Sergey Shinder on dev.to, nobody was alarmed at first: the organisation's database module sets deletion_protection, and AWS is supposed to refuse deleting an RDS instance with that flag enabled. The instance disappeared anyway.

How a null overrode the defaults

The platform team owns the database module, whose variable for the flag defaults to true. Most teams do not call that module directly. They use a wrapper that adds naming and tagging conventions and re-exposes the same setting with a default of null, accompanied by a comment saying that when a team says nothing, the platform module's own default should apply.

As Shinder explains, that is not how Terraform interprets the value. Variables are nullable unless declared otherwise, and passing null into a nullable variable is an explicit set rather than an omission, so the module's own default is never consulted. The null then reached the aws_db_instance resource, where supplying null is equivalent to omitting the argument entirely. That handed the decision to the AWS provider, whose default for deletion protection is false.

The blast radius

Every database provisioned through the wrapper had therefore been created unprotected. Each plan had shown deletion_protection = false, one line among roughly sixty in a module the teams trusted. When the organisation eventually enabled the AWS Config managed rule rds-instance-deletion-protection-enabled, it flagged fourteen of twenty-two instances, all of them built through the wrapper since its first release. QA was rebuilt from a nightly snapshot copy, and an afternoon's worth of test data went with it.

The remediation

Shinder lists several changes made afterwards:

  • Every variable in the platform module now declares nullable = false, so an incoming null is replaced by the module's default, which produces the outcome the wrapper's author originally wanted.
  • The wrapper sets no defaults of its own and forwards values only when a team explicitly supplies them.
  • The AWS Config rule stays enabled in every account to catch regressions.
  • The database resource includes a precondition that fails the plan unless the resolved value is true.

Why it matters

The failure here is structural rather than a one-off mistake. In Terraform, null means unset, and an unset value falls through to the next consumer in the chain — in this case a provider that knows nothing about an internal standard. Wrapper modules using null defaults to signal inheritance of an inner default are a common pattern, and the resulting failure is quiet: plans succeed, resources get created, and nothing surfaces until a destroy works when it should not.

The fixes generalise well beyond this team. Declaring nullable = false lets module defaults survive passthrough. Guardrail settings such as deletion protection can be enforced with preconditions, so a wrong value stops the plan instead of shipping silently. External checks like AWS Config rules catch misconfiguration even when module authors get variable semantics wrong. And the incident is a reminder that workspace selection is as much a part of a destroy command's blast radius as the code itself.

  • #terraform
  • #aws
  • #rds
  • #infrastructure-as-code
  • #postmortem

Related posts