deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

Kubernetes 1.37 apply silently drops fields set to null or tilde, live test shows

A hands-on test against a Kubernetes 1.37.0 API server found that ConfigMap keys with null or tilde values vanish on apply and server-side dry-run reports success. The new KYAML output format fixes it.

Kubernetes 1.37 apply silently drops fields set to null or tilde, live test shows

A hands-on experiment documented on dev.to maps exactly what Kubernetes 1.37 does when it meets ambiguous YAML. Applying a ConfigMap whose value is written as ~ or null creates the object without that field — no error, no warning, and no log entry the author could find. Meanwhile, the release's headline fix for YAML ambiguity, the newly stable KYAML output format, works as claimed, but only for manifests that actually use it.

What was tested

Kubernetes 1.37 shipped KYAML (KEP-5295) as stable: a stricter dialect of YAML, heavily quoted and brace-delimited, which kubectl get -o kyaml can emit and kubectl apply accepts as ordinary input because it remains valid YAML underneath. Its stated purpose is eliminating the YAML 1.1 "Norway problem", where an unquoted NO is parsed as the boolean false rather than the string "NO", named after the country code.

The dev.to author wanted to observe a live cluster rather than trust the announcement. Standing up a kind cluster failed first: kubelet in both the 1.37.0 and 1.36.1 node images hard-refuses to start on hosts with a cgroup v1 hierarchy, which is not new in 1.37. Instead, the author ran the 1.37.0 kube-apiserver and etcd 3.5.17 binaries directly as processes, which was enough for a genuine apply and get round trip without kubelet or a container runtime.

Two failure modes, one of them silent

Thirteen hand-written ConfigMap manifests, each containing one ambiguous value, went to the live server. Ten were rejected outright: eight boolean-looking values (NO, no, Yes, OFF, y, n, true, False) and two number-looking ones (1.0 and 010). Because ConfigMap.data is typed as map[string]string in the Go API, the decoder returns a BadRequest — it refuses to coerce a boolean into a string field — and nothing is written. Annoying, but safe.

null and ~ behave differently. They are neither strings nor booleans, so no type mismatch exists to reject. The decoder treats a null map value as empty and drops the key on the way in. The object is created missing that key entirely — not set to an empty string, simply absent.

Server-side dry-run misses it too

According to the write-up, --dry-run=server does catch the boolean and number rejections, returning the same BadRequest with no object created. It does not catch the null case: the dry run reports success and prints the already-damaged object. The manifest's data block held two keys; the output showed one. Without counting fields, nothing looks wrong.

The same library in your own code is worse

The apiserver's strict typed decoding is not the whole ecosystem. client-go ships sigs.k8s.io/yaml to any controller or CLI, and plenty of code calls it directly. A ten-line Go program unmarshalling the same values into a map[string]string field returned no error while changing eleven of fourteen: NO, no, OFF, n and False all came back as the string "false" (with capitalisation normalised in the last case), Yes and y as "true", null and tilde as empty strings, 1.0 as "1", and 010 as "8", because the leading zero was read as octal before being stringified. That is the silent corruption KYAML was built for, and it lives in application code rather than inside the API server.

KYAML holds up

Rendering the same values through kubectl -o kyaml and feeding the result back through the Go unmarshaller corrupted nothing, since every value is quoted. A hand-written KYAML manifest containing a comment and a trailing comma — both illegal in plain JSON — applied cleanly to the live server, which treated the file as ordinary YAML. The author confirms the format works with any kubectl version, matching the announcement. The cost is size: the same ConfigMap rendered as 548 bytes of YAML, 693 bytes of KYAML (roughly 26% more) and 762 bytes of JSON.

Why it matters

The tilde case is dangerous precisely because everything reports success. Server-side validation, the type system, and even a server dry-run in CI all approve an apply that quietly deletes a field, so manifests using null or tilde as a deliberate value get no signal at all. The test also shows two decoders with different strictness sit between a manifest and stored data: the apiserver's rejects type mismatches loudly, while sigs.k8s.io/yaml in your own controllers rewrites values in place. KYAML closes the gap by forcing every value into explicit quotes at a modest size cost, and because it is still YAML it can be adopted incrementally with any kubectl. Until then, a stray ~ remains a field deletion that passes every check.

  • #kubernetes
  • #yaml
  • #kubectl
  • #configmap
  • #devops

Related posts