Traefik 3.7 service middlewares: announcement example fails, correct syntax verified

The example in Traefik's own announcement does not parse

Traefik 3.7, codename "Langres", shipped with service middlewares as its headline feature: the ability to attach a middleware to a service — including a single child of a weighted service — so it runs after the load balancer has already picked a backend. But according to a hands-on write-up on dev.to, the example YAML in Traefik's announcement post does not work. A container running 3.7.14 rejects the config at startup with the error field not found, node: middlewares.

The problem is placement. The announcement nests middlewares inside loadBalancer, next to servers. The author tested the config against a real Traefik 3.7.14 container via the file provider and also inspected the Go source for that version: the Service struct carries a Middlewares []string field, while ServersLoadBalancer has no such field at all. The key belongs one level up, as a sibling of loadBalancer on the service itself. Moving it there made the same container start cleanly.

The feature itself works

With the syntax fixed, the dev.to post set out to verify the core claim: that a service middleware can target one branch of a weighted service. The test used a 9:1 weighted split between a stable backend and a canary, with a middleware setting X-Canary: true attached only to the canary service.

The results were exact. Across 200 sequential requests, 180 landed on the stable backend and 20 on the canary — a perfect 90/10 match — and the header appeared on all 20 canary responses and none of the 180 stable ones. A second run of 500 requests fired 50 at a time produced the same split and the same isolation, ruling out a race between branch selection and middleware execution under concurrency.

One operational nuance emerged from the raw request sequence: the weighting is not random sampling but deterministic, with roughly one canary request per ten in a fixed pattern. Anyone testing a canary by hand with a handful of requests may simply never hit it, or misread a fixed slot in a cycle as a probability.

Ordering and the inline limitation

The post also checked the documented ordering claim — that router middlewares run before service middlewares — by layering a router-level and a service-level middleware that set the same X-Trace header. The service middleware overwrote the router's value on every one of 500 canary requests, while the stable branch only ever saw the router's value. The documented order held.

The announcement's limitation that weighted or mirrored children cannot carry inline middlewares also holds, but with a sting: attempting it anyway produces the same field not found error, because the WRRService struct has no Middlewares field either. And since the error fails the entire file provider configuration, one stray middlewares: line on a weighted child stops every router that provider manages from updating — not just the one being edited.

Older versions and an API blind spot

Running the same dynamic config unmodified against Traefik 3.6.25, codename "ramequin", produces the identical error, per the post: no partial support, and the whole file provider refuses to load. Teams sharing configs across versions should note that 3.6 will not quietly ignore the new field and carry on.

The dev.to author also found that Traefik's API disagrees with reality: while the canary header middleware was demonstrably live and modifying real traffic, the /api/http/middlewares endpoint reported usedBy: None for it. Anyone using the API to judge whether a middleware is safe to delete gets a misleading answer for service-attached middlewares.

Why it matters

Service middlewares are the reason many ops teams will upgrade to 3.7, yet the announcement's own example fails in a way that produces a hard startup error rather than a warning. The blast radius is also larger than it looks: a single misplaced key can take down the entire file provider, including dozens of unrelated routers across many files. Combined with a usedBy field that does not reflect service-level attachments, upgrading teams should validate configs before rollout, keep the middlewares key at service level, and avoid trusting the API for impact analysis of these new attachments. The upside, confirmed by testing, is that the underlying feature — per-branch middleware isolation in weighted services — behaves exactly as promised once the syntax is right.