New package brings zero-code OpenTelemetry tracing to Dagster pipelines

A new open-source Python package, opentelemetry-instrumentation-dagster, adds distributed tracing to Dagster pipelines with no changes to pipeline code. According to its author, Hirofumi Tsuda, writing on dev.to, installing the package and launching Dagster through the standard opentelemetry-instrument command is enough: every op, asset, multi-asset, asset check and dbt asset in a definitions file emits a trace span automatically, with no decorators or imports needed in user code.

Two packages, two trade-offs

The new library is a companion to dagster-otel, Tsuda's earlier project that provides an explicit @traced() decorator stacked under @op or @asset. That approach is opt-in and touches no framework internals, but requires editing every function that should be traced. The auto-instrumentation package makes the opposite trade: zero code changes in exchange for runtime patching. Tsuda frames the split as deliberate, mirroring how the OpenTelemetry Python ecosystem keeps framework instrumentation packages (such as those for Flask or Django) separate from the manual API and SDK rather than offering them as toggles.

How the patching works

Rather than reaching into already-built definition objects and swapping compute functions after the fact, the package patches the decorator factories themselves — dagster.op, dagster.asset, dagster.multi_asset and dagster.asset_check — which Tsuda describes as public, stable API. Each factory first wraps the incoming compute function with dagster_otel.traced() before handing it to the real decorator, so the wrapping happens before Dagster ever constructs an op or asset definition.

Ordering matters: the patch has to be active before a user module executes from dagster import asset, otherwise the patched name is never referenced. The opentelemetry-instrument launcher handles this generically by discovering every package registered under the opentelemetry_instrumentor entry point and calling its instrument() method before user code imports anything.

Multiprocess and Kubernetes execution

Dagster's multiprocess executor spawns a fresh interpreter per step, re-importing the definitions module from scratch, so a patch applied only in the original process would not survive. The launcher solves this indirectly: it places a directory containing a two-line sitecustomize.py at the front of PYTHONPATH and execs into the target command. Python imports any module named sitecustomize found on sys.path at startup, and spawned subprocesses inherit PYTHONPATH by default, so each child re-applies the instrumentation on its own.

The k8s_job_executor breaks that chain, because each step runs in a separate Kubernetes pod and PYTHONPATH is not among the environment variables Dagster forwards into pods. The workaround is static: bake sitecustomize.py into the container image so every pod loads it at interpreter startup. Tsuda says both paths were verified against a real kind cluster with Postgres-backed run storage and a Jaeger instance, with spans correctly parented across genuinely separate pods, and a reproducible setup ships in the repository.

Asset check support and coverage

The latest release adds @asset_check, previously the one decorator that escaped tracing. Because it is keyword-only, the same shape as @multi_asset, registering a fourth wrapper required no new dispatch logic. The wrinkle was context: AssetCheckExecutionContext lacks fields such as job_name and selected_asset_keys that op and asset contexts expose, so dagster_otel only learned to handle that context type in its 0.4.0 release, and the instrumentation package bumped its dependency accordingly. Tsuda reports verification two ways: a real materialize() run in the test suite, and a bare script run entirely through the launcher, confirming the patch was live before the script imported Dagster.

Coverage today spans @op in bare and named forms, @asset in both forms, @multi_asset, @asset_check, and @dbt_assets, which is covered for free because it calls multi_asset internally. @graph_asset is deliberately excluded — its function is a composition routine invoked once at definition time and never receives a runtime context, so tracing it would be incorrect rather than merely unnecessary. The ops it composes still get spans.

Why it matters

Distributed tracing is standard practice for web services but has usually demanded per-function effort in data pipelines. This package drops that cost to near zero for Dagster users, making it cheap to see how long each op or asset takes, how steps nest across a run, and how a run links to whatever triggered it — a sensor tick, a CI job or an upstream OpenTelemetry-instrumented service. The caveats are real: the project is alpha-stage and MIT-licensed, auto-instrumentation means giving up explicit per-function control, and the Kubernetes path requires image-level setup. For teams already running an OTel collector, though, the barrier to adding their pipelines to existing traces is now essentially a launcher flag. The code is available on GitHub and PyPI, and the author is soliciting feedback.