RealDiff brings runtime behavior diffing to pull requests in six languages

What RealDiff does

A developer has released RealDiff, an open-source tool that compares what two revisions of a codebase actually do at runtime rather than what their textual diff says. According to the project's README, surfaced through a Show HN post on Hacker News, the tool builds the base and proposed revisions of a pull request, instruments their test runs, and reports the first observable behavior change in each call tree — a point the project calls the frontier. Before comparing, it derives a noise baseline from repeated runs of the base revision (three runs in the README's overview), so flaky timing variation is filtered out rather than reported as a regression.

The example that motivates it

The README's demo centers on a one-line C# refactor: an extension method that sorted a list with List.Sort is rewritten to use OrderBy. List.Sort is not a stable sort; OrderBy is. That single change in an infrastructure helper alters a pricing engine in a separate project the diff never touched. In the demo output, DiscountEngine.SelectDiscount returns "SEASONAL_15" where it previously returned "CLEARANCE_40", and CheckoutTotals.Compute returns 85 instead of 60.

The detail that stings: two of the three tests that exercised the changed path contained no assertion that would have caught it, so a green test suite would not have flagged the shift. The demo script, which mutates only the sorting helper, verifies that the edited file contributes zero traced members while the reported frontier lands in the unedited pricing project.

How it is built

The public executable is a thin Rust launcher that handles argument routing, repository configuration, and language detection. It starts a sibling managed component responsible for ref resolution, builds, caching, instrumentation, and posting results. Each supported runtime has its own tracer, and every tracer emits the same language-neutral NDJSON trace contract, which a single-pass streaming Rust engine consumes to handle matching, noise filtering, frontier computation, and findings. Results land in a findings. file that can be posted to GitHub, Azure DevOps, or an MCP server.

The README also describes a coverage invariant enforced by conformance gates across all six tracers: every discovered module must satisfy discovered = instrumented + skipped, with zero patch failures. The intent is that instrumentation gaps are recorded rather than silently ignored.

Language support and stated limits

RealDiff covers .NET 8 (Mono.Cecil build-time IL weaving, xUnit), Java (a java.lang.instrument agent with ASM, Maven/Gradle), Node and TypeScript (a CommonJS require hook plus an ESM loader with Babel, supporting npm, pnpm, Yarn and Bun), Go (module-aware AST rewriting into the build cache), Rust (syn/quote source rewriting into a content-addressed build cache), and Python 3.12+ (PEP 669 sys.monitoring attached at process start via a staged sitecustomize.py, with pytest and unittest roots). The project describes itself as an early preview.

The limitations are documented candidly. Type and class initializers are structurally unobservable on both the CLR and the JVM, because hooks running under the initialization locks could deadlock startup. Rust macro expansions and compiler-owned code are unreachable through stable source rewriting; the README notes an earlier MIR-based prototype emitted zero runtime events. Node workers are out of scope and exactly one supported lockfile is required. In Python, native and C callables have no Python frame to monitor, and versions before 3.12 are refused outright. Unresolved TypeScript source maps are recorded as unresolved with lowered attribution confidence rather than guessing at an original file path.

Why it matters

Code review shows what a human edited; tests verify only what their assertions cover. RealDiff targets the gap between the two: behavior changes in code paths the diff never touched, exercised by tests that pass anyway. The frontier framing also avoids drowning reviewers in downstream ripple effects by reporting only the first divergence per call tree.

The trade-off is compute. Building both revisions and running baseline repeats fits CI pipelines with time to spare, not fast feedback loops, and the whole scheme leans on the determinism the demo explicitly verifies across fresh processes. As an early preview with per-language gaps, it is best read as a bet that review tooling should reason about executions rather than text — and its conformance gates across six runtimes are the mechanism by which that bet will be judged.